Reading

A reading list from the RSS feeds I follow (happy to get good recommendations)

See my feeds.yaml.

Bob Nystrom

• Representing Heterogeneous Data
  Aug 04, 2023

  As I mentioned in the last post, I’m working on taking my little videogame scripting language and turning it into a statically typed one. As much as possible, I’m trying to make the language simple and familiar. But sometimes those goals are in opposition and the most familiar solution to a problem is kind of a mess. So, I’m also exploring novel approaches and delving deeper into programming language history to scavenge forgotten ideas. The heterogeneous data problem#the-heterogeneous-data-problem One problem every language has to solve is giving users a way to represent heterogeneous data. By that, I mean: Data that might or might not be present. Imagine you have a record for storing a street address: rec Address var number Int var street String var apartmentNumber Int var city String var zipCode Int var state String end But some addresses don’t have apartment numbers. How do you store the apartment number when an address has one but also support its absence? Data that might be in one of several different forms. You’re making a game where a hero can wield weapons. Melee weapons like swords have a single number for how much damage they do. Ranged weapons like crossbows have a pair of numbers for the minimum and maximum range they can reach. How do different kinds of weapons have different fields? These are two sides of the exact same coin. You can treat optional data as data that can be in one of two forms: present with an associated value or absent with no value attached. Functional languages with an option or maybe type do exactly that: The language directly supports data that can have one of multiple forms, and they model absent data using that. Conversely, you could model data being in one of several different forms by having separate fields for all possible forms it could be in. At any point in time, only one of the fields has a value and the others are all absent. If you’ve ever found yourself building a struct or class and writing a comment that says “If this field is blah then this other field will be null.” then you’ve taken this path (and probably felt a little gross doing it). What other languages do#what-other-languages-do I don’t know if broad language tours are your thing, but so much of my job working on Dart involves researching how other languages solve a problem that I can’t help myself anymore. There are a handful of solutions to the problem. I’ll just throw out the ones I know: Null. The most famously maligned approach is to allow any reference variable to potentially refer to “null”, “nil”, or nothing. This means every reference type can directly also represent an absent value. Of course, the problem is that many data fields aren’t heterogeneous and should always be present. If you make every single reference nullable, you’ve lost the ability to distinguish ones that can be absent from ones that really shouldn’t be. This is why many newer statically typed languages either don’t support null at all (Rust and most other statically typed functional languages) or support non-nullable types (Dart, Kotlin, and TypeScript). Variant types. A “variant” type is a type that can hold a value of one of multiple different types. You can think of nullable references as a limited variant type that can hold either a value of one specific type or the special null value but that’s it. Some languages have looser variants that let you store values of any type in the same variable. Untagged unions. C lets you define a data structure whose fields all share overlapping memory. If you have a few different pieces of data that are disjoint—you should only ever have one of them and not the others—then this avoids the memory overhead of storing them all separately. However, in C, the language itself doesn’t keep track of which piece of data you have in the union. It will freely let you write one field and then read out another and it will happily just reinterpret the bits in memory as that other type. Efficient, yes. Safe? No. Sum types. Functional languages going all the way back to ML have a feature also sometimes confusingly called “unions” that is fairly different. Again, you have an object that can store one of a few different kinds of data. But the language also stores a tag in there so that it knows at runtime which piece of data you have. (This is why they’re also called “tagged unions” or “discriminated unions”.) The language uses pattern matching to cleverly prevent you from accessing the data as the wrong type. Subtyping. The object-oriented dual to sum types is subtyping: either inheritance or interface implementation. In an object-oriented language, we could model our weapon example like: interface class Weapon {} class MeleeWeapon implements Weapon { int damage; MeleeWeapon(this.damage); } class RangedWeapon implements Weapon { int minRange; int maxRange; RangedWeapon(this.minRange, this.maxRange); } Code that wants to work with weapons generally uses the Weapon supertype. The two subtypes for melee and ranged weapons each store the fields they need. If you want to go all the way to an object-oriented style, these fields would be private and then you’d have abstract methods in Weapon that are overridden in the subclasses to use them. It’s a complex, heavyweight approach, but a powerful and flexible one. There may be a couple of other weirder language features you can use to model varied data, but I think these are the big ones. Languages tend to pick and choose from this list: Dynamically typed languages essentially treat all variables as variant types. And all the ones I know also go ahead and allow null too. If you’re not going to have any static checking anyway, you may as well be maximally permissive, I guess. Statically typed functional languages lean really hard on sum types. Object-oriented languages obviously primarily use subtyping, though most also have nullable reference types. C makes pointer types nullable and supports untagged unions. It doesn’t have (checked) variants or subtyping, but it can approximate both by allowing pointers to be cast to different types. C++ takes everything C has and also explicitly supports subtyping. Newer, bigger multi-paradigm languages like C# and Swift tend to take just about all of the approaches. Whither for my little language?#whither-for-my-little-language OK, so what’s the right approach for my aspirationally simple and elegant statically typed game scripting language? I quite like object-oriented programming in general, but subtyping adds a lot of complexity to a static type system, so my current plan is to not have subtyping in the language at all. That rules out that approach. My goal is for the language to be fairly high level and expressive. It’s supposed to be a language that makes making games fun, not necessarily a high-performance machine for engineering giant AAA games. I want you to have a good time tinkering on pixelly 2D games, not write the next Unreal Engine in it. To that end, the language is garbage-collected. That means memory safety, which rules out untagged unions. Back when this language was dynamically typed, it had nil, so that’s an obvious approach. But I’ve spent, like, way too much of my time rooting out nullable references from Dart and the last thing I want to do with my hobby project is to go back to square one. That basically just leaves sum types and variant types. Given that my language is statically typed and not object-oriented, sum types are the obvious approach. Everyone who uses sum types loves them, myself included. Algebraic datatypes are just cool. And, in fact, I went ahead and implemented a protype of sum types and pattern matching and destructuring in my language. It worked. It was… just OK. To explain why requires a little context An imperative, procedural language#an-imperative-procedural-language My language is unabashedly imperative. I like imperative programming, especially for scripting little games. Games are giant balls of mutable state. I’ve watched my kids and many others learn to program, and imperatively modifying stuff seems to be a natural way to think about defining a process. When you read a recipe for cake, you don’t see steps like: “Produce a new bowl of batter which is the previous bowl of batter and 2 cups of sugar.” It just says “Add 2 cups of sugar to the bowl.” Now, I know all of the problems with mutation of state and imperative code when programming in the large. I get it. But this is supposed to be a fun little language for fun little games and, to me, imperative programming fits that to a tee. The basic vibe I have for the language is similar to Pascal, C, or BASIC: In other words, a classic procedural language. Structures and functions. It looks like this: rec MeleeWeapon var damage Int end def attack(weapon MeleeWeapon, monster Monster, distance Int) if distance > 1 then print("You are out of range.") return end var damage = rollDice(weapon.damage) if monster.health damage then print("You kill the monster!") monster.health = 0 else print("You wound the monster.") monster.health = monster.health - damage end end What’s cool about simple procedural code is that even though I have no idea what language you know and you certainly have never programmed in this one, I’m still pretty confident that you understand this code. With sum types#with-sum-types Let’s see how it looks with something like sum types: rec Weapon case MeleeWeapon var damage Int case RangedWeapon var minRange Int var maxRange Int end def attack(weapon Weapon, monster Monster, distance Int) var isInRange = match weapon case MeleeWeapon(damage) then distance == 1 case RangedWeapon(min, max) then distance >= min and distance max end if !isInRange then print("You are out of range.") return end var damage = match weapon case MeleeWeapon(damage) then rollDice(damage) case RangedWeapon(min, max) then max - min end if monster.health damage then print("You kill the monster!") monster.health = 0 else print("You wound the monster.") monster.health = monster.health - damage end end The sort of weird rec syntax is defining a sum type, Weapon, with type constructors MeleeWeapon and RangedWeapon. I’m still noodling on the syntax. Now, the code here works. And it’s safe. The compiler and the structure of the pattern matching code itself prevent you from accessing the wrong fields from a weapon of a different kind. That’s cool. But it’s so much weirder than the previous code. In a procedural language, the idiomatic way to access fields on records is simply record.field. That syntax is in almost every programming language all the way back to Algol. But once you hop over to sum types, you lose that syntax entirely and have to instead sort of “invert” the code and use pattern matching and destructuring. I do love pattern matching and destructuring—I just spent the past year of my life adding it to Dart. But for this language, I’m pushing really hard on simplicity. If possible, I don’t want two different ways to access state on a value, depending on whether the field is case-specific or not. More to the point, there’s no graceful way to handle mutable sum type fields using pattern matching. SML eschews mutability in general and then works around it by allowing you to define explicit mutable ref types. But that’s definitely not how my language rolls. Variant records#variant-records There is one other approach to heterogeneous data that I found that I didn’t put in the list up there because, as far as I can tell, it’s basically a dead end in the evolutionary history of programming languages. Some versions of Pascal have a thing called “variant records”. A record in Pascal is your basic “collection of fields” struct type. A variant record says that some of those fields are only accessible when the record is one of a few different enumerated states. In C, it’s common to wrap an untagged union in a struct along with a tag enum indicating which branch of the union is active: typedef enum { WEAPON_MELEE, WEAPON_RANGED } WeaponType; typedef struct { WeaponType type; union { struct { int damage; } melee; struct { int minRange; int maxRange; } ranged; } as; } Weapon; Using it looks something like: Weapon weapon; weapon.type = WEAPON_MELEE; weapon.as.melee.damage = 6; A variant record in Pascal (as I understand it from the half dozen ancient slideshows I’ve been able to find about it) essentially models that pattern directly. The cool thing about this feature is that the variant-specific fields are accessed using the same familiar field access syntax used everywhere else. That also means variant-specific fields can be mutable. Of course, the not cool thing about using that same field syntax is that there’s nothing preventing you from accessing the wrong variant field: Weapon weapon; weapon.type = WEAPON_MELEE; weapon.as.melee.damage = 6; printf("Min range %d\n", weapon.as.ranged.minRange); // Oops. There is a type tag, but the language doesn’t know and doesn’t check it. This is definitely true in C and I think true in Pascal. (It’s always hard to talk about Pascal definitively because there’s no “Pascal”, just a huge family of loosely-related Pascal-ish languages.) In a memory safe language like mine, I definitely don’t want users to be able to reinterpret memory. But that’s a solvable problem. Record cases#record-cases Which, finally, brings us to the feature I designed for my language. It’s very close to variant records in Pascal. The type declaration looks just like the sum type example: rec Weapon case MeleeWeapon var damage Int case RangedWeapon var minRange Int var maxRange Int end The difference is that you don’t need to rely on pattern matching to access the variant fields. They’re just fields: def attack(weapon Weapon, monster Monster, distance Int) if weapon is MeleeWeapon and distance > 1 or distance weapon.minRange or distance > weapon.maxRange then print("You are out of range.") return end var damage = if weapon is MeleeWeapon then rollDice(weapon.damage) else weapon.maxRange - weapon.minRange end if monster.health damage then print("You kill the monster!") monster.health = 0 else print("You wound the monster.") monster.health = monster.health - damage end end Of course, you lose the compile-time safety that pattern matching gives you where you can’t access fields of the wrong type. But we don’t need to go all the way to C’s level of unsafety. Instead, when you access a case-specific field on a record, if the record’s type tag is set to a different case, the access throws a runtime error. This preserves memory safety. This is a real trade-off. The feature I have here provides strictly less static safety than using sum types. There is a slight performance cost to checking the type tag when accessing case-specific fields. In return, you get simpler, more familiar syntax for working with case-specific fields, including mutable ones. Also, it allows a single record to have a mixture of shared and case-specific fields: rec Weapon var name String var bonus Int case MeleeWeapon var damage Int case RangedWeapon var minRange Int var maxRange Int end Here, name and bonus can be accessed on all Weapon instances, but the other fields are case specific. It sort of combines product and sum types into a single construct. I’ve found this to be really handy in practice. I haven’t decided if I’m totally sold on this feature yet. But in the (admittedly small) amount of example code I’ve written using it so far, it seems to feel pretty nice. For a small game scripting language, I think it may strike a decent balance between static safety and simplicity. Update: What about flow typing?#update-what-about-flow-typing When I first posted this, the most common reply was why not do some sort of flow typing? In code like: def attack(weapon Weapon, monster Monster, distance Int) if weapon is RangedWeapon and (distance weapon.minRange or distance > weapon.maxRange) or distance > 1 then print("You are out of range.") return end # ... end The compiler could do control flow analysis to determine that the .minRange and .maxRange calls are guarded by an is RangedWeapon and thus allow them. But if you don’t guard the code with that kind of check, you’d get an error: def attack(weapon Weapon, monster Monster, distance Int) if distance weapon.minRange or # Error! Can't access .minRange here. distance > weapon.maxRange then print("You are out of range.") return end # ... end This is definitely a thing you can do! TypeScript, Kotlin, Flow, Dart, and others all support it. The general technique is called “control flow analysis” and the specific feature is called “flow typing”, “smart casts”, or “type promotion” depending on which language. Is it a good fit for my language? I do like that it makes imperative code “just work” while being safe. But that “just” is doing a lot of heavy lifting. We do this analysis in Dart and it is fantastically complex. Proving that a certain piece of code can only be reached by going through some other piece of code first gets hard quickly in the presence of loops and closures. It seems like every release of Dart, we ship more extensions to flow analysis because users keep expecting it to be smarter and smarter. Also, it isn’t sound in many cases that users expect to work. Once the variable that you’re type testing can escape the current function, the compiler generally can’t prove that it won’t be mutated between when you test its type and when you use it as the more precise type later. Overall, my feeling is that it works out pretty well for Dart, but it’s a large sort of messy feature that feels a little too magical. A goal with my hobby language is that you should be able to have the whole language loaded into your head and rarely be surprised by what it does. Flow analysis in Dart still fairly often surprises me and I literally work on the language full-time. There’s also the question of what you promote the tested variable to. In my language as it currently stands, there is no subtyping. MeleeWeapon isn’t a subtype of Weapon, it’s a case constructor. The weapon is MeleeWeapon syntax looks like a type test, but it’s really more like an enum case check. So after that test, what type would weapon have? It would still have to be Weapon. I guess I could make this work by not promoting the type but by having the type checker track an extra “known case” property for each static type and then use that. That might work. But even with that, I worry that it would quickly become annoying. Let’s say you refactor the above code to: def attack(weapon Weapon, monster Monster, distance Int) if weapon is RangedWeapon and checkRange(weapon, distance) or distance > 1 then print("You are out of range.") return end # ... end def checkRange(weapon Weapon, distance Int) Bool distance weapon.minRange or distance > weapon.maxRange end That no longer works. Inside checkRange() the compiler has lost track that weapon is always a RangedWeapon. You could come up with a way to annotate that, but now we’re back to subtyping and all the complexity it involves. So, overall, yes, subtyping and flow analysis is a thing that could work here, but I’m trying to avoid it because I feel like it’s a bigger lump of complexity than I want to take on. I’d be more inclined to do sum types and destructuring, even though it feels a little weird in an imperative language, then do this kind of complex control flow analysis.

• Type Checking If Expressions
  Jan 03, 2023

  I have this hobby project I’ve been hacking on for several years. It’s a fantasy console, very much inspired by the delightful PICO-8. Like PICO-8, my console has its own built-in scripting language. Because I’m me, I of course took the opportunity to design an entirely new language. My goal for the project is a fun way to build small-ish 2D games. I want its scripting language to be expressive enough to be joyful, but small enough that you can learn the whole language and never need to consult a reference manual after that. My dream is a goofy little pixellated IDE where you can get lost in your own flow state and just make shit without having to periodically hit StackOverflow and then get distracted by the wonders/horrors of the Internet. I don’t know if I’ll ever pull this off or the language will ever see the light of day, but it’s a fun therapeutic thing for me to noodle on. A dynamically typed scripting language#a-dynamically-typed-scripting-language To make a language that fits in your head (or at least my head, whose working space seems to get smaller every year), I needed to jettison as many feature as I could. My experience across a range of hobby and not-so-hobby languages is that static types add roughly an order of magnitude of complexity, so types were one of the first things to go. Like most scripting languages, I made mine dynamically typed. Here’s an example: def onTick() var d = 0 if buttonHeld(2) then d = d - 1 end if buttonHeld(3) then d = d + 1 end if d != 0 then h = h + d else if h > 0 then h = h - 0.5 else if h 0 then h = h + 0.5 end if h -3.0 then h = -3.0 end if h > 3.0 then h = 3.0 end x = x + h if y 200 then v = v + 0.8 end y = y + v if y > 200 then y = 200 v = 0 end if buttonPressed(0) then if y == 200 then playSequence() v = -10.0 end end end Another simplification I made is to eliminate the distinction between statements and expressions. As in Ruby, Kotlin, and most functional languages, everything is an expression. The previous chained if could be written in a more explicitly expression-y style like: h = h + if d != 0 then d else if h > 0 then -0.5 else if h 0 then 0.5 end Unifying statements and expressions means the language doesn’t need a separate if statement and conditional expression. Also, I don’t know, I just like expression-oriented languages. An imperative language#an-imperative-language Even though the language is expression-oriented, it’s not explicitly functional. Functional languages are close to my heart, but this is a game scripting language. A game world is basically a big ball of incrementally updated mutable state. For the kind of programs and user experience I have in mind, I think an imperative, procedural style is easy to learn, and fun to program in. I want users thinking about their game, not, like, monads and persistent data structures. So while everything is an expression in my language, it doesn’t at all shy away from side effects and imperative control flow. Variables are assignable. Fields are settable. There are loops and breaks and early returns. All of those are as natural and idiomatic as they are in C++, JavaScript, C#, or any of the other languages that the majority of the world’s code is written in. Handmade Seattle#handmade-seattle Last fall, I attended the wonderful Handmade Seattle conference. I had a particularly inspiring conversation with Devine Lu Linvega about their tiny uxn VM. They had this idea to build the smallest possible system and programming language for their own use. Then they rebuilt their own personal tools—text editor, music stuff, etc.—using that. Now, UXN is really minimal. I get a certain satisfaction from programming in assembly, but it’s not the language I would want to use for my own joy. But it did make me rethink the scripting language for my fantasy console. I picked dynamic types because that made the language smaller and I figured it would be a good fit for my (entirely hypothetical at this point) users. But is it what I’d want to use to make little 2D videogames? The game I’ve spent the most time hacking on is my also-perennially-incomplete roguelike Hauberk. I’ve rewritten it several times, but every incarnation has been in a statically typed language: C++, C#, Java, and now Dart. My most pleasurable time spent working on Hauberk is when I’m refactoring and the type system guides me to what’s left to clean up. I just really like working with types. (It’s OK if you don’t. As our Burger Sovereign says, have it your way.) After talking to Devine, I realized that if I was making this fantasy console for me personally, its language would be typed. So over the past few weeks, I’ve been sketching out a statically typed variant of my console’s scripting language. I don’t know if it will really come together, but I thought maybe it would be fun to write about the exploration. Type checking if expressions#type-checking-if-expressions I slapped together a new prototype interpreter for my language. (The main implementation is a bytecode VM in C++, which is pretty fast but not exactly easy to hack on.) Then I dutifully started adding a type checking pass to it. One of the first challenges I hit is how to type check if expressions. As the title up there implies, that’s what this post is really about. Because it turns out that having if be an expression while also fully embracing an imperative style gets a little weird when it comes to type checking. I’ll walk through a bunch of examples and build up to the type checking rules I have settled on (so far, at least). We’ll start simple: var love = if isFriday then "in love" else "not in love" end We need a type for the if expression so that we can infer a type for the variable love. In this case, the type is obviously String since both the then and else branches evaluate to strings. So the basic rule we’ll start with is: An if expression’s type is the type of the branches. Different branch types#different-branch-types But what if they don’t have the same type? What about: var love = if isFriday then "in love" else 0 end Here, love could end up being initialized to either a String or an Int. Now what type do we choose? Crystal’s answer is String | Int. Union types are cool but definitely too complex for the language I’m trying to make. In Kotlin, which is also typed and expression-oriented, the answer is, apparently, {Comparable & java.io.Serializable}. Which I have to say does not seem super helpful. I assume that the compiler goes looking for a shared supertype of the two branch types, String and Int. Since String and Int both happen to implement Comparable (and I guess some serialization interface), you get that as the common supertype. In object-oriented languages with subtyping and where the type hierarchy forms a lattice, this common supertype is the least upper bound, and it’s a natural answer to the problem. It shows up in other languages when type-checking conditional ?: expressions and a few other places. It works, but, as we can see in the Kotlin example here, it doesn’t always produce intuitive or useful results. More to the point, one of the other features I jettisoned from my scripting language is subtyping, so LUB is off the table. Without subtyping, every type is disjoint: a value of one type is never a value of any other type too. That means that if the two branches of an if have different types, then there is no possible type I can infer that contains all of their values. The only other response is to make it a type error. That’s the next rule: If the branches have different types, it’s a compile error. Imperative ifs and unused values#imperative-ifs-and-unused-values That rule does work: It’s basically SML’s rule for if expressions. But I want my scripting language to feel familiar to users programming in an imperative style. Consider: var daysNotInLove = 0 if isFriday then print("in love") else daysNotInLove = 1 end Here, the two branches have different types. The then branch has type String because in my language, print() returns its argument. (That makes it handy for stuffing some debug printing in the middle of an expression.) The else branch has type Int because an assignment expression yields the assigned value. According to the previous rule, this is a type error because we don’t know what type of value the if expression evaluates to. But it doesn’t matter since the if’s value isn’t being used anyway. There’s no need for the compiler to yell at you, and code like this turns out to be very common in practice. To address this, the type checker takes some of the surrounding context into account. When an if expression appears in a location where its value won’t be used, then it’s no longer an error for the branches to have different types. How complex is tracking that context? Not too bad, actually. There are a handful of cases: In a block or function body where you have a sequence of expressions, the result is the value of the last expression. The values of all of the preceding expressions are discarded. So in an expression sequence, all but the last expression are in a “value not used” context. Like other expression-oriented languages, functions in my language implicitly return the value that the function body expression evaluates to: def three() Int print("About to return three...") 3 end def onInit() print(three()) # Prints "About to return three..." then "3". end But if a function has no return type (the same as void or unit in other languages), it doesn’t return a value. In that case, even the last expression in the body is a “value not used” context. Loop expressions don’t produce values, so their body is always a “value not used” context. (I’m toying with the idea of allowing break expressions to yield a value from the loop, but they don’t right now.) Whenever an if or match expression is in a “value not used” context, then we push that context into the branches too. Likewise with the right-hand side of and and or logic operators since those are control flow expressions to. That’s it. After I came up with this rule, I did some poking around and it seems like Kotlin does something similar. It frames it by saying that when you use an if “as an expression” then the two branches must have the same type. That’s roughly the distinction I’m making here too: when an if appears in a statement-like position where its value is discarded, then the branches can disagree. Missing else#missing-else This rule allows us to support an even more important flavor of if expressions that are common in imperative code: those without else clauses. In SML and some other functional languages, every if expression must have an else clause because the presumption is that you will be using the value produced by the expression and you need a value even when the condition is false. But in imperative code, it’s obviously common to have ifs whose main purpose is a side effect and where an else clause isn’t needed. In fact, when I analyzed a huge corpus of real-world Dart, I found that only about 20% of if statements had else branches. Now that we understand when an if expression is in a context where it’s value isn’t used, we can allow omitting else branches those. The next rule is: An if expression can omit the else branch when in a context where its value isn’t used. Exiting branches#exiting-branches We’re almost there. It’s starting to feel like we really are type-checking an imperative language, not ML in BASIC’s clothing. I coded this up and successfully wrote some little example programs. It was starting to feel like a real typed language! I could stop here, but there’s one last bit of type checking logic for if expressions. I haven’t decided if it’s worth keeping. Consider: def onInit() var love = if isFriday then "in love" else return end end When isFriday is true, then this initializes love with the string “in love”. When isFriday is false, then the return exits from the function entirely so love never gets initialized at all. So even though these branches don’t evaluate to the same type, love is always initialized with a String. This code should be fine. Or, at least, it should be sound according to the type system. Whether this is good style is definitely open for debate. I could probably not allow code like this. But my default stance is to be as permissive as possible without breaking soundness, and this is a corner where I can be. The trick is that expressions like break, return, and throw are special. While they are expressions grammatically, they don’t actually evaluate to values. If you do: var x = return That x never gets initialized. A return expression always jumps out of the surrounding code instead of producing a value. Languages that have expressions which can do control flow model this by giving these expressions a special type variously called “bottom”, ⊥ (“up tack”), Never, noreturn, etc. This type means “You’re never gonna get a value from me.” When checking the two branches of an if expression, if one branch has that special type (the compiler calls it “unreachable” right now), then we just use the type of the other branch for the if expression’s type. That allows the above example to work. In the sample code I’ve written so far, it rarely comes into play. It’s usually more idiomatic to hoist that control flow out of the if entirely. But we can type check it easily, so the language lets you do it. The rules altogether#the-rules-altogether That’s where I’m at right now. It took me a few iterations to get to a point where all the if expressions I expected to be able to write in my example programs actually type checked correctly but it seems pretty stable now. The rules are: When an if expression is in a context where its value is not used, then there is no restriction on what types the branches can have and we’re done. Otherwise, there must be an else branch and: If both branches have type “unreachable” then the if expression’s type is also “unreachable”. If one branch has type “unreachable” then the if expression’s type is the type of the other branch. Otherwise, the two branches must have the same type and the type of the if is that type.

• Stupid Dog
  Feb 13, 2022

  It’s my 30th birthday, and Megan and I are walking down Duval Street in Key West. There is a pet store here, which must be the most unlikely place for a pet store in all the world. Everyone here is a tourist. What kind of idiot buys a dog while on vacation? You are in the window surrounded by other puppies. A tottering orange ball of fuzz with little triangular ears, like a fox who lost its tail. Ink-black eyes and round button nose. My wife, pregnant and hormonal, is instantly in love with you. We go inside to play with the puppies, because why would you not want to play with cute puppies while on vacation? You curl up in my cupped hands, calm and sleepy. Megan gives me a lingering series of Significant Looks, which I do my best to ignore. I generally try to avoid making dumb decisions, and impulsively buying a puppy while on vacation eight hours away from home is clearly in that category. I make a silent vow to never let Megan know that I also fell in love with you the moment I saw you. It’s the next day and Megan and I are sitting at a Burger King across the street from the pet store. She is presenting her case for why we should buy this very cute half-Pomeranian half-Yorkie puppy even though: We are on vacation. We are expecting our first child in a couple of months and already have more than enough to keep us busy. We have absolutely no plans or intentions on getting a pet. Our house is tiny and barely fits the two of us and two cats, much less a baby and a dog. We are an eight-hour drive from home. You are, in fact, quite expensive. Do I even need to write the numbered list here? It’s obviously a terrible idea. Family members that we’re vacationing with look at us like we are idiots when we say we’re considering it. I think very very hard about my beloved first dog, Snickers, named because she was chocolate brown with swirls of white and caramel. She was a Sheltie, the smallest of her litter, a funny little runt of a puppy who entered my life when I was feeling small and cast out too. When I was seven, I was playing in a neighbor’s front yard while she was out with me. I got distracted and heard a screech of tires behind me. When I turned around, Snickers was on her back motionless in the street. I carried her limp body all the way back to my house. The shocked look on my mother’s face when she saw me arrive, tears streaming down my face, broken dog in my arms, is my most vivid childhood memory. I am the kind of person who analyzes 100 steps ahead. Getting a puppy means buying a ticket for the whole ride, including the end of it, in whatever form it takes. I am choosing to break my own heart. What if it’s not worth all that pain? Megan is driving us north from Miami. You are sleeping in my lap, impossibly warm, cold wet nose pressed against my arm. We talk about names. You almost end up “Dorito” because of your triangular orange ears, but it doesn’t stick. Instead, you become “Ginny”. Not for Ginny Weasely like everyone assumes but short for the liquor genever. When we buy you, I also buy some books on dog training because I am the kind of person who wants to be a very good dog owner that follows all the dog owning rules. Rule #1 is crate training starts immediately. A dog won’t respect you as the pack leader if they are free to sleep where you do. They’ll think they run the pack. It’s late at night by the time we get home, and all the pet stores have closed. Next to the bed, I try to improvise some crate-like thing for your first night involving cardboard boxes, a cooler, and anything else I can find around the house. You yowl in terror, or perhaps just disappointment at the poor accommodations. Eventually, I give up. I pick you up and lay you on my chest. You are asleep instantly, tiny heart fluttering against mine. You are taking your sweet time exploring every single corner of the jungle that is our backyard, looking for lizards. It is three o’clock in the morning. The green and brown anoles are sleeping under leaves where you never find them. I am shambling after you in pajama pants and old boots telepathically trying to get you to pee with every ounce of willpower I have. I am keenly aware of my dwindling opportunities to get a good night’s sleep before the baby arrives. Yet here we are. Over the past few weeks Megan and I have mostly learned to tell the difference between your yowl that means “I am a puppy and don’t like being alone in this crate” and the yowl that means “My bladder is too small to last the night and I need to pee right now.” Sorry about the mistakes we made while figuring that out. I pee in the yard too, hoping you’ll get the hint. I deeply regret ever seeing you in that pet store window. Half-awake, I have vivid fantasies of building a time machine, traveling back in time, and slapping myself in the middle of that Key West Burger King. After patrolling the perimeter of the yard in tiny puppy steps, twice, you finally pee. I tell you what a good girl you are and give you a treat. Because you are an adorable tiny puppy and I am a nerd with a DSLR, I take copious photos of you and upload them to Flickr. One day, I get an email from cuteoverload.com asking if they can share some of them. The next day, you are Internet famous. Millions of people look at your butthole. You are oblivious to your celebrity, as you are to most things in life. As you grow, your puppy fuzz is replaced by silk, still orange on top and white on the bottom like a fox. The whorls of fur behind your ears are the softest thing I have ever felt in my life. I run my fingers through them constantly, to your delight. I take you on walks every day, still trying to be some kind of adequate dog owner. The training manuals say that to teach a puppy walk properly on a leash you should stop walking whenever the dog pulls against it. Then wait patiently until the dog turns and makes eye contact with you. That way, they learn that you are in charge of walking and that pulling means the walk ends. It takes us half an hour to walk a single block. Despite repeating this process hundreds of times, you fail to make the mental connection that if you stop yanking on the leash then we can actually walk where you want to go. You never, ever stop pulling on the damn leash. I start a running joke of whispering insults in your ear that sound like terms of endearment. It’s not like you speak English, so when I say, “I hate your face,” all you hear is the caring tone I use to say it. “You’re so fucking stupid,” I coo into your ear. I rub your fuzzy belly to make sure you know how much I love your stupid face. Five weeks earlier than expected, Megan and I become parents. We haven’t had time to set up a nursery yet so we hastily buy a bassinet. The first night the baby is home from the NICU, we realize there isn’t enough room at the foot of the bed for both the bassinet and the crate. We let you sleep on the bed. You are immensely proud of earning a place on the bed while the new pink hairless dog is relegated to the funny crate. You are listless and frothing at the mouth. Megan and I are terrified. You had been in the yard stalking all of the various animals that live in the barely-tamed-jungle that is Orlando. This time, you found a toad and you’d managed to get your mouth on it before I could get to you. Did it poison you? Are you going to die? We rush you to the emergency vet. She takes your vitals while you sit there limp and apathetic. Your blood sugar is dangerously low. She gives you a treat. Then another. You immediately perk up. You’re fine, totally fine. I remember that a couple of days prior I put my foot down and decided you were ready to graduate from soft puppy food to kibble which is better for your teeth. You were uninterested in the new food. I confidently told Megan, “If she gets hungry enough, she’ll eat. It’s not like she’s going to starve herself.” Apparently, somehow during the thousands of years humans spent turning a wolf into the strange-colored, tiny, eternally cub-like thing that you are, we broke your survival instinct. Sorry about that. We learn to mix some wet food into your kibble. It is something like four in the morning and Megan and I are pushing a baby stroller and a luggage cart across the SeaTac airport with you and the cats crammed in pet carriers perched on top of our suitcases. We’ve just flown 3,000 miles to our new home. Our brilliant idea to take a red-eye flight so the baby would sleep through it failed disastrously. Megan and I spent the entire flight taking turns trying to get the baby to sleep on our laps. You seem to have mostly survived the flight intact. The cats were less thrilled by the experience. We get to the hotel, and I take you out just as the sun starts to come up. The parking lot of an Extended Stay in Redmond is my first view of our new home in the Pacific Northwest. It’s fifty degrees and rainy. I’m beyond tired. You take your sweet time before finding an acceptable spot to pee. We are on PetFinder looking for a second dog. You are just so… weird. Not food motivated. Impossible to train. High strung. Frankly, not very smart. You bark all the time, at everything. We tell ourselves that a second dog will give you company when we’re out. They will teach you how to dog like a normal dog. We find a part-Chihuahua part rat terrier that is, in defiance of everything I understand about genetics, practically your twin. We bring you to meet him at a dog park to see if you get along. You bark a lot. He tolerates you admirably well. For the rest of your lives, you will be his asshole boss. You’ll eat the best parts out of his food bowl and steal his treats. When he tries to sit next to us on the couch, you will wedge yourself in the middle and glare at us both. He will live in fear of your ire which is capricious and often. We name him Benny for “beignet” because he’s golden with a sprinkling of white on top. Maybe they were right about the crate training. We rent a house out in the sticks. There is a green belt behind the yard and we get deer sometimes. Occasionally, we hear an owl hoot. Inexplicably, you are absolutely terrified of the sound. Megan and I hoot at you to tease you and you bark back at us, shivering in fear. Later, we buy a house with a fenced in back yard. You stand proudly in the grass waiting for me to make eye contact with you. The moment I do, you throw your head back in delight and run full tilt to me. Over and over again from Megan to me and back. You wear yourself out and lay in the sun panting “keh keh keh”. When you snuggle with us on the couch later, you still smell like dirt and grass. I am in the basement awkwardly taking photos of myself. I’m speaking at a conference and they asked for a headshot. You wander in and I pick you up and start taking pictures of us together. They are my favorite photos of myself. When I finish my writing my book, I put one of them on the back cover. I get email from readers telling me how delighted they are with the cover. I set that photo as my profile image on all of my various accounts. When ride share drivers pick me up, they joyfully tell me in not-great English that they love my dog. You are still a celebrity. I am in bed staring at the dark ceiling, doing math in my head. Tonight, you are the same age in dog years that I am in real years. From this moment on, you are older than me. I picture you racing ahead of me into the future. I try not to think of where that path leads or when you’ll get there. You are sleeping stretched out, pressed against my thigh, trying to get every square millimeter of contact that you can. I run my fingers through the silky fur on your back, feel your breath rise and fall. We rent a cabin on the Pacific coast. You spend the first half of the drive shaking in excitement and anxiety, paws on the door looking out the window. The whole car smells like your horrid breath and Megan is coated in orange fur. Eventually you wear yourself out and pass out in her lap, head draped over her arm towards me. I rub the fur between your ears while I drive. After settling in, we go explore tide pools. You sniff everything. Every rock, bit of kelp, rotting fish, even the wind. It’s cold and wet, but you don’t mind. When we get back to the cabin, I wipe sand and seaweed from your paws. You fall asleep in front of the fireplace. The vet is explaining to us that you have hyperthyroidism and congestive heart disease. That’s why you pee all the time—often in the house—and why you seem so tense and wound up all the time. Well, that and because you are half Pomeranian. You get put on an expensive pill regimen which you will take for the rest of your life. Since you aren’t interested in food even on a good day, getting you to take them will be a constant battle. At least the peeing inside gets marginally better. The barking does not. I start writing a second book. I work on my laptop on the couch and you lay pressed against one leg the entire time. Once you’ve fallen asleep and let down your jealous guard over access to me, Benny claims the other side. We take another trip out to the Olympic Peninsula with friends. You spend much of it sleeping on the couch in front of the fire, like an overweight self-warming throw pillow. You are given affection from everyone, which you regard as your proper due. In return, you grace us with an endless series of farts that leave the cabin smelling like some sort of reptilian tire fire. It’s six o’clock in the morning. I am barely awake making coffee while you are outside for your morning consitutional. I hear a strange sort of yelp at the back door. I open it just in time to see you collapse. I scoop you up and run into the dining room, yelling out for Megan. I sit on the floor and cradle you, rocking back and forth. If these are your last moments, I want them to be peaceful. I hold your limp body in my arms. Your tongue hangs out, lifeless and blue. I am seven years old again. After an endless minute, you come back to us. Your back stiffens. Your legs poke out like a fainting goat and then relax. Your eyes start darting around and eventually you are able to pick your head up. Ten minutes later and you are back to normal, or at least as normal as you ever are. Honestly, a loss in cognitive function would be hard to detect. The vet begins to use the phrase “congestive heart failure” now. The pill regimen gets more complex. We throw out yet another rug too pee-stained to repair and decide to not have a rug in the dining room “for now”. You are mostly enjoying your retirement. You spend so much time laying on the back of the couch so that you can look out the back window and bark at anyone with the audacity to walk on the sidewalk that the cushion loses its shape entirely. We alternate between trying to give you as much affection as we can in the time we have left and cursing your name as we discover another part of the house ruined by your sneaky peeing. I discover right in the middle of a push-up, my nose an inch from the floor, that you and Benny have surreptitiously saturated every inch of the carpet in the office. Megan and I learn how to put in new flooring. We enter the phase where you take medications to cope with the side effects of other medications. You get over a dozen different pills a day, each on their own specific schedule. I print a chart and put it on the fridge to make sure I don’t miss any. It is exhausting, but mostly for us. You are still relatively content, lazing around the house. Stealing Benny’s food. Barking at people outside the window. Sometimes, at night, you crawl between my legs on top of the blankets, pinning me and forcing me to sleep on my back. I tolerate it as best I can. You’re warm and your ears are still velvet-soft. One morning, I wake up to discover you have leaked little brown droplets all over the blanket. You are sitting up in bed, panting, clearly uncomfortable. (I’m not exactly comfortable with the situation myself.) A couple of days later, you faint again. Perhaps a stroke. Dog.exe has stopped working. When you finish your reboot cycle, you aren’t the same. You don’t recognize Megan and snap at her viciously. You stand stock still in the middle of the dining room floor, eyes glazed. It’s clearly time. I feel an unexpected relief. For years, I have held a deep fear that I wouldn’t know when it was over. That I wasn’t adult enough to handle the responsibility of making the call. I’d procrastinate too long and leave you in misery or choose prematurely and cut your life short. Or I would just be wracked in anguish the whole time. Your last gift to me is a swift decline so that there’s no doubt. I learn there is an entire category of vet-adjacent businesses that specialize in this “transition”. Early on a Tuesday morning, I start calling them to see who can come out today, as soon as possible, right now. The women who answer the phone have well-rehearsed, soothing voices. I stumble over my words. I don’t know the right social protocol to ask, “Can you please come kill my dog?” You are sitting next to me on the couch, pressed warmly against my side like always. You’ve mostly calmed down but are sitting up and tense. A kind woman radiating Pacific Northwest Earth Mother vibes is sitting on the other side of you with a small array of syringes in front of her on the coffee table. We’ve tucked a blanket under you. The kids have already said goodbye to you, tears pouring down their faces. Surprisingly, that turns out to be the hardest part, realizing how much you are part of their stories too. The first shot makes you fall asleep. Literally, not in the “put to sleep” figurative sense. As the woman pushes the plunger, you loll slowly over onto my leg. You’re napping, head on my thigh, more comfortable than I’ve seen you in weeks. Like old times. I run my fingers through the fur on your back, up to where it gets shorter and velvety between your ears, around your side where it gets thinner and silkier, to your chest where it thickens into whorls. I leave my fingers there where I can feel your heart beating. I want to be here for it, to know that I was with you from beginning to end. When the second shot goes in, your music stops. With practiced timing, the woman has your body wrapped up in the blanket and out the door in minutes. The second the door closes, I feel the mask fall off my face and the tears start. It’s later that day. I’ve thrown out the pills, pill poppers, prescriptions, aftercare instructions, wrecked towels and blankets, every reminder of your failing health. I am out for a walk to get out of the house. It hits me a few blocks away. The answer to the question I carried since the day we got you. Will the life we have with you outweigh the pain at the end? Will it be worth it? It is. Every minute. It’s a couple of years later. Your ashes are in a little box in my nightstand. I remember the stress and exhaustion of the last few months of your life, but it feels vague, like I remember the stories I told people about it but not the actual experience. I remember all of the good moments with you vividly. The feel of your fur between my fingers, your body pressed against my side, your horrific breath panting in my face. Megan and I talk about your antics. I still miss you and your stupid face. Ginny 2008 – 2019

• 640 Pages in 15 Months
  Jul 29, 2021

  My book Crafting Interpreters on programming languages is done. OK, OK. I know I said it was done like fifteen months ago. But now it’s really done. And by that I mean, the print, e-book and PDF versions are done. You can buy it. You can hold it in your hands. And I do mean “hands” plural. Because this little “handbook” turned out way larger than I anticipated: This is a proof copy, so it looks a little different than the final design. Look at that thing. 640 pages, eight inches wide, ten inches tall. If you get tired of reading it, it can serve as a doorstop or protect you from small-arms fire. Remember back on Mr. Roger’s Neighborhood when he would take you to a factory and show you how pencils or umbrellas are made? I love that stuff, so I thought maybe you might like to see what I spent the past year on. You can read this as a peek behind the curtain, or maybe a long apology for why it took so long. Where I left off#where-i-left-off What I said in my last post was that Crafting Interpreters was finished, and by that I meant that I’d written all the code, dotted the last sentence, and scanned the last illustration. The content of the book was complete. But it wasn’t a book yet. Just a pile of Markdown and PNG files that my embarrassingly bad Python code would begrudgingly turn into a website. My goal has always been to make a real book, with pages and everything. Completing the content was just one (granted, the most important) step in that journey. Once I finished the last chapter and put it online, I took about a month-long break. I had been writing every single day for almost four years and I was fried. Also, in case you don’t remember, early 2020 was not exactly a Totally Fine Time in World History. A new build system#a-new-build-system After a few weeks, my desire to have this project completely done returned and I eased my way back into working on it. I fixed a whole pile of typos and other mistakes that readers filed bugs for (thanks!). Then, for no real good reason at all, I decided to rewrite the whole build system for the book in Dart. The build script I wrote for my first book was dead simple. Literally a single Python script that took a Markdown file for each book chapter and rendered it to HTML while weaving in the code snippets. The world’s dumbest static site generator. I started with that for Crafting Interpreters but then it grew and grew. My second book includes every single line of code for two complete interpreters, which it builds up incrementally across thirty chapters. I needed to not just build the HTML for the book’s website, but also make sure the code really did work. I gave the build system the ability to not only generate the site for the book, but also to slice and dice the code. Given a chapter, or even a single point within a chapter, it can output a program containing all of the code for the interpreters up to that point. Then I can take that code, compile it, and run it through my automated test suite to make sure the code I’m showing you does what it’s supposed to. Useful, but really straining the limits of how much code I want to maintain in a dynamically typed language like Python, at least, with my (low) level of Python expertise. Also, it was, frankly, really slow. So over a period of a couple of weeks, I rewrote the whole thing in Dart. I work at Google on the Dart language team as my day job, so picking Dart was not an unbiased choice. But it’s my build system for my book, and I know Dart and many of its core libraries and packages like the back of my hand. Hell, I personally wrote the initial version of the canonical markdown package. I found a pretty nice package for mustache templates, so I converted the book’s old Liquid templates to that. I didn’t find a good syntax highlighter. But it’s not like I looked very hard either. It seemed like just the kind of fun thing to implement from scratch, so I whipped one up loosely based on Pygments. The end result is a new build system that generates exactly the HTML and syntax-highlighted code that I want. Also, it is literally ten times faster than the old Python one. As you’ll see, it turned out to be handy that I had better control over the Markdown processing, but at the time I was basically just doing this for fun and to procrastinate the real work. I planned to implement some clever incremental rebuild logic in the dev server, but it builds so fast that I just rebuild everything on every refresh. Once I had the new build system outputting nice clean HTML and I’d deleted the old Python stuff, it was time to get started. Designing the book#designing-the-book Doing a large graphic design project like a book works a lot like web dev or game programming where it sort of has two levels. First you set up your “framework” or “engine”. On the web, this is your web framework and all of your CSS and HTML templates. In games, it’s your game engine. Then you pour content into that structure. With the right framework, adding content is easy. For graphic design using a program like Adobe InDesign, the way it works is you set up styles and masters. A master defines the margins and grid lines for a page. It’s the fences that corral all the text to keep those feral letters from running around eating all the whitespace. Styles are like CSS: they let you take a semantic kind of text or object and associate specific fonts, styles and colors for it. In theory, you get the masters and styles right and then typesetting is pretty easy and mechanical. Now, I did not make my life easy when it comes to book design. Book design is literally a two-dimensional spatial exercise and I made my job harder both horizontally and vertically. If you’ve read any of it on the web, you know my book has: Prose, of course. So many words. A lot of asides that need to be right next to certain pieces of text, code or illustrations that they refer to. Some of these can get pretty long. Plenty of code. Also, each code snippet has a little location blurb next to it telling you where the code goes in the resulting program. How wide?#how-wide Horizontally, the main text column needs to be wide enough to fit the longest line of code. I can use shorter lines and wrap the snippets more, but that makes them harder to read. It also makes them vertically taller, which causes its own problem. Then I need room next to that for the asides, since they often remark directly on specific sentences. I can make those narrower too, of course. But then they wrap and get taller. Some asides are fairly long and if they get too tall, they start colliding with each other or overlapping location snippets. Oh, and since I ended up writing a 200k+ word book, it’s going to have a high page count. That means a thick book. Thick books need wider inner margins so the text doesn’t disappear into the spine. All of this points towards a pretty wide page. Most CS textbooks—at least the ones on my bookshelf—are 7.5 inches wide. I tried hard to come up with a design that fit the code, asides, and healthy margins in that width while still giving a text size that didn’t require a magnifying glass. Eventually, I conceded defeat. Once I tried designing a set of metrics for an 8 inch wide page, everything fell into place. I could have enough breathing room around the text to make it enjoyable to read, a decent length for the code snippets, and plenty of room for the asides. (Using a narrower font for the asides helped too). Here are the final horizontal metrics. How tall?#how-tall That left the other dimension. If I were going with a real publisher doing a full offset print run, I could pick whatever page size I wanted. But since I’m self publishing, that would mean paying up front for thousands of copies to be printed and, I don’t know, storing all the boxes in my garage or something. Print on demand worked great for my last book, and I planned to do the same thing for this one. That meant sticking to the limited set of page sizes that KDP and IngramSpark support. The only reasonable one that is 8 inches wide is 8”×10”, so that’s what I picked. The end result is a book that feels big, but hopefully not awkwardly huge. I’m sorry. I promise to write a smaller book if I ever write another. The metrics for a complete chapter spread. Vertically, text is aligned to a classic 12pt baseline grid. The whole time I was picking these margins and metrics, I was also selecting fonts and building styles. You can’t do graphic design in a vacuum, so I picked a test chapter and just typeset it and tweaked the fonts over and over again. Eventually I got a set of fonts and styles I liked, some masters that seemed workable, and I was ready to go. The framework was done and now it was time to open the content hose. XML, like its 1999#xml-like-its-1999 Of course, problem #1 is that there is no content hose. I have to build one. InDesign doesn’t know what the hell Markdown or my weird ass build system is. I sure as hell didn’t want to, like, copy every chapter from my browser, paste it into InDesign and then manually apply all the styles. I have a high pain tolerance, but I’m not a masochist. What InDesign can do is import XML. Even better, you can set it up to automatically apply certain paragraph or character styles to certain tag names. InDesign’s XML support, alas, has not matured much since the time I described it as being implemented by a narcoleptic intern. For example, in HTML you can italicize a word in a header by taking an italics tag and nesting it inside a header tag. InDesign cannot comprehend such advanced data modeling. It needs a flat series of unnested tags and if you need italics in your header, you damn well better have a unique tag for it. But now I had an ace up my sleeve. Since I had microscopic control over my build system and its Markdown processing, I could write my own custom XML exporter that generated exactly the tags that would make InDesign not cry and avoid as many InDesign XML import bugs as possible. JavaScript, in my InDesign?#javascript-in-my-indesign Even so, XML import only gets you so far. Specifically, it gets you a “story” in InDesign terms: a single continuous narrative of text that fills the main text box and spans multiple pages: What the initial XML import looks like. The pink text is where an illustration goes and the "@" tells me where to anchor the aside. The main story is where the prose and code snippets go, but the asides and the location markers need to be pulled out float off to the side. With my last book, I yanked those out manually. I literally cut each aside from the main text column and pasted it into a new text box. It took forever, and that book was less than half as long as this one with way fewer code snippets. There are 1,133 code snippets in Crafting Interpreters. If I had to manually cut and paste the location markers for each of those, I would lose my mind. Worse, I would make a lot of mistakes, which would just create more work for myself. I’m an engineer so I am morally opposed to error-prone manual effort that can be automated. Did you know that InDesign can be scripted using JavaScript? Well, you do now, which apparently places you in a tiny minority because there are, like, no docs for it out there. You can find a couple of auto-generated references, a few sad cries for help from graphic designers clearly out of their element with no responses, and that’s it. JavaScripting InDesign is a special kind of pain. There is no debugger. There are no stack traces. There aren’t even debug prints. There is literally just alert(), and you can only call it once. And it halts your script. Fortunately, I actually learned JavaScript back when that’s all browsers gave you, so I can hack it. Just like developing for IE6. I managed to cobble together a horrific script that would find all of the asides and location markers, pull them out of the main text flow, and leave them off to the side: Asides and location markers are pulled out of the main flow and into their own text boxes. What I wasn’t able to get the script to do was position the boxes correctly. But InDesign has a thing called “anchors” where you can lock the position of an element relative to another. A couple of carefully crafted Object Styles would even set the horizontal metrics correctly and align the text to the right baseline. All I had to do was anchor each text box and it worked perfectly! Wait, did I say “perfectly”? I meant it worked right about half the time and the other half the time InDesign would inexplicably turn off the fucking borders of nearby code snippets. What it looks like when InDesign hates you. I lost hours of my life to this stupid bug. Eventually, I realized that some completely random subset of location tags would have to be manually positioned because anchoring invariably broke some borders. Editing, again#editing-again All of the above took me a good month or so and then it was time to stop fiddling around and get to work on the actual content. The first thing I did was another editing pass of the entire book, front to back. I had already done three drafts of each chapter as I was writing them, but I wanted to do one more now that it was done so that I could get a better feel for continuity. It turns out I repeated the same dumb jokes a lot. I fixed (most of) those. This took five months. There’s nothing fun to say about this, it was just a chore. Copy editing#copy-editing Next I hired an actual professional copy editor, Kari Somerton, to go through and do the same thing. She was great. Most of the editing world uses Microsoft Word and “Track Changes” to handle the editing process. Like most software engineers, I live and breathe plaintext and Git. That way I can see diffs of the changes, and go back through history. I didn’t want to abandon my workflow so I asked Kari to ramp up on Git and my weird completely bespoke build system. She handled it with aplomb and churned through the book in no time. She found hundreds and hundreds of mistakes. This despite me doing four drafts and readers filing hundreds of issues already. Professional copy editors are worth every penny. Typesetting the whole thing#typesetting-the-whole-thing Once the words were as good as they were gonna get, it was time to get them onto pages. The process went like this: Create a new InDesign file for the next chapter. Export it to XML. Import the XML into InDesign. Run my little JavaScript script to pull out the asides and location markers. Go through and anchor the side bar stuff. Fix up whitespace at the end of pages. The first five steps are a piece of cake. I’d wake up in the morning, brew a cup of coffee, shamble upstairs to the iMac, and get started. I could grind through those steps while half awake and get a chapter done in half an hour or so. It was peaceful. Almost meditative. And then step six. You see, that right there is the hard thing about typesetting a book. And it’s the really hard thing about typesetting this book. Because it turns out there are a bunch of constraints on how content can be fit vertically in a page. Obviously, we can’t slice an illustration in half and put the top half on one page and the bottom half on the next. The asides also really need to fit on one page too, or it gets confusing to keep track of what they refer to. Whenever possible, it’s nice for the code snippets to not get split across pages too. Some of those can be over a dozen lines long. (This is another reason why wider horizontal metrics helped. Because if I had made the code snippets narrower, they would end up taller, which would make them harder to fit on the page.) And you don’t want a header alone at the end of a page with no content after it. And it’s good to avoid widows and orphans… Take all of those rules and restrictions, and mix in the completely fixed height of a page and you got yourself a real constraint solving problem. Or, in my case, 640 of them, all interwoven with each other. Because, you see, InDesign is happy to solve all of this for you by just pushing content to later pages. Code snippet too long? Move it to the next page. No room for prose under the header? Move it all to the next page. What that gives you is a ton of dead white space at the bottom of pages. It looks terrible and wastes space, like this: I thoughtfully left room on the page for your own doodles. Speaking of illustrations…#speaking-of-illustrations At one level, the illustrations were easy. I specifically chose black and white pen and ink because it’s print friendly. When I first scanned the images as I wrote each chapter, I brought them in at glorious 1200 DPI. Here’s a crop: A little level adjustment in Photoshop produces: So detailed! Exporting these to high resolution bitmaps that print well was a snap, and they look great. (Well, as great as my handwriting looks, I guess.) Incorporating the illustrations into the page layout was another story. Halfway through typesetting, a lightbulb went off and I finally realized why most books say, “Refer to Figure 123 to see blah blah blah…” That gives the typesetter freedom to put Figure 123 wherever the hell it fits on any nearby page. In my dumb book, because I am an idiot, the prose just refers directly to the illustration. The illustration needs to be right there or the text doesn’t make sense. I didn’t think about that when each chapter was an infinitely scrolling web page, and by the time I realized, it was too late. With the hundreds of illustrations and thousand-plus code snippets, I had given myself thirty giant interrelated bin-packing exercises. The hard part of typesetting was figuring out how to adjust things to minimize that dead space. Sometimes I’d split a code snippet in two. Maybe add a little extra padding around one image to spread stuff across the page a little. Or crowd another one so that it just fits on the page. Sometimes I’d tweak an illustration to make it shorter to fit on a page or taller to eat up some whitespace. This was the real challenge of typesetting the book and why it took me two months to get through all the chapters. Front matter and back matter#front-matter-and-back-matter Did you know that there are professional indexers? People whose job it is to write indexes for books? They even write books about how to write indexes. (One would presume that these books have truly superb indexes.) I did not hire one of those eminently skilled professionals. Instead, I spent two weeks going through every damn chapter again doing my best to pretend that I know what I’m doing. InDesign’s support for indexes is actually pretty nice. You can basically just select some text and say, “Make an index entry for this.” Then it collects all of those and generates an index for the whole book. But actually adding all of those entries is a mind-numbing chore. The index is the main piece of back matter—the stuff at the end of a book after its main content. There is also front matter. You’ll never guess where that goes. I put together a title page, copyright page, dedication, and acknowledgements. Then I let InDesign generate a table of contents for me. This was a magical moment. At this point, I had a complete book: Here's what the inside looks like. Or, at least, I had the inside of one. Cover design#cover-design A lot of authors obsess over their cover and envision it the whole time they are writing. Despite what the aphorism says, people do judge books by their cover and a good one makes a big difference. At least, that’s the case over in fiction land. In computer science, judging by the other books I have laying around, the artistic merit of the cover appears to be somewhat less critical. I guess when the prof says you have to buy the book to pass the class, a clip art cover is sufficiently compelling. Since I am not a professor who can garner sales by fiat, I spent a lot of time on the cover design. I take photos, so I thought it could look nice to put something detailed on the cover to liven it up. I went through my thousands of photos trying to find something that fit. And, while I have some pretty pictures, none of them felt like they worked as covers. They felt arbitrary. Eventually I realized that the visual language of the book is those pen and ink illustrations. So I drew a bigger more detailed version of the mountain illustration I use as a metaphor for the compilation process. I also hand-lettered a new title: It’s a real typeface (Acumin Pro Extra Condensed) but I hand-traced a printout to give it some imperfect charm. I picked a color palette to try to give it a sort of a mimeographed 1950s scouting manual vibe. Proofreading the proof#proofreading-the-proof Now I really had a book. I uploaded the PDF exports to KDP and ordered a proof copy. A week later, a surprisingly heavy box arrived. This was the first moment I really understood just how big this book I wrote is. Up until this point, it was just data files. But seeing it fill up an Amazon box clarified the scale of the project in a way that the time I spent never quite did. So I had a book, but it still wasn’t done. Because the typesetting process involved a lot of manual labor. To err is human, so now I had to proofread—to literally go through the proof and read it looking for mistakes. I marked them all with sticky notes: I put an "x" on each sticky note to mark when I had applied the fix to the InDesign files. Here’s where it got stressful. If you’re a programmer, then source control and diff is deeply ingrained in your workflow. Whenever I make a change, I take for granted that I can then see a diff in the commit to verify that I changed only what I intended and nothing else. I did put the InDesign files in a Git repo, but they are giant opaque binary files. Also, InDesign has a habit of changing them even when it doesn’t seem like I’ve actually made any real changes. There’s nothing quite like syncing all the styles across the chapters, seeing every single file marked changed and wondering, “Did I just accidentally move every bullet list item 3 points to the left?” It felt like flying blind at exactly the stage in the process where I really wanted to see exactly what was happening. Did I engineer my way out of this problem? You’re damn right I did. I wrote a Dart script that would take a PDF of the book, extract every page, and then generate a single huge PNG file with every page tiled across it. It looks like this: If you zoom in, each page is about this big: If you're on a retina display, this is a 100% zoom. Every time I changed the InDesign files and committed them, I exported a PDF for that commit and generated a tile image. Then I wrote a little Photoshop action that would take two of those and draw a big red border around any pixels that differed. Here is what all of the proofreading changes look like: And zoomed in: There isn’t enough detail in the tile image to tell exactly what changed, but the red tells me which pages need a visual inspection. I know I should be proud of, like, writing an entire textbook on programming languages. But, honestly, I think I’m most proud of this dumb little script. It was such a relief to be able to programmatically verify that, yes, this PDF looks exactly like the last one except for the one expected change. Ebooks are also books#ebooks-are-also-books Once I incorporated all the proofreading fixes, the print edition was really for real totally done. But people also like reading ebooks and Kindle, so I needed to make those too. Again, writing my own build system helped. I tweaked it to be able to export the antiquated XHTML that EPUB requires as well as all the weird metadata and manifest stuff that goes into one. A few command-line invocations later and I had Kindle and EPUB e-books. I tested in a bunch of readers and tweaked the CCS to try to find a compromise between all of their renderers. Updating the site and launching#updating-the-site-and-launching At this point—and now we’re talking this past weekend as I write this sentence—I had a folder with all of the final files for every edition of the book. All that remained was to update the front page of the book’s website to point to where people can get them. I took some photos and put on my web designer hat for a while. I tried to make it tolerably responsive. Then I wrote this blog post. It’s weird to put that in the past tense since I just wrote it. But I guess it’s in the past now. Tomorrow, I’m going to upload the files to the various sites and stores. I’ll update the site to have links to all of those as they go live. Once the stores have processed the uploads and everything is available, I’ll publish this post and update the site. I’ll write a note for the mailing list and feel nervous emailing that many people. If you’re reading this, I already did all that and the book is really, really done. What next?#what-next Ever since I finished the last chapter, people have been asking me what I’m going to do next. I have people I consider close friends now who have never known me when I wasn’t writing this book. Many ask what I’m going to write next, or suggest a topic. I interpret this as a compliment—they like my writing and want more. But it also feels like asking a mom in labor whether she plans to have more kids. I’ve had my legs in the stirrups for six years, so I’m gonna just relax a bit after all the pushing is over before I even think about another book baby. What I really plan to do is… not plan to do anything. This writing project has been an enormous exercise in self-applied delayed gratification. On top of that, the pandemic brought its own basket of deferred dreams. I’ve gotten so skilled at denying myself and postponing, so good at tuning out what I feel like doing, that it’s hard to even hear those feelings any more. I don’t remember what my own joy sounds like. So I’m gonna check out for a while and go hunting for my pleasure centers. Maybe make some more music. Maybe go fishing. Probably spend more time with friends and family (safely). I might get back to working on my roguelike. Or maybe I won’t do any of those. Maybe I’ll just go out in the backyard and bask mindlessly in the sun like a lizard. The important part is I won’t decide until I feel like it. I’m sure eventually I’ll get the itch to work on something more sizeable again. (But, God-willing, I will never spend six years on a single project again in my life.) Until then, I hope you enjoy the book. Despite all the work, I had a lot of fun making it, and it was always a joy to hear from readers who liked it. It’s the only thing that kept me going this long and enabled me to get it done.

• Crafting “Crafting Interpreters”
  Apr 05, 2020

  It took three years and 200,000 words more than I expected, but my second book, Crafting Interpreters, is complete. I finished the third draft of the last chapter today, marking the last of around 1,400 days of continuous writing. This book was much harder than my first book, along every axis. It’s larger, more technically complex, much more deeply intertwined, and it had the misfortune of aligning with a really difficult period in my life. Today feels less like coasting past the finish line at the Tour de France, arms raised in triumph, and more like dragging myself onto the beach, clutching sand in relief after a storm-thrashed ordeal at sea. Before I get into all that, I have a minor confession to make. When I finished my first book, I wrote a long post about how I cobbled together enough willpower to reach the end of the last page. Everything in there is true, but there is one fact I superstitiously omitted. Halfway through writing Game Programming Patterns, I discovered a new passion: programming languages. It had been a long time since a topic ignited my brain to the same degree, and I was on fire. I spent basically every free hour (and many not-so-free hours—sorry, family) designing and hacking on programming languages. I read every book I could get my hands on, went to conferences, blogged, I even dreamed about programming languages. This infatuation was the main reason I stopped working on my first book for two years. I have a personality quirk where when I’m excited about something I just have to teach it to other people. Hermione Granger, arm waving feverishly to get the teacher’s attention, is my spirit animal. It was inevitable that I would write something about interpreters. But I couldn’t just drop one half-finished book to start another. I have gigs of unfinished projects laying around, but—maybe because the completed chapters were already online—I couldn’t bear to abandon Game Programming Patterns. So I made a promise to myself. If I finished that book, then I would let myself write a second book on interpreters. In part because of that promise, I did manage to complete the chapters, and then the print and e-book editions. What I thought was merely a hobby and personal goal turned out to be a life-changing experience. My little self-published vanity project has hundreds of five-star reviews, and has been translated to Korean, Japanese, Chinese, German, and Polish. The book did so much better than I expected that I’m still not sure how to process it, beyond feeling immense gratitude to everyone who read it, bought a copy, or cheered me on. The seed of a book#the-seed-of-a-book Once I finished the print edition of Game Programming Patterns, I took some time off. But it didn’t take too long for that itch to write about interpreters to come back. I knew exactly what I was getting into with writing a book now, how hard the grind can be. At first, I just noodled around. I wasn’t committed to doing anything. It was more a sort of recreational intellectual exercise. If I were to do a book, what would it look like? You know, hypothetically speaking. The very first note I wrote to myself said: high-level goal: a *small* book that builds a complete, efficient interpreter. instead of a wide text about programming language*s*, it is a single path through the language space. aim for 60k words. My first book was about 90,000 words, and I didn’t want to hike a trail that long again. I also had a meta-goal to make programming languages more approachable, and I figured a short text would help. I had this vision of something you could literally hold in your hand or have open next to your laptop while you followed along. To make a small book, I needed a small language and a small implementation. One of my other side projects was a scripting language named Wren. Wren is written in C, with a simple single-pass bytecode compiler inspired by Lua. Building Wren taught me how much functionality you can pack into a few thousand lines of clean C code. For this hypothetical book, I figured a bytecode VM in C like that would be a great fit. It would also give me the chance to cover a bunch of really fun topics like stack-based VMs, object representation, and garbage collection. But Wren wasn’t the right language. I like Wren (obviously), but it has some design quirks that I think make it a better language for users but maybe not for teaching. For the book, I wanted a dynamically-typed scripting language in the vein of languages like JavaScript, Python, and Lua. I started tinkering on a new toy language, tentatively named “Vox”. The goal was to keep things as simple as possible without taking any shortcuts around the hard problems in implementing a language. I wanted a rich expression and statement syntax to cover parsing. First-class functions and closures because they are powerful and challenging to implement efficiently. Classes and methods because that paradigm is so prevalent but omitted by many compiler books. At some point, I realized that dropping readers straight into C was too unfriendly of an introduction. It’s hard to teach high-level concepts like parsing and name resolution while also tracking pointers and managing memory. OK, so we’ll build two interpreters. First, a simple one in a high-level language to focus on concepts. Then a second bytecode VM in C to focus on performance and low-level implementation techniques. Somehow, I didn’t notice that maybe this “handbook” wasn’t going to be as pocket-sized as I hoped. My first choice for the high-level implementation language was JavaScript. I implemented most of a Vox interpreter in JS, but never really liked it. I wanted to write the interpreter in an object-oriented style because there are techniques like the Visitor pattern for doing language stuff in OOP that aren’t covered well elsewhere. Doing OOP in JS means deciding whether to use classes or a prototypal style. The former is cleaner but infuriates some segment of readers. The latter is verbose and confusing to those not already steeped in prototypes. Also, I missed static types. People reading code in a book don’t get the luxury of seeing the code in a debugger where they can see what values are in various variables. Static type annotations in the code help. So I switched to Java. I don’t love Java but it seemed like the least biased choice for a statically typed object-oriented language. I found you can tame a lot of its infamous verbosity by simply not programming in 1990s enterprise Java style. Maybe it’s not idiomatic to have public fields, but it’s a hell of a lot shorter. In parallel, I started building the bytecode VM in C, porting over bits of Wren’s implementation and stripping out the Wren-specific stuff. I spent the spring and summer of 2016 circling between these three pieces—the design of Vox itself, the Java interpreter, and the C bytecode VM. This was a delightful, satisfying period of time. The three parts played off each other in challenging ways. Sometimes I would change the language to make one interpreter simpler, but find doing so made the other interpreter more complex. Other times I’d hit on some trick that made everything get smaller and cleaner. Getting back on the horse#getting-back-on-the-horse I remember the exact moment I committed to writing the book. I was stuck on a tricky language design problem: constructor syntax. I knew I wanted classes, which meant some way to construct instances. Adding a new keyword felt too special-purpose for my minimal language. I like Smalltalk and Ruby’s approach of making new be a method on the class object itself, but that requires metaclasses and a lot of other machinery. I was struggling to find a way to add instantiation without making the language much bigger. Then I remembered JavaScript’s thing where you can simply invoke a “class” as if it were a function to create new instances. That has all sorts of weird baggage in JavaScript because everything does in JS, but the concept and syntax were perfect. I already had first-class classes. And I already had closures which meant a function call syntax that could be applied to arbitrary expressions. So “constructors” just became what you got when you invoked a class. I felt like Vox had gelled, like it was a language now. And my two implementations were coming along well too. I was surprised by how few hacks or ugly corners I ran into. The codebases kind of fell together and the more I tweaked them, the nicer they got. It felt more like I had discovered them than that I had created them. It would be a shame to not write the book and put them out there into the world. They wanted me to. I committed to writing the book, and I restarted my rule of writing every single day. I had a few thousand lines of pretty Java and C code, but how do I turn that into a book that can be read in linear order? Compact codebases tend to be highly intertwined with many cyclic dependencies. I didn’t want readers to have to slog through ten chapters before they could even run main(). This was the real technical challenge of writing the book—how do I take two implementations of the same language, and break them into incremental pieces that I can build up a chapter at a time? I made this problem harder for myself because of the meta-goal I had. One reason I didn’t get into languages until later in my career was because I was intimidated by the reputation compilers have as being only for hardcore computer science wizard types. I’m a college dropout, so I felt I wasn’t smart enough, or at least wasn’t educated enough to hack it. Eventually I discovered that those barriers existed only in my mind and that anyone can learn it. My main overarching goal of the book is to pass on that feeling, to get readers to understand there’s no magic in there and nothing keeping them out. To nail that conceit, I wanted to include every single line of code used by the interpreters in the book. No parser generators, nothing left as an exercise for the reader. If you type in all of the code in the book, you get two complete, working interpreters. No tricks. So not only did I need to break these two interpreters into chapters, I needed to do it without any cheating. I wanted a hard guarantee that at the end of each chapter, you had a program that you could type in, compile, run, and do something with. I knew I wouldn’t be able to verify this manually, so it was time to create some tools. A bespoke build system#a-bespoke-build-system I wrote my first book in Markdown. I slapped together a tiny Python script that converts the Markdown to HTML and transcludes the code snippets which are stored in separate C++ files. When I started my second book, I took that script and started growing it. It evolved throughout writing the book, but in the end, here is how it works. All of the code for the interpreters are stored in separate source files. I have a Java project that contains the complete Java interpreter that you get by the end of that part of the book. Likewise, there’s a C project for the bytecode VM. I can edit and build those in an IDE, run tests, debug them, etc. They’re real programs. Meanwhile, the text of the book is authored in Markdown, one file per chapter, just like my first book. To include a snippet of code in the book, I put a tag in the Markdown like this: Which can be any of: ^code is-alpha Once we've found an identifier, we scan the rest of it using: Here, the ^code line says “look up the snippet named ‘is-alpha’ and insert it here.” When the build script generates the HTML for this chapter, it goes off and hunts through the code for that snippet. Over in the code, special comments delimit snippets. The one included here looks like this: //> Scanning on Demand is-alpha static bool isAlpha(char c) { return (c >= 'a' && c 'z') || (c >= 'A' && c 'Z') || c == '_'; } // The //> line begins the snippet and says what chapter the snippet appears in and the name of the snippet. The // line ends the snippet. Pretty straightforward. This let me build the book, but didn’t ensure that the thing I built actually worked. So I wrote a separate script that instead of building the book, builds programs. For each chapter, it collects all of the snippets that appear in that chapter and the previous ones and writes them out to separate source files. In other words, it produces a separate interpreter, one for each chapter, containing only the code that readers have seen so far. I put together a Makefile to build those per-chapter versions of each interpreter to make sure they compiled. Of course, compiling successfully doesn’t mean they do anything useful. Writing a single correct interpreter is hard. Writing thirty of them—there are thirty chapters in the book—is much harder. I had already harvested a little test runner from Wren and ported most of Wren’s tests over to be Lox tests. (I changed the name of the language in the book since there was already a language out there named “Vox.”). I took that test runner and extended it to be able to run the tests on each chapter’s version of the interpreters. Of course, the tests don’t all pass—the interpreters aren’t complete! So I added metadata to track which tests I expected to pass by which point in the book. With this in place, I could automatically verify that the code that I was showing readers did exactly what I expected. More complex snippets#more-complex-snippets The snippet markers look pretty straightforward, and in many cases they are. But reality tends to get messier and I didn’t allow myself to sweep any of that mess under nearby rugs. Some changes don’t just add code to the interpreter. I try to minimize it, but often you need to replace some existing code. A few lines of code may appear in chapter 5 and then later get superseded in chapter 9 by something more powerful. Obviously, I can’t jam both of those snippets into the same source file and expect it to compile. Remember, the source files that I hand author are themselves valid Java and C programs that I can build and run. If a function contained several versions of its body mixed together, odds are slim that the compiler will like what it sees. So, for any piece of code that later gets replaced—in other words code that is not part of the very final version of each interpreter—there is a different snippet syntax: static void concatenate() { /* Strings concatenate //> Garbage Collection concatenate-peek ObjString* b = AS_STRING(peek(0)); ObjString* a = AS_STRING(peek(1)); // int length = a->length + b->length; char* chars = ALLOCATE(char, length + 1); memcpy(chars, a->chars, a->length); memcpy(chars + a->length, b->chars, b->length); chars[length] = '\0'; } This block comment contains a snippet of code. The header indicates that this snippet is named “concatenate” and first appears in the “Strings” chapter. Then, later, it gets removed when the “concatenate-peek” snippet in the “Garbage Collection” chapter appears. In other words, that latter snippet replaces the previous two lines. By storing the code for this snippet inside a block comment, I ensure that the code as it is in the raw source file is still valid. In some places where the interpreter gets revised multiple times, the code can get pretty complex. Here is the main() function of the bytecode VM: int main(int argc, const char* argv[]) { //> A Virtual Machine main-init-vm initVM(); // /* Chunks of Bytecode main-chunk /* Chunks of Bytecode main-constant /* Chunks of Bytecode main-constant /* Chunks of Bytecode main-chunk-line /* A Virtual Machine main-chunk /* A Virtual Machine main-negate /* Chunks of Bytecode main-chunk /* Chunks of Bytecode main-chunk-line /* Chunks of Bytecode main-disassemble-chunk /* A Virtual Machine main-interpret //> Scanning on Demand args if (argc == 1) { repl(); } else if (argc == 2) { runFile(argv[1]); } else { fprintf(stderr, "Usage: clox [path]\n"); exit(64); } freeVM(); // /* A Virtual Machine main-free-vm /* Chunks of Bytecode main-chunk return 0; } Maintaining this is not super fun. But, thankfully, I have a build and test system to tell me when I break something. Slicing up the interpreters#slicing-up-the-interpreters So I had a tool that could let me split the interpreters across the chapters. If it was possible to break these interpreters into chapters at all, it would let me do so. Now I just had to figure out where to carve the seams. This was the most technically challenging part of the book writing process. I wasn’t sure if it was going to work at all. I spent several weeks sketching out potential lists of chapters, sprinkling snippet markers throughout the code, and seeing if the result built. I’d get a compile error because a snippet in an early chapter tried to call a function in some later chapter and I would have to go back and reorganize things. I hand-drew dependency graphs between language features and tried to untangle them. Here’s an example of how this process unfolded: To teach functions I want show that recursion works. But to have recursive functions I need control flow. Otherwise, every recursive function recurses infinitely without a base case. So control flow has to come before functions. For control flow, I need side effects so that I can show that a certain code path is not taken. The obvious way to do side effects is to have a print() function that displays output. But I don’t have functions yet. That’s a cycle. Crap. Sometimes I had to change the language itself to break cycles. The above example is why Lox has a built in print statement instead of a print function. Because that way we can introduce the print statement before control flow, which is in turn before functions. I had to break a couple of cycles like that but, eventually, to my surprise, I got it all sorted out. I had a complete list of chapters for both interpreters. Every line of code was sorted into a snippet that belonged to one of those chapters. I could build and run each chapter’s code. Best of all, each chapter had a reasonably coherent concept and a roughly similar amount of code. Before, I felt like I had a language and code that wanted to get out there into the world. Now I felt like I had a book. Or, at least, I had all of the code for a book. A chapter at a time#a-chapter-at-a-time I wrote my first book one chapter at a time. I drafted, edited, illustrated each chapter and put it online before moving to the next one. Serial publishing for the digital age. I really loved that process. It helped build an audience for the book and gave me incremental feedback which made the book better and kept me going. I don’t think I could write a whole book in the dark. I intended to publish this book the same way, but the deeply interconnected nature of the chapters made that much harder. I didn’t want to discover a problem with the code in chapter 28 that forced me to tweak things in an earlier chapter that readers had already read. I didn’t want to paint myself into a corner or invalidate any previously-published material. So the entire time I was designing the language, coding the interpreters, and splitting the codebases into chapters, I had not done any actual writing. I didn’t want to put down any prose until I knew the code was solid. So I spent the summer of 2016 just hacking on code. It was, honestly, a blast. The programming part is definitely the fun part, and it was a joy to tinker on the code and figure out how to break it into chapters. Sort of like making a jigsaw puzzle and solving it at the same time. After a few months, it was all there. Every single line of code for the entire book. A complete list of chapters. And I hadn’t written a single word of prose. In theory, “all” that remained was writing some text to explain the code I had already written along with some pictures. But, for me at least, English is a much more taxing language to write than C or Java. I had all of the difficult work ahead of me, and all of the fun was done. Illustrating by hand#illustrating-by-hand Well, not all of the fun. I did still have the illustrations to do. With my last book, I hand-drew little sketchy diagrams to show various bits of architecture. I wanted even more illustrations for this book to make the concepts less abstract, less opaque. Unlike a videogame, you can’t see a garbage collector doing its thing. Visual metaphors really help. I liked the hand-drawn look. It furthered my meta-goal of making the material more approachable, more human. But I wanted to up the quality. I wanted them to be more intricate and contain more information. I wanted the drawings to be more detailed. Less like margin doodles and more like, well, illustrations. Maybe even some lowercase letters. The ultimate goal for me is a print book, so I stuck with black and white ink. I wanted a tighter, more “spidery” style, so I got some technical pens. People often ask me what programs I used for the illustrations, assuming I did them all digitally. Here are the main tools I used: I went with Pigma Microns in 01 and 005. If I were doing it again, I think I'd do Faber-Castell Pitt pens. There are two kinds of illustrations in the books: diagram-like ones that show meaningful information, and drawings that are for metaphors or just to be silly jokes. The process is different for each. I draw each diagram in pencil on graph paper. That lets me erase and move things around until I get it where I like: All of the vertical and horizontal lines in the illustrations generally fall on the graph paper rules or halfway between them. Then I tape a piece of tracing paper on top and draw over it in ink: I make mistakes sometimes, usually when lettering like "upvaluels" here. I fix that in Photoshop after scanning. I hand letter everything. It takes a long time. I used to do graphic design, and I have this weird tic where any time I see something that looks handwritten, I look for multiple instances of the same letter to see if they are different or if the design just used a handwriting font. It’s almost always a handwriting font and I die a little inside to see the illusion evaporate. Well, this is my damned book and no reader will ever feel that disappointment. Every single fucking letter in every one of the illustrations was hand lettered and is unique. Here is the hand-lettered logotype for the book. Each "R" is different! Also, if that’s not obsessive enough, I spent time changing my own handwriting to better match the text font of the book. I taught myself to write double-story “a” and “g” letters and practiced by filling pages of paper with the same letter over and over. Look at the loop under the "g" in "filling" and the finial on the "a" in "apples". I also wanted to make sure that the illustrations and text matched each other across the book. To give the text a consistent size, I printed a little height guide: The dotted line indicates the x-height. I picked a ratio for that to match the fonts I use for text and code. I slid this paper under the tracing paper and lettered on top of those lines to keep the metrics the same across the book. To keep the diagram size and line thickness consistent, each illustration has a pair of registration marks a fixed distance apart: The little marks that the pencils are pointing at. I scan each illustration into Photoshop for clean up and processing. I use those marks when cropping to ensure that the image maintains the right size relative to other images. I recorded a video of the whole process if you want to see it in action: Even in timelapse, it takes a long time. Writing this all out makes me sound like a crazy person. What the hell am I doing with my life? Or, more importantly, what could I have been doing instead of doing all that? Too late now, I guess. The picture-like drawings have a different workflow since they don’t have a lot of straight lines or align to a grid. How will readers understand what a stack is without this helpful illustration? I draw those on regular sketch paper using a non-photo blue pencil. Then I ink on top of that. I scan the paper in RGB and use the blue channel, which mostly makes the blue pencil marks disappear. The sketch paper bleeds the ink more than I like but I didn't want to change paper partway through the book, so I stuck with it. It’s a lot of work for each image, and this doesn’t include all of the work after scanning it. And I wanted a lot of them. By the end, I had this stack of paper: Such a small image for so much work. I went through two full pads of tracing paper, two pads of graph paper, a sketch pad, and several pens. I drew 181 illustrations. Writing is suffering#writing-is-suffering I had the code, and I had a process for illustrations. The remaining work was just writing all the words and drawing all the pictures. So that’s what I did. I started at chapter one and started writing. For each chapter, I wrote an outline and then a first draft. I did an editing pass over that to fix all the major problems. Then a second pass where I read the whole chapter out loud to fix cadence and other stuff. This is the same process I used for the first book. I stumbled onto something that worked, so I wasn’t about to mess it up. I posted each chapter online, and then spent a day fixing bugs that readers noticed. Then I moved on to the next chapter. I wrote. And wrote. And wrote. Every single day. Every now and then I would have a trip or something where I couldn’t write. As with my first book, I would bank days by writing multiple sessions per day beforehand and then spend those banked days on days that I didn’t write. But for the most part, I wrote every day. In the blog post I wrote after my first book, I whined about how I had to write on days when I traveled for work, on holidays, when the kids had sniffles. At the time, it truly was one of the hardest things I’ve ever done. This time was something else entirely. I wrote the day my grandfather died (peacefully, unsurprisingly) and the day my aunt died (tragically, days after retiring). I wrote the day I found out my Mom had cancer and my children saw me cry for the first time. I was flying to Louisiana to keep my Mom company when I turned on my phone during the layover and discovered a dear friend had had a stroke. I wrote that evening. I woke up the next day and found out she had died. I wrote that morning sitting next to my brother in the waiting room of the hospital while my Mom got her PET scan. The morning of my friend’s memorial service, I wrote in the hotel. Later that day, I openly sobbed in front of a room full of people. The next day, my wife found out her aunt had terminal cancer. I wrote on the flight home. See that dog up there in my profile photo? That’s Ginny. She’s on the back cover of my first book. Her myriad health problems finally caught up with her last spring. People sometimes ask, “When did you know you were an adult?” For me it was the day I made the call to put my dog down. The hardest part was watching my kids say goodbye to her. I’m tearing up now writing about it. I ran my fingers through Ginny’s silky fur as the sedatives took her away. I only got through 59 words that afternoon. I wrote the day the US somehow elected a racist, abusive, corrupt demagogue, and every day afterwards as I saw my country and others turn towards hate and authoritarianism. I wrote while climate change and income inequality worsened. And now here I am writing at home on the same desk where I work now, quarantined like most of you all, hoping to survive the worst pandemic the world has seen in a century. This is not about how disciplined I was. Because during what have been some of the worst years of my life, a weird inversion happened. It’s not that I was going through that shit and still writing in addition to it. I had to keep writing. Writing was one thing I could still control in the face of many things I could not. If I could make it through the book, maybe I could make it through the other things too. If I had skipped a day it would have meant that the cancer or the deaths beat me that day, that they were stronger than me. I feared what it would mean to me to let go. I got through these four years and kept writing, but I paid a price. When I read the earlier chapters, they have a whimsy and light-heartedness that later chapters lack. We’re all going through dark times, and I don’t feel light. The past few years left a mark on me, and that mark shows up in the book. I miss the goofier person I used to be, sometimes. But I’d like to believe that maybe the person I am now is a little more honest. Maybe some of those jokes were a mask. And, thankfully, Mom is in remission. Psychological self examination aside, I did keep up the writing. Which is good because, man did I underestimate this book. I was aiming for 60,000 words and hoped to get it done in about a year. Here I am four years later sitting on a quarter of a million words. People sometimes ask what it’s like writing something that big. I’ve been asking myself that for the past couple of weeks. And the weird thing is, I don’t know. I’ve had my head down for the past four years and haven’t looked past the next paragraph or two the entire time. What does it feel like to write an email or draw a picture? Writing the book felt like that. I just happened to do it over and over again. I feel like a marathon runner who’s been watching his feet the whole time and didn’t even notice when he stumbled over the finish line. Now what?#now-what Crafting Interpreters is complete now. I had to stop here for a minute and look at that sentence. I’ve been working on this book every day for around 1,400 days. I can’t wait to take a break. So that’s the next step. My plan was to finish the book right before spring break and enjoy a week on the beach with family. That beach trip went the way of so many other plans in early 2020, but I still intend to take a long break. I don’t know if you noticed, but we all have a lot of other shit to deal with right now. I’m going to relax. Every morning since 2016, I’ve woken up with a task I had to do. Until I got my writing done for the day, it was on my mind, weighing me down. Writing left me drained. If you’ve ever had a newborn, you know the feeling of always having to carry the baby around. After a while, it’s like you forget what it’s like to have two free arms. I’ve been carrying this baby for four years, so I’m looking forward to having both arms for a while. Once I’m recharged, the real fun starts. Having the book online is important, but for me, Crafting Interpreters was always meant to be a book with pages and a cover. So after a long bout of editing and bug fixing, I’m going to get started doing the page layout for the print edition. I love graphic design, and I can’t wait to hold it in my hands. If you’d like to hold it in your hands when it comes out, I have a mailing list where I’ll let you know when the book is done. In the meantime, I think I’ve earned some rest.

Antirez

• In defense of linked lists
  Nov 04, 2022

  A few days ago, on Twitter (oh, dear Twitter: whatever happens I’ll be there as long as possible – if you care about people that put a lot of energy in creating it, think twice before leaving the platform). So, on Twitter, I was talking about a very bad implementation of linked lists written in Rust. From the tone of certain replies, I got the feeling that many people think linked lists are like a joke. A trivial data structure that is only good for coding interviews, otherwise totally useless. In a word: the bubble sort of data structures. I disagree, so I thought of writing this blog post full of all the things I love about linked lists. So, get ready to read a sentimental post about a data structure, and don't tell I didn't warn you. Linked lists are educational. When your teacher, or the page of a book, or anything that exposes you for the first time to linked lists shows you this little circle with an arrow pointing to another circle, something immense happens in your mind. Similar to what happens when you understand recursion for the first time. You get what data structures made of links truly are: the triviality of a single node that becomes a lot more powerful and complex once it references another one. Linked lists show the new programmer fundamental things about space and time in computation: how it is possible to add elements in a constant time, and how order is fundamentally costly, because if you want to insert an element “in place” you have to go from one node to the other. You immediately start thinking of ways to speed up the process (preparing you for the next things), and at the same time you understand, deeply, what O(1) and O(N) really mean. Linked lists are augmentable. Add a pointer to the previous element, and now it is possible to go both sides. Add “far” pointers from time to time, and you have a skip list with completely different properties. Change every node to hold multiple items and your linked list becomes unrolled, providing very different cache obviousness properties. Linked lists can be embedded, too. The Linux kernel, for instance, has macros to add a field to any structures in order to link them together. There is more: linked lists are composable. This is a bold property: you can split a linked list into two in O(1), and you can glue two linked lists in O(1) as well. If you make judicious use of this property, interesting things are possible. For instance, in Redis modules implementing threaded operations, the thread processing the slow request dealt with a fake client structure (this way there was no locking, no contention). When the threaded command finally ended its execution, the output buffer of the client could be glued together to the actual buffer of the real client. This was easy because the output buffer was represented with a linked list. Linked lists are useful: Redis can be wrong, but both Redis and the Linux kernel can’t. They are useful because they resemble certain natural processes: adding things in the order they arrive, or in the reverse order, is natural even in the physical world. Pulling items incrementally is useful too, as it is moving such items from head to tail, or moving them a position after the current one. Linked lists are simple. It is one of those rare data structures, together with binary trees and hash tables and a few more, that you can implement just from memory without likely stepping into big errors. Linked lists are conceptual. A node pointing to itself is the most self centered thing I can imagine in computing: an ideal representation of the more vulgar infinite loop. A node pointing to NULL is a metaphor of loneliness. A linked list with tail and head connected, a powerful symbol of a closed cycle. For all those reasons, I love linked lists, and I hope that you will, at least, start smiling at them. Comments

• Scrivendo Wohpe
  Jul 17, 2022

  (English translation of this post: http://antirez.com/news/136) Dopo due anni di lavoro, finalmente, Wohpe, il mio primo libro di fantascienza, ma anche il mio primo scritto di prosa di questa lunghezza, è uscito nelle librerie fisiche italiane, su Amazon, e negli altri store digitali. Lo trovate qui: https://www.amazon.it/Wohpe-Salvatore-Sanfilippo/dp/B09XT6J3WX Dicevo: il primo scritto di questa lunghezza. Ma posso considerarmi del tutto nuovo alla scrittura? Ho scritto per vent’anni in questo blog e in quelli passati che ho tenuto nel corso del tempo, e molto spesso ho usato Facebook per scrivere brevi racconti, frutto di fantasie o basati su fatti reali. Oltre a ciò, ho scritto di cose tecniche, specialmente riguardo la programmazione, per un tempo altrettanto lungo, e sono stato un lettore di racconti e di romanzi per tutto il corso della mia vita. E allora perché scrivere Wohpe è stato anche imparare a scrivere da zero? Nei primi mesi di scrittura del romanzo, ma anche prima, nei mesi precedenti, quando mi preparavo scrivendo lunghi racconti che poi avrei cestinato, mi è successo ciò che accade spesso a coloro che imparano a giocare a scacchi. Tanti seguono questo percorso: imparano le regole, e va bene, lo sappiamo che con quelle si fa poco; le regole permettono solo di muovere i pezzi in maniera legale. Ma poi, subito dopo, imparano dei rudimenti di tecnica e di strategia, magari studiando duramente per qualche settimana. Però quando sono alla scacchiera, se una mossa non è brutalmente peggiore o migliore di un’altra, tutte le mosse sembrano equivalenti. Il giocatore di scacchi poco abile, poco esperto, non ha un vero gusto per le mosse; non riesce a valutarle non solo per ciò che sono in termini assoluti, ma neanche secondo una sua propria idea. Il risultato è un gioco casuale. Solo più avanti, dopo molte ore di gioco, ella finalmente riuscirà a esprimere delle scelte che, a prescindere dal fatto siano esse giuste o sbagliate, hanno quantomeno una coerenza, sono state davvero pensate: voglio muovere il cavallo qui, per queste precise ragioni, e per tali ragioni preferisco questa mossa a tutte le altre possibili. Così chi scrive ed è agli inizi, se una frase è buona, con certezza non lo sa (e per continuare col paragone di sopra, così come il gioco dello scacchista sarà casuale, la sua scrittura sarà casuale). Sposta una virgola, cambia una parola. Suona bene o male? Ha delle idee che si è formato scrivendo a scuola e poi scrivendo da adulto, ma queste idee lo assistono poco quando l’ambizione è quella di scrivere una prosa di livello letterario. L’autore alle prime armi non ha un suo stile, perché prima di non saper scrivere non sa ancora leggere: quando legge un libro che adora, raramente capisce esattamente *cosa* accade nella pagina di così convincente, e così anche quando rilegge se stesso, non sa se ha scritto bene o male. Leggi, se vuoi imparare a scrivere! Dicono tutti. Peccato non sia vero: bisogna prima di tutto scrivere per imparare a scrivere, così come bisogna fare dei cortometraggi per imparare a fare il regista, e guardare gli altri film non sarà sufficiente (anche se sarà certamente utile). E per imparare a scrivere, il tipo di lettura che serve davvero è la rilettura di alcuni libri che abbiamo scelti come modelli; quello sì che è utile: la rilettura per comprenderne le forme, fino in fondo. Ora questo semplice fatto, di aver capito quale sia il mio stile, e di saper finalmente leggere e avere un giudizio che emerge immediatamente quando ho tra le mani un’opera, è già un risarcimento più che sufficiente dei due anni di sforzi di scrittura nei quali mi sono profuso. Ed è una vera fortuna che l’esperienza in sé sia stata di così grande valore, perché quello che molti nuovi autori non sanno è quanto violento possa essere il mercato editoriale mondiale, e quello italiano in particolare. In Italia un libro di fantascienza che ha un buon successo, edito da un piccolo o medio editore, vende 500 copie. Deve andare bene, per avere questi numeri. La gran parte dei libri vende meno di 100 copie. A noi informatici queste cifre fanno rabbrividire. Il più stupido programma che ho scritto ha avuto dieci volte più utenti. Mi spingerei a dire che il più stupido programma che ho scritto e che ho pubblicato con un minimo di energie, ha avuto dieci volte questi *lettori*, gente che ne hanno letto il codice sorgente per capire come funzionasse. Io sono stato, e per questo ringrazio non so bene chi, un programmatore di una certa notorietà, direte voi. Ciò che voglio dire va molto oltre la mia esperienza personale. Anche coloro che non sono conosciuti da nessuno e provano a fare, nella programmazione, una cosa mediamente interessante, appena ben descritta e documentata, e la mostrano un po’ in giro, ricevono un interesse enorme rispetto a quello che spetta agli autori di fiction. I motivi sono tanti e abbastanza ovvi, non vale neppure la pena di soffermarsi su di essi, dunque perché vi racconto queste cose? Per questo motivo: Sono poche le attività, oltre alla letteratura, dove c’è lo stesso mostruoso scompenso tra le forze necessarie per produrre un’opera e la scarsa risposta del pubblico. Chi decide di dedicare molto tempo alla scrittura, deve conoscere questo fatto da subito. Io per fortuna sapevo già tutto, grazie ai miei amici scrittori; però lo stesso certe sfumature di questa irrilevanza finiscono per essere difficili da accettare. E allora perché tutti scrivono? Sono straripanti le file di quelli che tentano la fortuna editoriale. Credo sia un meccanismo simile a quello che accade nell’IT, con tanti che provano a creare un nuovo linguaggio di programmazione: il fallimento è quasi certo, ma il tentativo stesso è una delle imprese più soddisfacenti alle quali dedicare le proprie migliori energie. Ora sono a un bivio: potrei scrivere altra prosa, rimettermi a scrivere codice, o provare a tenere vive le due attività allo stesso tempo. Cosa farò non lo so ancora. Per ora, vediamo cosa succede con Wohpe, sia con la versione italiana che con la traduzione in inglese, a cui in questo momento una capace traduttrice sta lavorando. E su questa cosa delle traduzioni, eseguite col supporto dell’autore, e di quale esperienza significativa sia dal punto di vista filologico, magari vi parlerò qualche altra volta (io e Bridget parliamo l’inglese e l’italiano, ma siamo madrelingua lei di una e io dell’altra lingua, e ciò è molto interessante quando si collabora tra traduttore e autore). Chiudo il post dicendo a chi mi legge: se ne avete voglia, scrivete prosa! Io ora lo so per certo: non è un caso che la scrittura sia stata per centinaia di anni considerata l’arte più alta nella quale cimentarsi. Scrivendo si cercano delle cose, e se si insiste abbastanza si finisce per trovarle davvero. Comments

• Writing Wohpe
  Jul 17, 2022

  (Traduzione italiana di questo post: http://antirez.com/news/137) [Sorry for the form of this post. For the first time I wrote a post in two languages: Italian and English. So I went for the unusual path of writing it in Italian to start, translating it with Google Translate, and later I just scanned it to fix the biggest issues. At this point GT is so good you can get away with this process.] After two years of work, finally, Wohpe, my first science fiction book, but also my first prose writing of this length, has been released in Italian physical bookstores, on Amazon, and in other digital stores. You can find it here: https://www.amazon.it/Wohpe-Salvatore-Sanfilippo/dp/B09XT6J3WX I was saying: the first writing of this length. But can I consider myself entirely new to writing? I have written for twenty years in this blog and in the past ones that I have kept over time, and very often I have used Facebook to write short stories, the result of fantasies or based on real facts. On top of that, I've been writing about technical stuff, especially programming, for an equally long time, and I've been a short story and novel reader my entire life. So why was writing Wohpe also learning how to write from scratch? In the first months of writing the novel, but also earlier, in the previous months, when I was preparing myself by writing long stories that I would then throw away, what happened to me often happens to those who learn to play chess. Many follow this path: they learn the rules, and that's okay, we know that you don't do much with them; the rules only allow you to move pieces legally. But then, soon after, they learn some rudiments of tactics and strategy, perhaps studying hard for a few weeks. But when they are on the board, if one move is not brutally worse or better than another, all the moves seem equivalent. The unskilled, inexperienced chess player has no real taste for moves; she fails to evaluate them not only for what they are in absolute terms, but not even according to his own idea. The result is a casual game. Only later, after many hours of play, will she finally be able to express choices that, regardless of whether they are right or wrong, have at least a coherence, they were really thought: I want to move the horse here, for these precise reasons, and for these reasons I prefer this move to all other possible ones. So whoever writes and is at the beginning, if a sentence is good, he certainly does not know it (and to continue with the above comparison, just as the chess player's game will be casual, his writing will be casual). Move a comma, change a word. Does it sound good or bad? He has ideas that he formed by writing at school and then writing as an adult, but these ideas help him little when the ambition is to write a literary level prose. The novice author does not have his own style, because before not knowing how to write he does not yet know how to read: when he reads a book he adores, he rarely understands exactly * what * happens on the page that is so convincing, and so also when he re-reads himself, he does not know if he wrote well or badly. Read, if you want to learn to write! They all say. Too bad it's not true: first of all you have to write, to learn how to write, just as you have to make short films to learn to be a director, and watching other films will not be enough (although it will certainly be useful). And to learn to write, the kind of reading that is really needed is the rereading of some books that we have chosen as models; what is useful: the rereading to fully understand its forms. Now this simple fact, of having understood what my style is, and of finally knowing how to read and have a judgment that immediately emerges when I have a work in my hands, is already more than enough compensation for the two years of writing efforts in which I have lavished. And it is fortunate that the experience itself was of such great value, because what many new authors do not know is how violent the world publishing market can be, and the Italian one in particular. In Italy a science fiction book that has a good success, published by a small or medium publisher, sells 500 copies. It has to be fine to have these numbers. Most books sell less than 100 copies. To us computer scientists these figures make us shiver. The stupidest program I wrote had ten times as many users. I would go as far as to say that the stupidest program I've written and published with a minimum of energy has had these * readers * ten times, people who have read its source code to understand how it works. I have been, and for this I thank I don't know who, a programmer of a certain notoriety, you might say. What I mean is far beyond my personal experience. Even those who are not known by anyone and try to do something on average interesting, just well described and documented, in programming, and show it around a bit, receive enormous interest compared to what belongs to the authors of fiction. The reasons are many and quite obvious, it is not even worth dwelling on them, so why am I telling you these things? For this reason: There are few activities, besides the literature, where there is the same monstrous imbalance between the forces needed to produce a work and the poor response of the public. Anyone who decides to devote a lot of time to writing must know this fact right away. Fortunately, I already knew everything, thanks to my writing friends; however, certain nuances of this irrelevance end up being difficult to accept. So why is everyone writing? The ranks of those who try their editorial fortune are overflowing. I think it's a similar mechanism to what happens in IT, with many trying to create a new programming language: failure is almost certain, but doing so is one of the most satisfying companies to devote your best energy to. Now I'm at a crossroads: I could write more prose, get back to coding, or try to keep the two activities alive at the same time. What I'll do I don't know yet. For now, let's see what happens with Wohpe, both with the Italian version and with the English translation, which a capable translator is working on right now. And on this matter of the translations, carried out with the support of the author, and what a significant experience it is from the philological point of view, maybe I will talk to you some other time (Bridget and I speak English and Italian, but we are native speakers of one and I of the other language, and this is very interesting when collaborating between translator and author). I close the post by saying to those who read me: if you feel like it, write prose! I now know for sure: it is no coincidence that for hundreds of years writing has been considered the highest art in which to try one's hand. By writing you look for things, and if you insist enough you end up really finding them. Comments

• Programming and Writing
  May 14, 2021

  One year ago I paused my programming life and started writing a novel, with the illusion that my new activity was deeply different than the previous one. A river of words later, written but more often rewritten, I’m pretty sure of the contrary: programming big systems and writing novels have many common traits and similar processes. The most obvious parallel between the two activities is that in both of them you write something. Code is not prose written in a natural language, yet it has a set of fixed rules (a grammar), certain forms that most programmers will understand as natural and others that, while formally correct, will sound hard to grasp. There is, however, a much deeper connection between the two activities: a good program and a good novel are both the sum of local and global elements that work well. Good code must be composed of well written and readable single statements, but overall the different parts of the program must be orthogonal, designed in a coherent way, and have clean interactions. A good novel must also succeed in the same two scales of the micro and the macro. Sentences must be well written, but the overall structure and relationship between the parts is also crucial. A less structural link between programming and writing is in the drive you need when approaching one or the other: to succeed you need to make progresses, and to make progresses you have to be consistent. There is extensive agreement on the fact that programs and novels don’t write themselves, yet. Twenty years of writing code helped me immensely with this aspect; I knew that things happen only if you sit every day and write: one day one hundred words, the other day two thousands, but rare is the day I don’t put words on the page. And if you have written code that is not just a “filler” for a bigger system, but a creation of your own, you know that writer block also happens in programming. The only difference is that for most people you are an engineer, hence, if you don’t work, you are lazy. The same laziness, in the case of an artist, will assume the shape of a fascinating part of the creative process. The differences. I believe the most sharp difference between writing and programming is that, once written, edited and finalized, a novel remains immutable, mostly. There are several cases of writers returning on their novels after several years, publishing a bug fixed version of it, but this is rare and, even when happens, a one-shot process. Code evolves over time, is targeted by an endless stream of changes, often performed by multiple people. This simple fact has profound effects on the two processes: programmers often believe that the first version of a system can be quite imperfect; after all there will be time to make improvements. On the other hand writers know they have a single bullet for every novel, to the point that writing prose is mostly the act of rewriting. Rewriting sentences, whole chapters, dialogues that sound fake, sometimes two, three, or even ten times. I believe programming, in this regard, can learn something from writing: when writing the first core of a new system, when the original creator is still alone, isolated, able to do anything, she should pretend that this first core is her only bullet. During the genesis of the system she should rewrite this primitive kernel again and again, in order to find the best possible design. My hypothesis is that this initial design will greatly inform what will happen later: growing organically something that has a good initial structure will result in a better system, even after years of distance from the original creation, and even if the original core was just a tiny faction of the future mass the system would eventually assume. In case you are interested, a quick update about my sci-fi novel. After many self-reviews I sent the manuscript to my editor, Giulio Mozzi. He will send me the change proposals in a few weeks. I’ll start a new review process informed by his notes, and hopefully finalize the novel in one or two months. Then, finally, I’ll be ready to publish the Italian version. A the same time the finalized novel will be sent to my translator, in the US, and when she ends the translation the English version will be published as well. It’s a long journey, but one that I deeply enjoyed taking. Comments

• The open source paradox
  Oct 03, 2020

  A new idea is insinuating in social networks and programming communities. It’s the proportionality between the money people give you for coding something, and the level of demand for quality they can claim to have about your work. As somebody said, the best code is written when you are supposed to do something else [1]. Like a writer will do her best when writing that novel that, maybe, nobody will pay a single cent for, and not when doing copywriting work for a well known company, programmers are likely to spend more energies in their open source side projects than during office hours, while writing another piece of a project they feel stupid, boring, pointless. And, if the company is big enough, chances are it will be cancelled in six months anyway or retired one year after the big launch. Open source is different, it’s an artifact, it’s a transposition in code of what you really want to do, of what you feel software should be, or just of all your fun and joy, or even anger you are feeling while coding. And you want it to rock, to be perfect, and you can’t sleep at night if there is a fucking heisenbug. So if a user of your software is addressing you because some part of your code sucks, and is willing to work with you to do something about it, and is very demanding, don’t think they are abusing you because they are not paying you. It’s not about money. You can ignore bugs if you want, and ignore their complains, you can do that since you don’t have a contract to do otherwise, but they are helping you, they care about the same thing you care: your software quality, grandiosity, perfection. The real right you have, and often don’t exploit, is that you are the only one that can decide about the design of your software. So you are entitled to refuse a pull request, or a proposal to follow good practices, because you feel that what somebody is contributing does not fit in the big picture of what you are designing and building. But if you recognize that somebody is talking you about something that is, really, a defect in your software, don’t do the error of reducing the interaction to a vile matter of money. You are doing work for free, they are risking their asses deploying what you wrote, you both care about quality. EDIT: If you write OSS and you are upset about user demands, have you ever thought that maybe, at this point, your work is more similar to office work for some reason? EDIT 2: A HN user asked the reasons for such title. The paradox is that the OSS writer cares and is often willing to fix code she writes for free, more than the other paid work she does. [1] "The best programs are the ones written when the programmer is supposed to be working on something else." - Melinda Varian. https://twitter.com/CodeWisdom/status/1309470447667421189 Comments

• The end of the Redis adventure
  Jun 30, 2020

  When I started the Redis project more than ten years ago I was in one of the most exciting moments of my career. My co-founder and I had successfully launched two of the major web 2.0 services of the Italian web. In order to make them scalable we had to invent many new concepts, that were already known in the field most of the times, but we didn’t know, nor we cared to check. Problem? Let’s figure out a solution. We wanted to solve problems but we wanted, even more, to have fun. This was the playful environment where Redis was born. But now Redis is, incredibly, one of the main parts of so many things. And year after year my work changed from building this thing to making sure that it was also as useful as possible, as reliable as possible. And in recent years, what I do every day changed so much that most of my attention is spent in checking what other developers tell me about the Redis code, how to improve it, the changes it requires to be more correct or faster or more secure. However I never wanted to be a software maintainer. I write code in order to express myself, and I consider what I code an artifact, rather than just something useful to get things done. I would say that what I write is useful just as a side effect, but my first goal is to make something that is, in some way, beautiful. In essence, I would rather be remembered as a bad artist than a good programmer. Now I’m asked more and more, by the circumstances created by a project that became so important, to express myself less and to maintain the project more. And this is indeed exactly what Redis needs right now. But this is not what I want to do, and I stretched myself enough during the past years. So, dear Redis community, today I’m stepping back as the Redis maintainer. My new position will be, on one side, an “ideas” person at Redis Labs, in order to provide inputs for new Redis possibilities: I’ll continue to be part of the Redis Labs advisory board. On the other hand however my hands will be free, and I’ll do something else, that could be writing code or not, who knows, I don’t want to make plans for now. However I’m very skeptical about me not writing more code in the future. It’s just too much fun :D I leave Redis in the hands of the Redis community. I asked my colleagues Yossi Gottlieb and Oran Agra to continue to maintain the project starting from today: these are the people that helped me the most in recent years, and that tried hard, even when it was not “linear” to follow me in my very subjective point of views, to understand what my vision on Redis was. Since I don’t want to be part of how the new Redis development setup will be shaped (that is the most meta of the maintenance tasks, exactly what I want to avoid), I’ll just leave Yossi and Oran the task of understanding how to interface with the rest of the Redis developers to find a sustainable development model, you can hear directly from Yossi and Oran in this blog post: https://redislabs.com/blog/new-governance-for-redis/ I believe I’m not just leaving Redis in the hands of a community of expert programmers, but also in the hands of people who care about the legacy of the community spirit of Redis. In eleven years I hope I was able to provide a point of view that certain persons understood, about an alternative way to write software. I hope that such point of view will be taken into consideration in the evolution of Redis. Redis was the most stressful thing I did in my career, and probably also the most important. I don’t like much what the underground programming world became in recent years, but even if it was not an easy journey, I had the privilege to work and interact with many great individuals. Thank you for your humanity and your help, and for what you taught me. You know who you are! I want to also say thank you to the companies and individuals inside such companies that allowed me to write open source every day for so many years, with the freedom to do what I believed to be correct for the user base. Redis Labs, VMware and Pivotal, thank you for your great help and generosity. As I said, I don’t really know what there is for me in my future, other than the involvement with the Redis advisory board. I guess that for some time, just look around is a good idea, without doing too many things. I would like to explore more a few hobbies of mine. Writing blog posts is also a major thing that I wanted to do but did less and less because of time concerns. Recently I published videos in Italian language explaining technological concepts to the general public, I had fun doing that and received good feedbacks, maybe I’ll do more of that as well. Anyway I guess some of you know that I’m active on Twitter as @antirez. If you are interested in what an old, strange programmer will do next, see you there. Comments

• Redis 6.0.0 GA is out!
  Apr 30, 2020

  Finally Redis 6.0.0 stable is out. This time it was a relatively short cycle between the release of the first release candidate and the final release of a stable version. It took about four months, that is not a small amount of time, but is not a lot compared to our past records :) So the big news are the ones announced before, but with some notable changes. The old stuff are: SSL, ACLs, RESP3, Client side caching, Threaded I/O, Diskless replication on replicas, Cluster support in Redis-benchmark and improved redis-cli cluster support, Disque in beta as a module of Redis, and the Redis Cluster Proxy (now at https://github.com/RedisLabs/redis-cluster-proxy). So what changed between RC1 and today, other than stability? 1. Client side caching was redesigned in certain aspects, especially the caching slot approach was discarded in favor of just using key names. After analyzing the alternatives, with the help of other Redis core team members, in the end this approach looks better. Other than that, finally the feature was completed with the things I had in the backlog for the feature, especially the “broadcasting mode”, that I believe will be one of the most popular usage modes of the feature. When broadcasting is used, the server no longer try to remember what keys each client requested. Instead clients subscribe to key prefixes: they’ll get notifications every time a key matching the prefix is modified. This means more messages (but only for the selected prefixes), but no memory effort in the server side. Moreover the opt-in / opt-out mode is now supported, so it is possible for clients not using the broadcasting mode, to exactly tell the server about what the client will cache, to reduce the number of invalidation messages. Basically the feature is now much better both when a low-memory mode is needed, and when a very selective (low-bandwidth) mode is needed. 2. This was an old request by many users. Now Redis supports a mode where RDB files used for replication are immediately deleted if no longer useful. In certain environments it is a good idea to never have the data around on disk, but just in memory. 3. ACLs are better in a few regards. First, there is a new ACL LOG command that allows to see all the clients that are violating the ACLs, accessing commands they should not, accessing keys they should not, or with failed authentication attempts. The log is actually in memory, so every external agent can call “ACL LOG” to see what’s going on. This is very useful in order to debug ACL problems. But my preferred feature is the reimplementation of ACL GENPASS. Now it uses SHA256 based HMAC, and accepts an optional argument to tell the server how many bits of unguessable pseudo random string you want to generate. Redis seeds an internal key at startup from /dev/urandom, and later uses the HMAC in counter mode in order to generate the other random numbers: this way you can abuse the API, and call it every time you want, since it will be very fast. Want to generate an unguessable session ID for your application? Just call ACL GENPASS. And so forth. 4. PSYNC2, the replication protocol, is now improved. Redis will be able to partially resynchronize more often, since now is able to trim the final PINGs in the protocol, to make more likely that replicas and masters can find a common offset. 5. Redis commands with timeouts are now much better: not only BLPOP and other commands that used to accept seconds, now accept decimal numbers, but the actual resolution was improved in order to never be worse than the current “HZ” value, regardless of the number of clients connected. 6. RDB files are now faster to load. You can expect a 20/30% improvement, depending on the file actual composition (larger or smaller values). INFO is also faster now when there are many clients connected, this was a long time problem that now is finally gone. 7. We have a new command, STRALGO, that implements complex string algorithms. For now the only one implemented is LCS (longest common subsequence), an important algorithm used, among the other things, in order to compare the RNA of the coronaviruses (and in general the DNA and RNA of other organisms). What is happening is too big, somewhat a trace inside Redis needed to remain. Redis 6 is the biggest release of Redis *ever*, so even if it is stable, handle it with care, test it for your workload before putting it in production. We never saw big issues so far, but make sure to be careful. As we collect bug reports, we will prepare to release Redis 6.0.1 ASAP. A big thank you to the many people that wrote code with me in this release, and to all the companies that sponsored both my work (Thanks Redis Labs), and the the work of the other contributors (Thanks other companies). Also a big thank you to the many that signaled bugs with care, sometimes following the boring process of reiterating after making some changes, or that suggested improvements of any kind. As usually you can find Redis 6 in different places: at https://redis.io as tarball, and in the Github repository tagged as “6.0.0”. Enjoy Redis 6, antirez Comments

• Redis 6 RC1 is out today
  Dec 19, 2019

  So it happened again, a new Redis version reached the release candidate status, and in a few months it will hit the shelves of most supermarkets. I guess this is the most “enterprise” Redis version to date, and it’s funny since I took quite some time in order to understand what “enterprise” ever meant. I think it’s word I genuinely dislike, yet it has some meaning. Redis is now everywhere, and it is still considerably able to “scale down”: you can still download it, compile it in 30 seconds, and run it without any configuration to start hacking. But being everywhere also means being in environments where things like encryption and ACLs are a must, so Redis, inevitably, and more than thanks to me, I would say, in spite of my extreme drive for simplicity, adapted. But what’s interesting is that, even additions may be done in very opinionated ways. Redis ACLs hardly resemble something you saw in other systems, and SSL support was written in a few iterations in order to finally pick the idea that was the most sounding, from the point of view of letting the core as clean as possible. I’m quite happy with the result. Redis 6 does not bring just ACLs and SSL, it is the largest release of Redis ever as far as I can tell, and the one where the biggest amount of people participated. So, let’s start with credits. Who made Redis 6? This is the list of contributors by commits (it’s a terrible metric, but the one I can easily generate), having at least two commits, and excluding merge commits. Also note that the number of commits in my case may be inflated a lot by the fact that I fix many small stuff here and there constantly. 685 antirez 81 zhaozhao.zz 76 Oran Agra 51 artix 28 Madelyn Olson 27 Yossi Gottlieb 15 David Carlier 14 Guy Benoish 14 Guy Korland 13 Itamar Haber 9 Angus Pearson 8 WuYunlong 8 yongman 7 vattezhang 7 Chris Lamb 5 Dvir Volk 5 meir@redislabs.com 5 chendianqiang 5 John Sully 4 dejun.xdj 4 Daniel Dai 4 Johannes Truschnigg 4 swilly22 3 Bruce Merry 3 filipecosta90 3 youjiali1995 2 James Rouzier 2 Andrey Bugaevskiy 2 Brad Solomon 2 Hamid Alaei 2 Michael Chaten 2 Steve Webster 2 Wander Hillen 2 Weiliang Li 2 Yuan Zhou 2 charsyam 2 hujie 2 jem 2 shenlongxing 2 valentino 2 zhudacai 00228490 2 喜欢兰花山丘 Thanks to all the above folks, it was a great team work ladies and gentlemen. The list of new features in the change log is the following: * Many new modules APIs. * Better expire cycle. * SSL * ACLs * RESP3 * Client side caching * Threaded I/O * Diskless replication on replicas * Redis-benchmark cluster support + Redis-cli improvements * Systemd support rewrite. * Redis Cluster proxy was released with Redis 6 (but different repository). * A Disque module was released with Redis 6 (but different repository). Many big things, as you can see. I’ll spend a few words on selected ones. RESP3 === After ten years we needed a new protocol, I talked extensively about it here http://antirez.com/news/125, but then changed my mind, so the RESP3 protocol in Redis 6 is “opt in”. The connection starts in RESP2 mode, and only if you do a handshake using the new HELLO command, you enter in the new protocol mode. Why a new protocol? Because the old one was not semantical enough. There are other features in RESP3, but the main idea was the ability to return complex data types from Redis directly, without the client having to know in what type to convert the flat arrays returned, or the numbers returned instead of proper boolean values, and so forth. Since RESP3 is not the only protocol supported I expect the adoption to be slower than expected, but maybe this is not a bad thing after all: we’ll have time to adapt. ACLs === The best introduction to Redis ACLs is the ACL documentation itself (https://redis.io/topics/acl), even if probably it needs some update to match the last minute changes. So it’s more interesting to talk about motivations here. Redis needed ACLs because people need ACLs in bigger environments, in order to control better which client can do certain operations. But another main point about adding ACLs to Redis was isolation in order to defend the data against application bugs. If your worker can only do BRPOPLPUSH, the chance of the new developer adding for debugging a FLUSHALL that ends in production code for error and creates a nightmare for 5 hours, is lower. ACLs in Redis are for free, both operationally, because if you don’t use them, you can avoid knowing they are supported at all, and from the point of view of performances, since the overhead is not measurable. I guess it’s a good deal to have them. Bonus point, we have a Redis modules interface for ACLs now, so you can write custom authentication methods. SSL === It’s 2019, almost 2020, and there are new regulations. The only problem was doing it right. And doing it right required doing it wrong, understanding the limitations, and then abstracting the Redis connections in order to do it right. This work was entirely performed without my help, which shows how the Redis development process changed in recent times. Client side caching === I blogged about it here http://antirez.com/news/130, however I think that right now this is the most immature feature of Redis 6. Yep, it’s cool that the server can assist you in caching client side values, but I want to improve this before Redis 6 GA is out. Especially it could be very good to add a new mode that requires the server to maintain no state about clients, or very little state at all, and trade this with more messages. Moreover right now the messages to expire certain “cache slots” can’t be compiled in a single one. There is more work to do in January about this feature, but it will be a good one. Disque as a module === Finally I did it :-) https://github.com/antirez/disque-module, and I’m very happy with the result. Disque as a module really shows how powerful is the Redis module system at this point. Cluster message bus APIs, ability to block and resume clients, timers, AOF and RDB control of module private data. If you don’t know what Disque is, check the repository: the README is quite extensive. Cluster Proxy === My colleague Fabio worked for months at this Redis Cluster proxy: https://github.com/artix75/redis-cluster-proxy. It is ages that I want to see it happening, the client landscape is very fragmented when the topic is Redis Cluster support, so now we have a (work in progress) proxy that can do many interesting things. The main one is to abstract the Redis Cluster for clients, like if they were talking to a single instance. Another one is to perform multiplexing, at least when it is simple and clients just use simple commands and features. When there is to block or to perform transactions, the proxy allocates a different set of connections for the client. The proxy is also completely threaded, so it can be a good way in order to maximize the CPU usage in case most of your CPU time is spent in I/O. Check the project README for status and give it a try! Modules === With Redis 6 the modules API is totally at a new level. This is one of the part of Redis that matured faster in our history, because Redis Labs used the modules system from day zero in order to develop very complex stuff, not just trivial examples. Some time ago I started the Disque port, and this also motivated to bring me new features to the modules system. The result is that Redis is really a framework in order to write systems as modules, without having to invent everything from scratch, and being BSD licensed, Redis is really an open platform to write systems. Internals === There are tons of improvements to the Redis internals: the way commands are replicated changed quite a bit, the expires are now using a different algorithm which is faster and more cache obvious. Status and ETA === Today we went RC1, and I hope that between end of March, or at worst, May, you’ll see the GA ready. Right now Redis 6 is definitely testable and the chance you run into a bug is very small. Yet it includes a ton of code changes, and the new features are composed of new code that nobody ran in production before. So if you find bad things, please report them in the issue system describing at your best what happened. Thanks everybody that made this release possible and that will work in the next months to bring it to a very stable state. Oh, I almost forgot! This is the LOLWUT command interactive art for version 6: img://antirez.com/misc/lolwut6.png Every run displays a different landscape that is randomly generated. Comments

• Client side caching in Redis 6
  Jul 04, 2019

  [Note: this post no longer describes the client side implementation in the final implementation of Redis 6, that changed significantly, see https://redis.io/topics/client-side-caching] The New York Redis day was over, I get up at the hotel at 5:30, still pretty in sync with the Italian time zone and immediately went walking on the streets of Manhattan, completely in love with the landscape and the wonderful feeling of being just a number among millions of other numbers. Yet I was thinking at the Redis 6 release with the feeling that, what was probably the most important feature at all, the new version of the Redis protocol (RESP3), was going to have a very slow adoption curve, and for good reasons: wise people avoid switching tools without very good reasons. After all why I wanted to improve the protocol so badly? For two reasons mainly, to provide clients with more semantical replies, and in order to open to new features that were hard to implement with the old protocol; one feature in particular was the most important to me: client side caching. Rewind back to about one year ago. I arrived at Redis Conf 2018, in San Francisco, with the firm idea that client side caching was the most important thing in the future of Redis. If we need fast stores and fast caches, then we need to store a subset of the information inside the client. It is a natural extension of the idea of serving data with small delays and at a big scale. Actually almost every very large company already does it, because it is the only way to survive to the load eventually. Yet Redis had no way to assist the client in such process. A fortunate coincidence wanted Ben Malec having a talk at Redis Conf exactly about client side caching [1], just using the tools that Redis provides and a number of very clever ideas. [1] https://www.youtube.com/watch?v=kliQLwSikO4 The approach taken by Ben really opened my imagination. There were two key ideas Ben used in order to make his design work. The first was to use the Redis Cluster idea of “hash slots” in order to divide keys into 16k groups. That way clients would not need to track the validity of each key, but could use a single metadata entry for a group of keys. Ben used Pub/Sub in order to send the notifications when keys where changed, so he needed some help by the application in all its parts, however the schema was very solid. Modify a key? Also publish a message invalidating it. On the client side, are you caching keys? Remember the timestamp at which you cache each key, and also when receiving invalidation messages, remember the invalidation time for each slot. When using a given cached key, do a lazy eviction, by checking if the key you cached has a timestamp which is older than the timestamp of the invalidation received for the slot this key belongs to: in that case the key is stale data, you have to ask the server again. After watching the talk, I realized that this was a great idea to be used inside the server, in order to allow Redis to do part of the work for the client, and make client side caching simpler and more effective, so I returned home and wrote a document describing my design [2]. [2] https://groups.google.com/d/msg/redis-db/xfcnYkbutDw/kTwCozpBBwAJ But to make my design working I had to focus on switching the Redis protocol to something better, so I started writing the specification and later the code for RESP3, and the other Redis 6 things like ACL and so forth, and client side caching joined the huge room of the many ideas for Redis that I abandoned in some way or the other for lack of time. Yet I was among the streets of New York thinking about this idea. Later went to lunch and coffee break with friends from the conference. When I returned to my hotel room I had all the evening left, and most of the next day before the flight, so I started writing the implementation of client side caching for Redis 6, closely following the proposal I wrote to the group one year ago: it still looked great. Redis server-assisted client side caching, finally called “tracking” (but I may change idea), is a very simple feature composed of just a few key ideas. The key space is split into “caching slots”, but they are a lot more than the hash slots used by Ben. We use 24 bits of the output of CRC64, so there are a bit more than 16 millions different slots. Why so much? Because I think you want to have a server with 100 millions of keys, and yet an invalidation message should not affect more than a few keys in the client side cache. The memory overhead inside Redis to take the invalidation table is 130 megabyte: an array of 8 bytes pointers to 16M entries. That’s ok with me, if you want the feature you are going to make a great use of all the memory you have in the clients, so to use 130MB server side is fine; what you win is a much more fine grained invalidation. Clients enable the feature in an “opt in” way, with a simple command: CLIENT TRACKING on The server replies the good old +OK, and starting from that moment, every command that is flagged as “read only” in the command table, will not just return the keys to the caller, it will also, as a side effect, remember the caching slots of all the keys the client requested so far (but only the ones using a read only command, that's the agreement between the server and the client). The way Redis stores this information is simple. Each Redis client has an unique ID, so if client ID 123 performs an MGET about keys hashing to the slot 1, 2, and 5, we’ll have the Invalidation Table with the following entry: 1 -> [123] 2 -> [123] 5 -> [123] But later also client ID 444 will ask about keys in the slot 5, so the table will be like: 5 -> [123, 444] Now some other client changes some key in the slot 5. What happens is that Redis will check the Invalidation Table, to find that both clients 123 and 444 may have cached keys on that slot. We’ll send an invalidation message to both clients, as a result they will be free to deal with it in any form: either remember with a timestamp the last time the slot was invalidate, and check later in a lazy way the timestamp (or incremental “epoch” if you like it more: it is safer) of the cached object, and evict it based on the comparison. Otherwise the client is free to reclaim the objects directly, by taking a table of what it cached about this specific slot. This approach with a 24 bit hash function is not an issue, because we’ll not have a very long list at all, even when caching tens of millions of keys. After sending the invalidation messages, we can remove the entries from the invalidation table, this way we'll no longer send invalidation messages to those clients until they don't read again keys for such slot. Note that clients are not forced to really use all the 24 bits of the hash function. They may just use, for instance, 20 bits, and then also shift the invalidation messages slots that Redis sends them. Not sure if there are many good reasons to do that, but in memory constrained systems may be an idea. If you followed closely what I said, you are thinking that the same connection receives both the normal client replies, and the invalidation messages. This is possible with RESP3, because invalidations are sent as “push” message types. Yet if the client is a blocking one, and not an event driven client, this starts to be complex: the application need some way to read new data from time to time, and that looks complex and fragile. It is perhaps, in that case, much better to use another application thread, and a different client connection, in order to receive the invalidation messages. So you are allowed to do the following: CLIENT TRACKING on REDIRECT 1234 Basically we can say that all the keys we get with the current connection, we want the invalidation message to be sent to client 1234 instead. Multiple clients may ask to have the invalidation messages redirected to a single client for instance, in case of connection pools. All you need to do is to create this special connection to receive the invalidation messages, call CLIENT ID to know which ID this client connection has, and later enable tracking. There is one problem left: what happens if we lose the connection with the server from the invalidation link? We may run into troubles since invalidation messages will no longer be received. Normally the application will detect the link is severed, and will reconnect again, flushing the current cache (or taking more soft resolutions, like putting all the timestamps for the slots a few seconds in the future to have some time to populate the cache while serving data that may be a few seconds stale). Yet it may be a better idea if the invalidation thread pings from time to time the connection to make sure it is alive. However in order to reduce the risk of stale data, Redis will also start to inform the clients that redirected the invalidation messages to some other client, that is now disconnected, about the situation, just using special push messages: at the next query performed the client will know. What I described was just merged into Redis unstable. Probably it’s not the final word, but we have more months before the first Redis 6 release candidate, there is time to change everything: just send me your feedbacks. I’m also looking at ways to enable the feature for RESP2. That would work only when redirection is enabled, and the client listening for messages should probably go into Pub/Sub mode so that we could send kinda of Pub/Sub messages. In this way old clients can be fully reused. I hope this was enough to stimulate your appetite: if we execute this inside Redis very well, and then document it for the client authors to know how to provide support, data may go a lot nearer to the application than ever, even in applications ran by small teams that so far avoided trying to implement client side caching. For large teams and very large applications doing this already, the overhead could be reduced, together with the complexity of the implementation. Comments

• The struggles of an open source maintainer
  May 16, 2019

  Months ago the maintainer of an OSS project in the sphere of system software, with quite a big and active community, wrote me an email saying that he struggles to continue maintaining his project after so many years, because of how much psychologically taxing such effort is. He was looking for advices from me, I’m not sure to be in the position of giving advices, however I told him I would write a blog post about what I think about the matter. Several weeks passed, and multiple times I started writing such post and stopped, because I didn’t had the time to process the ideas for enough time. Now I think I was able to analyze myself to find answers inside my own weakness, struggles, and desire of freedom, that inevitably invades the human minds when they do some task, that also has some negative aspect, for a prolonged amount of time. Maintaining an open source project is also a lot of joy and fun and these latest ten years of my professional life are surely memorable, even if not the absolute best (I had more fun during my startup times after all). However here I’ll focus on the negative side; simply make sure you don’t get the feeling it is just that, there is also a lot of good in it. Flood effect I don’t believe in acting fast, thinking fast, winning the competition on time and stuff like that. I don’t like the world of constant lack of focus we live in, because of social networks, chats, emails, and a schedule full of activities. So when I used to receive an email about Redis back in the early times of the project, when I still had plenty of time, I was able to focus on what the author of the message was trying to tell me. Then I could recall the relevant part of Redis we were discussing, and finally reply with my real thoughts, after considering the matter with care. I believe this is how most people should work regardless of what their job is. When a software project reaches the popularity Redis reached, and at the same time once the communications between individuals are made so simple by the new social tools, and by your attitude to be “there” for users, the amount of messages, issues, pull requests, suggestions the authors receive will grow exponentially. At the same time, at least in the case of Redis, but I believe this to be a common problem, the amount of very qualified people that can look at such inputs from the community grows very slowly. This creates an obvious congestion. Most people try to address it in the wrong way: using pragmatism. Let’s close the issue after two weeks of no original poster replies, after we ask some question. Close all the issues that are not very well specified. And other “inbox zero” solutions. The reality is that to process community feedbacks very well you have to take the time needed, otherwise you will just pretend your project has a small number of open issues. Having a lot of resources to hire core-level experts for each Redis subsystem, to work at OSS full time, would work but is not feasible. So what happens? That you start to prioritize more and more what to look at and what not. And you feel you are a piece of shit at ignoring so many things and people, and also the contributor believes you don’t care about what others have to give you. It’s a complex situation. Usually the end result is to develop an attitude to mostly address critical issues and disregard all the new stuff, since new stuff are yet not in the core, and who wants to have a larger code base with even more PRs and issues there? Maybe also written in a more convoluted way compared to your usual programming style, so, more complexity, and good luck when there is a critical bug there to track the root cause. Role shifting As a result of the “flood effect” problem exposed above, you suddenly also change job. Redis became popular because I supposedly am able to design and write software. And now instead most of the work I do is to look at issues and pull requests, and I also feel that I could do better many of the contributions I receive. Some will be better quality than I could do, because there are also better programmers than me contributing to Redis, but *most* for the nature of big numbers will be average contributions that are just written to solve a given problem that was contingent for the folks that submitted it. While, when I design for Redis, I tend to think at Redis as a whole, because it’s years I write this thing. So what you were good at, you have no longer time to do. This in turn means less organic big new features. My solution with that? Sometimes I just stop looking at issues and PRs for weeks, because I’m coding or designing: that is the work I really love and enjoy. However this in turn creates ways more pressure on me, psychologically. To do what I love and I can do well I’ve to feel like shit. Time There are two problems related to working at the same project for a prolonged amount of time, at least for me. First, before of the Redis experience I *never* worked every week day of my life. I could work one week, stop two, then work one month, then disappear for other two months. Always. People need to recharge, get new energy and ideas, to do creative work. And programming at high level is a fucking creative job. Redis itself was created like that for the first two years, that is, when the project evolved at the fastest speed. Because the sum of the productivity of me working just when I want is greater than the productivity I’ve when I’m forced to work every day in a steady way. However my work ethics allowed me to have a very discontinue schedule when I was working alone with my companies. Once I started to receive money to work at Redis, it was no longer possible for my ethics to have my past pattern, so I started to force myself to work under normal schedules. This for me is a huge struggle, for many years at this point. Moreover I’m sure I’m doing less than I could because of that, but this is how things work. I never found a way to solve this problem. I could say Redis Labs that I want to return to my old schedule, but that would not work because at this point I really “report” to the community, not to the company. Another problem is that working a lot at the same project is also a complex matter, mentally speaking. I used to change project every six months in the past. Now for ten years I did the same thing. In that regard I tried to save my sanity by having sub-projects inside Redis. One time I did Cluster, another time disk-storage (now abandoned), another was HyerLogLogs, and so forth. Basically things that bring value to the project but that, in isolation, are other stuff. But eventually you have to return back to the issues and PRs page and address the same things every day. “Replica is disconnecting because of a timeout” or whatever. Let’s investigate that again. Fear I always had some fear to lose the technological leadership of the project. Not because I think I’m not good enough at designing and evolving Redis, but because I know my ways are not aligned with: 1) what a sizable amount of users want. 2) what most people in IT believe software is. So I had to constantly balance between what I believe to be good design, set of features, speed of development (slow), size of the project (minimal), and what I was expected to deliver by most of the user base. Fortunately there is a percentage of Redis users that understand perfectly the Redis-way, so at least from time to time I can get some word of comfort. Frictions Certain people are total assholes. They are everywhere, it is natural and if you ask me, I even believe in programming there are a lot more nice people than in other fields. But yet you’ll always see a percentage of total jerks. As the leader of a popular OSS project, in one way or the other you’ll have to confront with these people, and that’s maybe one of the most stressful things I ever did in the course of the Redis development. Futileness Sometimes I believe that software, while great, will never be huge like writing a book that will survive for centuries. Note because it is not as great per-se, but because as a side effect it is also useful… and will be replaced when something more useful is around. I would like to have time to do other activities as well. So sometimes I believe all I’m doing is, in the end, futile. We’ll design and write systems, and new systems will emerge; but anyone that just stays in software, instead of staying in “software big ideas”, will ever set a new mark? From time to time I think I had potentially the ability to work at big ideas but because I focused on writing software instead of thinking about software, I was not able to use my potential in that regard. This is basically the contrary of the impostor syndrome, so I guess I’ve a big idea of myself: sorry for that I should be more humble. That said, I was able to work for many years doing things I really loved, that gave me friends, recognition, money, so I don’t want to say it was a bad deal. Yet I totally understand people struggling a lot to stay afloat once their projects start to be popular. This blog post is dedicated to them. Comments

• Redis streams as a pure data structure
  Mar 22, 2019

  The new Redis data structure introduced in Redis 5 under the name of “Streams” generated quite some interest in the community. Soon or later I want to run a community survey, talking with users having production use cases, and blogging about it. Today I want to address another issue: I’m starting to suspect that many users are only thinking at Streams as a way to solve Kafka(TM)-alike use cases. Actually the data structure was designed to *also* work in the context of messaging with producers and consumers, but to think that Redis Streams are just good for that is incredibly reductive. Streaming is a terrific pattern and “mental model” that can be applied when designing systems with great success, but Redis Streams, like most Redis data structures, are more general, and can be used to model dozen of different unrelated problems. So in this blog post I’ll focus on Streams as a pure data structure, completely ignoring its blocking operations, consumer groups, and all the messaging parts. ## Streams are CSV files on steroids If you want to log a series of structured data items and decided that databases are overrated after all, you may say something like: let’s just open a file in append only mode, and log every row as a CSV (Comma Separated Value) item: (open data.csv in append only) time=1553096724033,cpu_temp=23.4,load=2.3 time=1553096725029,cpu_temp=23.2,load=2.1 Looks simple and people did this for ages and still do: it’s a solid pattern if you know what you are doing. But what is the in-memory equivalent of that? Memory is more powerful than an append only file and can automagically remove the limitations of a CSV file like that: 1. It’s hard (inefficient) to do range queries here. 2. There is too much redundant information: the time is almost the same in every entry and the fields are duplicated. At the same time removing it will make the format less flexible, if I want to switch to a different set of fields. 3. Item offsets are just the byte offset in the file: if we change the file structure the offset will be wrong, so there is no actual true concept of primary ID here. Entries are basically not univocally addressed in some way. 4. I can’t remove entries, but only mark them as no longer valid without the ability of garbage collecting, if not by rewriting the log. Log rewriting usually sucks for several reasons and if it can be avoided, it’s good. Still such log of CSV entries is also great in some way: there is no fixed structure and fields may change, is trivial to generate, and after all is quite compact as well. The idea with Redis Streams was to retain the good things, but go over the limitations. The result is a hybrid data structure very similar to Redis Sorted Sets: they *feel like* a fundamental data structure, but to get such an effect, internally it uses multiple representations. ## Streams 101 (you may skip that if you know already Redis Stream basics) Redis Streams are represented as delta-compressed macro nodes that are linked together by a radix tree. The effect is to be able to seek to random entries in a very fast way, to obtain ranges if needed, remove old items to create a capped stream, and so forth. Yet our interface to the programmer is very similar to a CSV file: > XADD mystream * cpu-temp 23.4 load 2.3 "1553097561402-0" > XADD mystream * cpu-temp 23.2 load 2.1 "1553097568315-0" As you can see from the example above the XADD command auto generates and returns the entry ID, which is monotonically incrementing and has two parts: -. The time is in milliseconds and the counter increases for entries generated in the same milliseconds. So the first new abstraction on top of the “append only CSV file” idea is that, since we used the asterisk as the ID argument of XADD, we get the entry ID for free from the server. Such ID is not only useful to point to a specific item inside a stream, it’s also related to the time when the entry was added to the stream. In fact with XRANGE it is possible to perform range queries or fetch single items: > XRANGE mystream 1553097561402-0 1553097561402-0 1) 1) "1553097561402-0" 2) 1) "cpu-temp" 2) "23.4" 3) "load" 4) "2.3" In this case I used the same ID as the start and the stop of the range in order to identify a single element. However I can use any range, and a COUNT argument to limit the number of results. Similarly there is no need to specify full IDs as range, I can just use the millisecond unix time part of the IDs, to get elements in a given range of time: > XRANGE mystream 1553097560000 1553097570000 1) 1) "1553097561402-0" 2) 1) "cpu-temp" 2) "23.4" 3) "load" 4) "2.3" 2) 1) "1553097568315-0" 2) 1) "cpu-temp" 2) "23.2" 3) "load" 4) "2.1" For now there is no need to show you more Streams API, there is the Redis documentation for that. For now let’s just focus on that usage pattern: XADD to add stuff, XRANGE (but also XREAD) in order to fetch back ranges (depending on what you want to do), and let’s see why I claim Streams are so powerful as a data structure. However if you want to learn more about Redis Streams and their API, make sure to visit the tutorial here: https://redis.io/topics/streams-intro ## Tennis players A few days ago I was modeling an application with a friend of mine which is learning Redis those days: an app in order to keep track of local tennis courts, local players and matches. The way you model players in Redis is quite obvious, a player is a small object, so an Hash is all you need, with key names like player:. As you model the application data further, to use Redis as its primary, you immediately realize you need a way to track the games played in a given tennis club. If player:1 and player:2 played a game, and player 1 won, we could write the following entry in a stream: > XADD club:1234.matches * player-a 1 player-b 2 winner 1 "1553254144387-0" With this simple operation we have: 1. A unique identifier of the match: the ID in the stream. 2. No need to create an object in order to identify a match. 3. Range queries for free to paginate the matches, or check the matches played in a given moment in the past. Before Streams we needed to create a sorted set scored by time: the sorted set element would be the ID of the match, living in a different key as a Hash value. This is not just more work, it’s also an incredible amount of memory wasted. More, much more you could guess (see later). For now the point to show is that Redis Streams are kinda of a Sorted Set in append only mode, keyed by time, where each element is a small Hash. And in its simplicity this is a revolution in the context of modeling for Redis. ## Memory usage The above use case is not just a matter of a more solid pattern. The memory cost of the Stream solution is so different compared to the old approach of having a Sorted Set + Hash for every object that makes certain things that were not viable, now perfectly fine. Those are the numbers for storing one million of matches in the configurations exposed previously: Sorted Set + Hash memory usage = 220 MB (242 RSS) Stream memory usage = 16.8 MB (18.11 RSS) This is more than an order of magnitude difference (13 times difference exactly), and it means that use cases that yesterday were too costly for in-memory now are perfectly viable. The magic is all in the representation of Redis Streams: the macro nodes can contain several elements that are encoded in a data structure called listpack in a very compact way. Listpacks will take care, for instance, to encode integers in binary form even if they are semantically strings. On top of that, we then apply delta compression and same-fields compression. Yet we are able to seek by ID or time because such macro nodes are linked in the radix tree, which was also designed to use little memory. All these things together account for the low memory usage, but the interesting part is that semantically the user does not see any of the implementation details making Streams efficient. Now let’s do some simple math. If I can store 1 million entries in about 18 MB of memory, I can store 10 millions in 180 MB, and 100 millions in 1.8 GB. With just 18 GB of memory I can have 1 billion items. ## Time series One important thing to note is, in my opinion, how the usage above where we used a Stream to represent a tennis match was semantically *very different* than using a Redis Stream for a time series. Yes, logically we are still logging some kind of event, but one fundamental difference is that in one case we use the logging and the creation of entries in order to render objects. While in the case of time series, we are just metering something happening externally, that does not really represent an object. You may think that this difference is trivial but it’s not. It is important for Redis users to build the idea that Redis Streams can be used in order to create small objects that have a total order, and assign IDs to such objects. However even the most basic use case of time series is, obviously, a huge one here, because before Streams Redis was a bit hopeless in regard to such use case. The memory characteristics and flexibility of streams, plus the ability to have capped streams (see the XADD options), is a very important tool in the hands of the developer. ## Conclusions Streams are flexible and have lots of use cases, however I wanted to take this blog post short to make sure that there is a clear take-home message in the above examples and analysis of the memory usage. Perhaps this was already obvious to many readers, but talking with people in the last months gave me the feeling that there was a strong association between Streams and the streaming use case, like if the data structure was only good at that. That’s not the case :-) Comments

• Gopher: a present for Redis
  Feb 25, 2019

  Ten years ago Redis was announced on Hacker News, and I use this as virtual birthdate for the project, simply because it is more important when it was announced to the public than the actual date of the project first line of code (think at it conception VS actual birth in animals). I’ll use the ten years of Redis as an excuse to release something I played a bit in the previous days, thinking to use it for the 1st April fool: but such date is far and I want to talk to you about this project now… So, happy birthday Redis! Here it’s your present: a Gopher protocol implementation. [… here Redis tries to stop the tears, but the emotion is too strong and there are bits (I mean zeros and ones) on the floor …] WTF are you saying?! should be your automatic question. Gopher in 2019 sounds a bit strange. However it is not *just* a joke, while it is largely a joke. The implementation is just 100 lines of code after all, excluding the external tool to render the pages into Redis keys. But… the thing is that there is really an active community around Gopher, a very small one but one that is growing in the latest years and months. There are people that feel that internet is no longer what it used to be. There is too much control, companies tracking, comments, likes, retweets, to the point that the content is no longer the king. One writes new things for them to be popular for 5 hours and disappear. There is no longer a discussion that can survive more than a few minutes without becoming some kind of flame, unless all the parties self-censor every possible feeling, uneasy word, and belief, to the point to make the discussion quite useless. Finally to load a stupid page with 1k of text requires to load 50 javascript files, to see the screen flickering since client-side rendering is cool, and so forth. On the other hand Gopher is a text only protocol that is great to deliver text only documents where the stress is in what you write. But that would be fetichism, for me the silver bullet of Gopher is that it is UNCOOL. Uncool enough that it will be forever, AFAIK, an alternative reality where certain folks can decide to separate from the rest, to experience a different way to do things, more similar to the old times of BBSs or the first years of the internet. A place where most people will not want to go just to read nerdy stuff in an 80 columns fixed size font. What you do in Gopher is to create your Gopher hole, that is, your space inside the Gopher universe, like your web site on the internet basically. There was no shortage of tools to do that already, but Redis is quite nice for a few reasons: you can change the Redis keys to change the site content in real time, that’s handy. You can use replication in order to duplicate a site, and can even just save your RDB file to have an exact copy of the whole Gopher hole to archive for backup or historical reasons. This Redis Gopher concept was created with the collaboration of Freaknet, a historical hacking laboratory experience here in Catania. https://it.wikipedia.org/wiki/FreakNet. Those folks do a lot of interesting stuff, including a retrocomputing hardware museum project in Palazzolo Acreide here: https://museo.freaknet.org/en/. How it works? Well it’s trivial, I hijacked the inline protocol, and specifically two kind of inline requests that were anyway illegal: an empty request or any request that starts with "/" (there are no Redis commands starting with such a slash). Normal RESP2/RESP3 requests are completely out of the path of the Gopher protocol implementation and are served as usually as well. If you open a connection to Redis when Gopher is enabled and send it a string like "/foo", if there is a key named "/foo" it is served via the Gopher protocol. The whole implementation is 100 lines of code. Initially I thought about using data structures and have semantical transformations to Gopher types, but that’s just complex and useless. Instead what I did was to provide an authoring tool for Gopher over Redis, you can find it here: https://github.com/antirez/gopher2redis To see that example Gopher hole running on a Redis instance just go to gopher://gopher.antirez.com, and btw that will be the address of my Gopher hole once I’ll build one in the next days. P.S. I suggest using the Lynx text only web/gopher browser to access Gopher. The gopher support is disabled by default, to enable it use the Redis unstable branch and use the “gopher-enabled” option, setting it to yes. However MAKE SURE to also password protect Redis: the Gopher protocol will still serve content, but at the same time normal Redis commands will not be accessible. This way (and assuming you don’t have data other than your Gopher keys to expose in the instance) you could make the instance public, as a true Gopher server. Well, have fun with Gopher! I hope this Gopher thing will go forward, I really believe there are a few of us that need to create a community outside the chaos of the modern Internet. No, it will not be possible to have no interactions. For instance I’ve no plans to stop blogging or using Internet. But certain slower higher quality communications need a place to prosper. Comments

• An update about Redis developments in 2019
  Feb 20, 2019

  Yesterday a concerned Redis user wrote the following on Hacker News: — https://news.ycombinator.com/item?id=19204436 — I love Redis, but I'm a bit skeptical of some of the changes that are currently in development. The respv3 protocol has some features that, while they sound neat, also could significantly complicate client library code. There's also a lot of work going into a granular acl. I can't imagine why this would be necessary, or a higher priority than other changes like multi-thread support, better persistence model, data-types, etc. — end of user comment — I’ve the feeling she/he (not sure) is not the only one that looks at ACLs as some sort of feature imposed by the Redis Labs goals, because “enterprise users” or something like that. Also the other points in the comment are interesting, and I believe everything is very well worth addressing in order to communicate clearly with the Redis community what’s the road ahead. For simplicity I’ll split this blog post into sections addressing every single feature mentioned in the original comment. ## RESP3 The goal of RESP3, as I already blogged in these pages, is to actually simplify the clients landscape. Hopefully every client will have a lower layer that will not try to reinvent some kind of higher level interface: redis.call(“get”,”foo”). There is no longer need to orchestrate conversions because now the protocol is semantical enough to tell the client what a given reply should look like in the hand of the caller, nor any need to know beforehand the command fingerprint for the majority of commands. What I think the user is referring is RESP3 support for out of band communications, that is the reply “attributes”. I really believe that in the future of Redis “client side caching” will be a big thing. It’s the logical step in every scalable system. However without server assistance client side cache invalidation is a nightmare. This is the reason why RESP3 supports attributes in replies, mainly. However probably Redis 6 *will not implement any of that*. Redis unstable, that will become Redis 6, already has a RESP3 implementation that is almost complete, and there are no attributes. The clients implementing RESP3 can just decide to discard attributes if they are willing to be really future-proof, and likely attributes will not be sent at all anyway even for future Redis versions if the user did not activate some kind of special feature. For instance, for client side caching, the connection will have to be put in some special mode. Moreover, as you know, Redis 6 will be completely backward compatible with RESP2. Actually I’m starting to believe that RESP2 support will never be removed, because it is almost for free, and there is no good reason to break backward compatibility once we did the effort to implement the abstraction layer between RESP2 and RESP3. Normally I don’t like to change things without a good reason, however RESP2 limitations were having a strong effect on the client ecosystem. I would like to have a client landscape where users, going from one client to the other, will feel at home, and the API will be the Redis API, not the layer that the client author invented. I’m not against an *higher level* API in addition to the lower level one btw, but there should be a common ground, and clients should be able to send commands without knowing anything about such commands. ## ACLs The ACL specification was redacted by myself four years ago. I waited so much time in order to convince myself this was really the time to implement it: we went a long way without any ACL using just tricks, mainly command renaming. However don’t believe that ACLs main motivation is enterprise customers in need for security. As a side effect, ACLs also allow authentication of users for security purposes, but the main goal of the feature is *operational*. Let me show you an example. You have a Redis instance and you plan to use the instance to do a new thing: delayed jobs processing. You get a library from the internet, and it looks to work well. Now why on the earth such library, that you don’t know line by line, should be able to call “FLUSHALL” and flush away your database instantly? Maybe the library test will have such command inside and you realize it when it’s too late. Or maybe you just hired a junior developer that is keeping calling “KEYS *” on the Redis instance, while your company Redis policy is “No KEYS command”. Another scenario, cloud providers: they need to carefully rename the admin commands, and even to mask such commands from being leaked for some reason. More tricks: so MONITOR will not show the commands in the output for instance. With ACLs you can setup Redis so that default users, without some authentication, will be prevented to run anything that is administrative or dangerous. I think this will be a big improvement for operations. Moreover ACLs is one of the best code I wrote for Redis AFAIK. There is nearly no CPU cost at all, unless you se key patterns, but even so it’s small. The implementation is completely self contained inside the acl.c file, the rest of the core has a handful of calls to the ACL API. No complexity added to the system because it is completely modular. Actually the ACL code allowed to do some good refactoring around the AUTH command. ## Multi threading There are two possible multi threading supports that Redis could get. I believe the user is referring to “memcached alike” multithreading, that is the ability to scale a single Redis instance to multiple threads in order to increase the operations per second it can deliver in things like GET or SET and other simple commands. This involves making the I/O, command parsing and so forth multi threaded. So let’s call this thing “I/O threading”. Another multi threaded approach is to, instead, allow slow commands to be executed in a different thread, so that other clients are not blocked. We’ll call this threading model “Slow commands threading”. Well, that’s the plan: I/O threading is not going to happen in Redis AFAIK, because after much consideration I think it’s a lot of complexity without a good reason. Many Redis setups are network or memory bound actually. Additionally I really believe in a share-nothing setup, so the way I want to scale Redis is by improving the support for multiple Redis instances to be executed in the same host, especially via Redis Cluster. The things that will happen in 2019 about that are two: A) Redis Cluster multiple instances will be able to orchestrate to make a judicious use of the disk of the local instance, that is, let’s avoid an AOF rewrite at the same time. B) We are going to ship a Redis Cluster proxy as part of the Redis project, so that users are able to abstract away a cluster without having a good implementation of the Cluster protocol client side. Another thing to note is that Redis is not Memcached, but, like memcached, is an in-memory system. To make multithreaded an in-memory system like memcached, with a very simple data model, makes a lot of sense. A multi-threaded on-disk store is mandatory. A multi-threaded complex in-memory system is in the middle where things become ugly: Redis clients are not isolated, and data structures are complex. A thread doing LPUSH need to serve other threads doing LPOP. There is less to gain, and a lot of complexity to add. What instead I *really want* a lot is slow operations threading, and with the Redis modules system we already are in the right direction. However in the future (not sure if in Redis 6 or 7) we’ll get key-level locking in the module system so that threads can completely acquire control of a key to process slow operations. Now modules can implement commands and can create a reply for the client in a completely separated way, but still to access the shared data set a global lock is needed: this will go away. ## Better persistence Recently we did multiple efforts in order to improve this kind of fundamental functions of Redis. One of the best thing that was implemented lately is the RDB preamble inside the AOF file. Also a lot of work went both in Redis 4 and 5 about replication, that is now completely at another level compared to what it used to be. And yes, it is still one of my main focus to improve such parts. ## Data structures Now Redis has Streams, starting with Redis 5. For Redis 6 and 7 what is planned is, to start, to make what we have much more memory efficient by changing the implementations of certain things. However to add new data structures there are a lot of considerations to do. It took me years to realize how to fill the gap, with streams, between lists, pub/sub and sorted sets, in the context of time series and streaming. I really want Redis to be a set of orthogonal data structures that the user can put together, and not a set of *tools* that are ready to use. Streams are an abstract log, so I think it’s a very worthwhile addition. However other things I’m not completely sure if they are worth to be inside the core without a very long consideration. Anyway in the latest years there was definitely more stress in adding new data structures. HyperLogLogs, more advanced bit operations, streams, blocking sorted set operations (ZPOP* and BZPOP*), and streams are good examples. ## Conclusions I believe that the Redis community should be aware about why something is done and why something is instead postponed. I do the error to communicate a lot via Twitter like if everybody is there, but many people happen to have a life :-D and don’t care. The blog is a much better way to inform the community, I need to take the time to blog more. Incidentally I love to write posts, so it’s a win-win. An important thing to realize is that Redis has not a solid roadmap, over the years I found that opportunistic development is a huge win over having a roadmap. Something is demanded? I see the need? I’m in the mood to code it? It’s the right moment because there are no other huge priorities? There are a set of users that are helping the design process, giving hints, ideas, testing stuff? It’s the right moment, let’s do it. To have a solid roadmap for Redis is silly because the size of the OSS core team is small, sometimes I remain stuck with some random crash for weeks… Any fixed long term plan would not work. Moreover as the Redis community gives feedbacks my ideas change a lot, so I would rewrite the roadmap every month. Yet blogging is a good solution to at least show what is the current version of the priorities / ideas, and to show why other ideas were abandoned. A final note: the level of freedom I've with Redis Labs about what to put inside the open source project side is almost infinite. I think this is kinda of a miracle in the industry, or just the people I work with at Redis Labs are nice folks that understand that what we are doing originated from the open source movement and is wise to keep it going in that way. But it's not a common thing. If I do errors in the Redis roadmap they are surely my errors. Comments

• Why RESP3 will be the only protocol supported by Redis 6
  Nov 09, 2018

  [EDIT! I'm reconsidering all this because Marc Gravell from Stack Overflow suggested that we could just switch protocol for backward compatibility per-connection, sending a command to enable RESP3. That means no longer need for a global configuration that switches the behavior of the server. Put in that way it is a lot more acceptable for me, and I'm reconsidering the essence of the blog post] A few weeks after the release of Redis 5, I’m here starting to implement RESP3, and after a few days of work it feels very well to see this finally happening. RESP3 is the new client-server protocol that Redis will use starting from Redis 6. The specification at https://github.com/antirez/resp3 should explain in clear terms how this evolution of our old protocol, RESP2, should improve the Redis ecosystem. But let’s say that the most important thing is that RESP3 is more “semantic” than RESP2. For instance it has the concept of maps, sets (unordered lists of elements), attributes of the returned data, that may augment the reply with auxiliary information, and so forth. The final goal is to make new Redis clients have less work to do for us, that is, just deciding a set of fixed rules in order to convert every reply type from RESP3 to a given appropriate type of the client library programming language. In the future of Redis I see clients that are smarter under the hood, trying to do their best in order to handle connections, pipelining, and state, and apparently a lot more simpler in the user-facing side, to the point that the ideal Redis client is like: result = redis.call(“GET”,keyname); Of course on top of that you can build more advanced abstractions, but the bottom layer should look like that, and the returned reply should not require any filtering that is ad-hoc for specific commands: RESP3 return type should contain enough information to return an appropriate data type. So HGETALL will return a RESP3 “map”, while LRANGE will return an “array”, and EXISTS will return a RESP3 “boolean”. This also allows new commands to work as expected even if the client library was not *specifically* designed to handle it. With RESP2 instead what happened was that likely the command worked using mechanisms like "method missing" or similar, but later when the command was *really* implemented in the client library, the returned type changed, introducing a subtle incompatibility. However, while the new protocol is an incremental improvement over the old one, it will introduce breaking incompatibilities in the client-library side (of course) and *in the application layer* as well. Because for instance, ZSCORE will now return a double, and not a string, so application code should be updated, or, alternatively, client libraries could implement a compatibility option that will turn the RESP3 replies back to their original RESP2 types. Lua scripts will also no longer work if not modified for the new protocol, because also Lua will see more semantical types returned by the redis.call() command. Similarly Lua will be able to return all the new data types implemented in RESP3. Because of all that, people are scared about my decision: I’m going to ship Redis 6 with support for *only* RESP3. There will be no compatibility mode to switch a Redis 6 server to RESP2, so you either upgrade your client library and upgrade your application (or use the client library backward compatibility mode), or you cannot switch to Redis 6. I’ve good reasons to do so, and I want to explain why I’m taking this decision, and how I’m mitigating the problems for users and client library authors. Let’s start from the mitigations: * Redis 5 will be fully supported for 2 years after the release of Redis 6. Everything critical will be back-ported to Redis 5 and patch-level releases will be available constantly. * Redis 6 is expected to be released in about 1 or 1.5 years. However Redis 6 will be switched to RESP3 in about 1 month. So people will use, experiment, and deal with an unstable Redis version that uses the new protocol for a lot of time. Given that unlike many other softwares, Redis unstable has a lot of casual users, both because it’s the default branch on Github, and because traditionally Redis unstable is never really so unstable, this will grant a lot of prior exposure. * I’m still not 100% sure about that, but the Lua scripting engine may have a compatibility mode in order to return the same types as of Redis 5. The compatibility however will not be enabled by default, and will be opt-in for each script executed, by calling a special redis.resp2_compat() function before calling Redis commands. So every Redis 6 server will behave the same regardless of its configuration, as Redis always did in the last 10 years. Those are the mitigations. And this is, instead, why I’ll not have Redis 6 supporting both versions: 1) It is more or less completely useless. If people switch Redis 6 to RESP2 mode, they are still in the past and are just waiting for Redis 7 to go out without RESP2 support and break everything. In the meantime, when you deal with a Redis 6 installation, you never know *what it replies*, depending on how it is configured. So the same client library may return an Hash or an Array for the same command. 2) It’s more work and more complexity without a good reason (see “1”). Many commands will require a check for the old protocol in order to see in what format to reply. 3) By binding the new Redis 6 features together with a protocol change, we are giving good reasons to users to do the switch and port their clients and applications. At some point everything will be over and we can focus on new things. Otherwise we’ll have a set of Redis 6 users that switched to the new server for the new features but are still with the old protocol, and Redis 7 will be the same drama again. 4) If somebody tells you that adapting the client libraries is a terrible work, well, I’ll beg to differ. Yes, there is some change to do, but now that I’m implementing the server side, I see that it’s not so terrible. What is terrible instead is that most client work is not payed at all and happens just because of passion and willingness to share with others. I bet that we’ll see many implementations of RESP3 in short time. 5) RESP3 is designed so that clients can automatically detect if it’s RESP2 or RESP3, and switch, so new clients will work both with Redis Well that’s all. I hope it clarifies my point of view and the reasons behind it, and also at the same time the mitigations that will be enabled during the protocol switch may serve to convince users that it will not be a very “hard” breakage. Comments

• Writing system software: code comments.
  Oct 06, 2018

  For quite some time I’ve wanted to record a new video talking about code comments for my "writing system software" series on YouTube. However, after giving it some thought, I realized that the topic was better suited for a blog post, so here we are. In this post I analyze Redis comments, trying to categorize them. Along the way I try to show why, in my opinion, writing comments is of paramount importance in order to produce good code, that is maintainable in the long run and understandable by others and by the authors during modifications and debugging activities. Not everybody thinks likewise. Many believe that comments are useless if the code is solid enough. The idea is that when everything is well designed, the code itself documents what the code is doing, hence code comments are superfluous. I disagree with that vision for two main reasons: 1. Many comments don't explain what the code is doing. They explain what you can't understand just from what the code does. Often this missing information is *why* the code is doing a certain action, or why it’s doing something that is clear instead of something else that would feel more natural. 2. While it is not generally useful to document, line by line, what the code is doing, because it is understandable just by reading it, a key goal in writing readable code is to lower the amount of effort and the number of details the reader should take into her or his head while reading some code. So comments can be, for me, a tool for lowering the cognitive load of the reader. The following code snippet is a good example of the second point above. Note that all the code snippets in this blog post are obtained from the Redis source code. Every code snipped is presented prefixed by the file name it was extracted from. The branch used is the current "unstable" with hash 32e0d237. scripting.c: /* Initial Stack: array */ lua_getglobal(lua,"table"); lua_pushstring(lua,"sort"); lua_gettable(lua,-2); /* Stack: array, table, table.sort */ lua_pushvalue(lua,-3); /* Stack: array, table, table.sort, array */ if (lua_pcall(lua,1,0,0)) { /* Stack: array, table, error */ /* We are not interested in the error, we assume that the problem is * that there are 'false' elements inside the array, so we try * again with a slower function but able to handle this case, that * is: table.sort(table, __redis__compare_helper) */ lua_pop(lua,1); /* Stack: array, table */ lua_pushstring(lua,"sort"); /* Stack: array, table, sort */ lua_gettable(lua,-2); /* Stack: array, table, table.sort */ lua_pushvalue(lua,-3); /* Stack: array, table, table.sort, array */ lua_getglobal(lua,"__redis__compare_helper"); /* Stack: array, table, table.sort, array, __redis__compare_helper */ lua_call(lua,2,0); } Lua uses a stack based API. A reader following each call in the function above, having also a Lua API reference at hand, will be able to mentally reconstruct the stack layout at every given moment. But why to force the reader to do such effort? While writing the code, the original author had to do that mental effort anyway. What I did there was just to annotate every line with the current stack layout after every call. Reading this code is now trivial, regardless of the fact the Lua API is otherwise non trivial to follow. My goal here is not just to offer my point of view on the usefulness of comments as a tool to provide a background that is not clearly available reading a local section of the source code. But also to also provide some evidence about the usefulness of the kind of comments that are historically considered useless or even dangerous, that is, comments stating *what* the code is doing, and not why. # Classification of comments The way I started this work was by reading random parts of the Redis source code, to check if and why comments were useful in different contexts. What quickly emerged was that comments are useful for very different reasons, since they tend to be very different in function, writing style, length and update frequency. I eventually turned the work into a classification task. During my research I identified nine types of comments: * Function comments * Design comments * Why comments * Teacher comments * Checklist comments * Guide comments * Trivial comments * Debt comments * Backup comments The first six are, in my opinion, mostly very positive forms of commenting, while the final three are somewhat questionable. In the next sections each type will be analyzed with examples from the Redis source code. FUNCTION COMMENTS The goal of a function comment is to prevent the reader from reading code in the first place. Instead, after reading the comment, it should be possible to consider some code as a black box that should obey certain rules. Normally function comments are at the top of functions definitions, but they may be at other places, documenting classes, macros, or other functionally isolated blocks of code that define some interface. rax.c: /* Seek the grestest key in the subtree at the current node. Return 0 on * out of memory, otherwise 1. This is an helper function for different * iteration functions below. */ int raxSeekGreatest(raxIterator *it) { ... Function comments are actually a form of in-line API documentation. If the function comment is written well enough, the user should be able most of the times to jump back to what she was reading (reading the code calling such API) without having to read the implementation of a function, a class, a macro, or whatever. Among all the kinds of comments, these are the ones most widely accepted by the programming community at large as needed. The only point to analyze is if it is a good idea to place comments that are largely API reference documentation inside the code itself. For me the answer is simple: I want the API documentation to exactly match the code. As the code is changed, the documentation should be changed. For this reason, by using function comments as a prologue of functions or other elements, we make the API documentation close to the code, accomplishing three results: * As the code is changed, the documentation can be easily changed at the same time, without the risk of making the API reference stale. * This approach maximizes the probability that the author of the change, that should be the one better understanding the change, will also be the author of the API documentation change. * Reading the code is handy to find the documentation of functions or methods directly where they are defined, so that the reader of the code can focus solely on the code, instead of context switching between code and documentation. DESIGN COMMENTS While a "function comment" is usually located at the start of a function, a design comment is more often located at the start of a file. The design comment basically states how and why a given piece of code uses certain algorithms, techniques, tricks, and implementation. It is an higher level overview of what you'll see implemented in the code. With such background, reading the code will be simpler. Moreover I tend to trust more code where I can find design notes. At least I know that some kind of explicit design phase happened, at some point, during the development process. In my experience design comments are also very useful in order to state, in case the solution proposed by the implementation looks a bit too trivial, what were the competing solutions and why a very simple solution was considered to be enough for the case at hand. If the design is correct, the reader will convince herself that the solution is appropriate and that such simplicity comes from a process, not from being lazy or only knowing how to code basic things. bio.c: * DESIGN * ------ * * The design is trivial, we have a structure representing a job to perform * and a different thread and job queue for every job type. * Every thread waits for new jobs in its queue, and process every job * sequentially. ... WHY COMMENTS Why comments explain the reason why the code is doing something, even if what the code is doing is crystal clear. See the following example from the Redis replication code. replication.c: if (idle > server.repl_backlog_time_limit) { /* When we free the backlog, we always use a new * replication ID and clear the ID2. This is needed * because when there is no backlog, the master_repl_offset * is not updated, but we would still retain our replication * ID, leading to the following problem: * * 1. We are a master instance. * 2. Our replica is promoted to master. It's repl-id-2 will * be the same as our repl-id. * 3. We, yet as master, receive some updates, that will not * increment the master_repl_offset. * 4. Later we are turned into a replica, connect to the new * master that will accept our PSYNC request by second * replication ID, but there will be data inconsistency * because we received writes. */ changeReplicationId(); clearReplicationId2(); freeReplicationBacklog(); serverLog(LL_NOTICE, "Replication backlog freed after %d seconds " "without connected replicas.", (int) server.repl_backlog_time_limit); } If I check just the function calls there is very little to wonder: if a timeout is reached, change the main replication ID, clear the secondary ID, and finally free the replication backlog. However what is not exactly clear is why we need to change the replication IDs when freeing the backlog. Now this is the kind of thing that happens continuously in software once it has reached a given level of complexity. Regardless of the code involved, the replication protocol has some level of complexity itself, so we need to do certain things in order to make sure that other bad things can't happen. Probably these kind of comments are, in some way, opportunities to reason about the system and check if it should be improved, so that such complexity is no longer needed, hence also the comment can be removed. However often making something simpler may make something else harder or is simply not viable, or requires future work breaking backward compatibility. Here is another one. replication.c: /* SYNC can't be issued when the server has pending data to send to * the client about already issued commands. We need a fresh reply * buffer registering the differences between the BGSAVE and the current * dataset, so that we can copy to other replicas if needed. */ if (clientHasPendingReplies(c)) { addReplyError(c,"SYNC and PSYNC are invalid with pending output"); return; } If you run SYNC while there is still pending output (from a past command) to send to the client, the command should fail because during the replication handshake the output buffer of the client is used to accumulate changes, and may be later duplicated to serve other replicas connecting while we are already creating the RDB file for the full sync with the first replica. This is the why we do that. What we do is trivial. Pending replies? Emit an error. Why is rather obscure without the comment. One may think that such comments are needed only when describing complex protocols and interactions, like in the case of replication. Is that the case? Let's change completely file and goals, and we see still such comments everywhere. expire.c: for (j = 0; j int expired; redisDb *db = server.db+(current_db % server.dbnum); /* Increment the DB now so we are sure if we run out of time * in the current DB we'll restart from the next. This allows to * distribute the time evenly across DBs. */ current_db++; ... That's an interesting one. We want to expire keys from different DBs, as long as we have some time. However instead of incrementing the “database ID” to process next at the end of the loop processing the current database, we do it differently: we select the current DB in the `db` variable, but then we immediately increment the ID of the next database to process (at the next call of this function). This way if the function terminates because too much effort was spent in a single call, we don't have the problem of restarting again from the same database, letting logically expired keys accumulating in the other databases since we are too focused in processing the same database again and again. With such comment we both explain why we increment at that stage, and that the next person going to modify the code, should preserve such quality. Note that without the comment the code looks completely harmless. Select, increment, go to do some work. There is no evident reason for not relocating the increment at the end of the loop where it could look more natural. Trivia: the loop increment was indeed at the end in the original code. It was moved there during a fix: at the same time the comment was added. So let's say this is kinda of a "regression comment". TEACHER COMMENTS Teacher comments don't try to explain the code itself or certain side effects we should be aware of. They teach instead the *domain* (for example math, computer graphics, networking, statistics, complex data structures) in which the code is operating, that may be one outside of the reader skills set, or is simply too full of details to recall all them from memory. The LOLWUT command in version 5 needs to display rotated squares on the screen (http://antirez.com/news/123). In order to do so it uses some basic trigonometry: despite the fact that the math used is simple, many programmers reading the Redis source code may not have any math background, so the comment at the top of the function explains what's going to happen inside the function itself. lolwut5.c: /* Draw a square centered at the specified x,y coordinates, with the specified * rotation angle and size. In order to write a rotated square, we use the * trivial fact that the parametric equation: * * x = sin(k) * y = cos(k) * * Describes a circle for values going from 0 to 2*PI. So basically if we start * at 45 degrees, that is k = PI/4, with the first point, and then we find * the other three points incrementing K by PI/2 (90 degrees), we'll have the * points of the square. In order to rotate the square, we just start with * k = PI/4 + rotation_angle, and we are done. * * Of course the vanilla equations above will describe the square inside a * circle of radius 1, so in order to draw larger squares we'll have to * multiply the obtained coordinates, and then translate them. However this * is much simpler than implementing the abstract concept of 2D shape and then * performing the rotation/translation transformation, so for LOLWUT it's * a good approach. */ The comment does not contain anything that is related to the code of the function itself, or its side effects, or the technical details related to the function. The description is only limited to the mathematical concept that is used inside the function in order to reach a given goal. I think teacher comments are of huge value. They teach something in case the reader is not aware of such concepts, or at least provide a starting point for further investigation. But this in turn means that a teacher comment increases the amount of programmers that can read some code path: writing code that can be read by many programmers is a major goal of mine. There are developers that may not have math skills but are very solid programmers that can contribute some wonderful fix or optimization. And in general code should be read other than being executed, since is written by humans for other humans. There are cases where teacher comments are almost impossible to avoid in order to write decent code. A good example is the Redis radix tree implementation. Radix trees are articulated data structures. The Redis implementation re-states the whole data structure theory as it implements it, showing the different cases and what the algorithm does to merge or split nodes and so forth. Immediately after each section of comment, we have the code implementing what was written before. After months of not touching the file implementing the radix tree, I was able to open it, fix a bug in a few minutes, and continue doing something else. There is no need to study again how a radix tree works, since the explanation is the same thing as the code itself, all mixed together. The comments are too long, so I'll just show certain snippets. rax.c: /* If the node we stopped at is a compressed node, we need to * split it before to continue. * * Splitting a compressed node have a few possible cases. * Imagine that the node 'h' we are currently at is a compressed * node contaning the string "ANNIBALE" (it means that it represents * nodes A -> N -> N -> I -> B -> A -> L -> E with the only child * pointer of this node pointing at the 'E' node, because remember that * we have characters at the edges of the graph, not inside the nodes * themselves. * * In order to show a real case imagine our node to also point to * another compressed node, that finally points at the node without * children, representing 'O': * * "ANNIBALE" -> "SCO" -> [] ... snip ... * 3a. IF $SPLITPOS == 0: * Replace the old node with the split node, by copying the auxiliary * data if any. Fix parent's reference. Free old node eventually * (we still need its data for the next steps of the algorithm). * * 3b. IF $SPLITPOS != 0: * Trim the compressed node (reallocating it as well) in order to * contain $splitpos characters. Change chilid pointer in order to link * to the split node. If new compressed node len is just 1, set * iscompr to 0 (layout is the same). Fix parent's reference. ... snip ... if (j == 0) { /* 3a: Replace the old node with the split node. */ if (h->iskey) { void *ndata = raxGetData(h); raxSetData(splitnode,ndata); } memcpy(parentlink,&splitnode,sizeof(splitnode)); } else { /* 3b: Trim the compressed node. */ trimmed->size = j; memcpy(trimmed->data,h->data,j); trimmed->iscompr = j > 1 ? 1 : 0; trimmed->iskey = h->iskey; trimmed->isnull = h->isnull; if (h->iskey && !h->isnull) { void *ndata = raxGetData(h); raxSetData(trimmed,ndata); } raxNode **cp = raxNodeLastChildPtr(trimmed); ... As you can see the description in the comment is then matched with the same labels in the code. It's hard to show it all in this form so if you want to get the whole idea just check the full file at: https://github.com/antirez/redis/blob/unstable/src/rax.c This level of commenting is not needed for everything, but things like radix trees are really full of little details and corner cases. They are hard to recall, and certain details are *specific* to a given implementation. Doing this for a linked list does not make much sense of course. It's a matter of personal sensibility to understand when it's worth it or not. CHECKLIST COMMENTS This is a very common and odd one: sometimes because of language limitations, design issues, or simply because of the natural complexity arising in systems, it is not possible to centralize a given concept or interface in one piece, so there are places in the code that tells you to remember to do things in some other place of the code. The general concept is: /* Warning: if you add a type ID here, make sure to modify the * function getTypeNameByID() as well. */ In a perfect world this should never be needed, but in practice sometimes there are no escapes from that. For example Redis types could be represented using an "object type" structure, and every object could link to the type the object it belongs, so you could do: printf("Type is %s\n", myobject->type->name); But guess what? It's too expensive for us, because a Redis object is represented like this: typedef struct redisObject { unsigned type:4; unsigned encoding:4; unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or * LFU data (least significant 8 bits frequency * and most significant 16 bits access time). */ int refcount; void *ptr; } robj; We use 4 bits instead of 64 to represent the type. This is just to show why sometimes things are not as centralized and natural as they should be. When the situation is like that, sometimes what helps is to use defensive commenting in order to make sure that if a given code section is touched, it reminds you to make sure to also modify other parts of the code. Specifically a checklist comment does one or both of the following things: * It tells you a set of actions to do when something is modified. * It warns you about the way certain changes should be operated. Another example in blocked.c, when a new blocking type is introduced. blocked.c: * When implementing a new type of blocking opeation, the implementation * should modify unblockClient() and replyToBlockedClientTimedOut() in order * to handle the btype-specific behavior of this two functions. * If the blocking operation waits for certain keys to change state, the * clusterRedirectBlockedClientIfNeeded() function should also be updated. The checklist comment is also useful in a context similar to when certain "why comments" are used: when it is not obvious why some code must be executed at a given place, after or before something. But while the why comment may tell you why a statement is there, the checklist comment used in the same case is more biased towards telling you what rules to follow if you want to modify it (in this case the rule is, follow a given ordering), without breaking the code behavior. cluster.c: /* Update our info about served slots. * * Note: this MUST happen after we update the master/replica state * so that CLUSTER_NODE_MASTER flag will be set. */ Checklist comments are very common inside the Linux kernel, where the order of certain operations is extremely important. GUIDE COMMENT I abuse guide comments at such a level that probably, the majority of comments in Redis are guide comments. Moreover guide comments are exactly what most people believe to be completely useless comments. * They don't state what is not clear from the code. * There are no design hints in guide comments. Guide comments do a single thing: they babysit the reader, assist him or her while processing what is written in the source code by providing clear division, rhythm, and introducing what you are going to read. Guide comments’ sole reason to exist is to lower the cognitive load of the programmer reading some code. rax.c: /* Call the node callback if any, and replace the node pointer * if the callback returns true. */ if (it->node_cb && it->node_cb(&it->node)) memcpy(cp,&it->node,sizeof(it->node)); /* For "next" step, stop every time we find a key along the * way, since the key is lexicographically smaller compared to * what follows in the sub-children. */ if (it->node->iskey) { it->data = raxGetData(it->node); return 1; } There is nothing that the comments are adding to the code above. The guide comments above will assist you reading the code, moreover they'll acknowledge you about the fact you are understanding it right. More examples. networking.c: /* Log link disconnection with replica */ if ((c->flags & CLIENT_SLAVE) && !(c->flags & CLIENT_MONITOR)) { serverLog(LL_WARNING,"Connection with replica %s lost.", replicationGetSlaveName(c)); } /* Free the query buffer */ sdsfree(c->querybuf); sdsfree(c->pending_querybuf); c->querybuf = NULL; /* Deallocate structures used to block on blocking ops. */ if (c->flags & CLIENT_BLOCKED) unblockClient(c); dictRelease(c->bpop.keys); /* UNWATCH all the keys */ unwatchAllKeys(c); listRelease(c->watched_keys); /* Unsubscribe from all the pubsub channels */ pubsubUnsubscribeAllChannels(c,0); pubsubUnsubscribeAllPatterns(c,0); dictRelease(c->pubsub_channels); listRelease(c->pubsub_patterns); /* Free data structures. */ listRelease(c->reply); freeClientArgv(c); /* Unlink the client: this will close the socket, remove the I/O * handlers, and remove references of the client from different * places where active clients may be referenced. */ unlinkClient(c); Redis is *literally* ridden of guide comments, so basically every file you open will contain plenty of them. Why bother? Of all the comment types I analyzed so far in this blog post, I'll admit that this is absolutely the most subjective one. I don't value code without such comments as less good, yet I firmly believe that if people regard the Redis code as readable, some part of the reason is because of all the guide comments. Guide comments have some usefulness other than the stated ones. Since they clearly divide the code in isolated sections, an addition to the code is very likely to be inserted in the appropriate section, instead of ending in some random part. To have related statements nearby is a big readability win. Also make sure to check the guide comment above before the unlinkClient() function is called. The guide comment briefly tells the reader what the function is going to do, avoiding the need to jump back into the function if you are only interested in the big picture. TRIVIAL COMMENTS Guide comments are very subjective tools. You may like them or not. I love them. However, a guide comment can degenerate into a a very bad comment: it can easily turn into a "trivial comment". A trivial comment is a guide comment where the cognitive load of reading the comment is the same or higher than just reading the associated code. The following form of trivial comment is exactly what many books will tell you to avoid. array_len++; /* Increment the length of our array. */ So if you write guide comments, make sure you avoid writing trivial ones. DEBT COMMENTS Debt comments are technical debts statements hard coded inside the source code itself: t_stream.c: /* Here we should perform garbage collection in case at this point * there are too many entries deleted inside the listpack. */ entries -= to_delete; marked_deleted += to_delete; if (entries + marked_deleted > 10 && marked_deleted > entries/2) { /* TODO: perform a garbage collection. */ } The snippet above is extracted from the Redis streams implementation. Redis streams allow to delete elements from the middle using the XDEL command. This may be useful in different ways, especially in the context of privacy regulations where certain data cannot be retained no matter what data structure or system you are using in order to store them. It is a very odd use case for a mostly append only data structure, but if users start to delete more than 50% of items in the middle, the stream starts to fragment, being composed of "macro nodes". Entries are just flagged as deleted, but are only reclaimed once all the entries in a given macro node are freed. So your mass deletions will change the memory behavior of streams. Right now, this looks like a non issue, since I don't expect users to delete most history in a stream. However it is possible that in the future we may want to introduce garbage collection: the macro node could be compacted once the ratio between the deleted entries and the existing entries reach a given level. Moreover nearby nodes may be glued together after the garbage collection. I was kind of afraid that later I would no longer remember what were the entry points to do the garbage collection, so I put TODO comments, and even wrote the trigger condition. This is probably not great. A better idea was instead to write, in the design comment at the top of the file, why we are currently not performing GC. And what are the entry points for GC, if we want to add it later. FIXME, TODO, XXX, "This is a hack", are all forms of debt comments. They are not great in general, I try to avoid them, but it's not always possible, and sometimes instead of forgetting forever about a problem, I prefer to put a node inside the source code. At least one should periodically grep for such comments, and see if it is possible to put the notes in a better place, or if the problem is no longer relevant or could be fixed right away. BACKUP COMMENTS Finally backup comments are the ones where the developer comments older versions of some code block or even a whole function, because she or he is insecure about the change that was operated in the new one. What is puzzling is that this still happens now that we have Git. I guess people have an uneasy feeling about losing that code fragment, considered more sane or stable, in some years old commit. But source code is not for making backups. If you want to save an older version of a function or code part, your work is not finished and cannot be committed. Either make sure the new function is better than the past one, or take it just in your development tree until you are sure. Backup comments end my classification. Let's try some conclusion. # Comments as an analysis tool. Comments are rubber duck debugging on steroids, except you are not talking with a rubber duck, but with the future reader of the code, which is more intimidating than a rubber duck, and can use Twitter. So in the process you really try to understand if what you are stating *is acceptable*, honorable, good enough. And if it is not, you make your homework, and come up with something more decent. It is the same process that happens while writing documentation: the writer attempts to provide the gist of what a given piece of code does, what are the guarantees, the side effects. This is often a bug hunting opportunity. It is very easy while describing something to find that it has holes... You can't really describe it all because you are not sure about a given behavior: such behavior is just emerging from complexity, at random. You really don't want that, so you go back and fix it all. I find this a splendid reason to write comments. # Writing good comments is harder than writing good code You may think that writing comments is a lesser noble form of work. After all you *can code*! However consider this: code is a set of statement and function calls, or whatever your programming paradigm is. Sometimes such statements do not make much sense, honestly, if the code is not good. Comments require always to have some design process ongoing, and to understand the code you are writing in a deeper sense. On top of that, in order to write good comments, you have to develop your writing skills. The same writing skills will assist you writing emails, documentation, design documents, blog posts, and commit messages. I write code because I have an urgent sense to share and communicate more than anything else. Comments coadiuvate the code, assist it, describe our efforts, and after all I love writing them as much as I love writing code itself. (Thanks to Michel Martens for giving feedbacks during the writing of this blog post) Comments

• LOLWUT: a piece of art inside a database command
  Sep 12, 2018

  The last few days have been quite intense. One of the arguments, about the dispute related to replacing or not the words used in Redis replication with different ones, was the following: is it worthwhile to do work that does not produce any technological result? As I was changing the Redis source code to get rid of a specific word where possible, I started to think that whatever my idea was about the work I was doing, I’m the kind of person that enjoys writing code that has no measurable technological effects. Replacing words is just annoying, even if, even there, there were a few worthwhile technological challenges. But there is some other kind of code that I believe has a quality called “hack value”. It may not solve any technological problem, yet it’s worth to write. Sometimes because the process of writing the code is, itself, rewarding. Other times because very technically advanced ideas are used to solve a not useful problem. Sometimes code is just written for artistic reasons. In some way the Twitter discussion of the last days, mostly uninformed, chaotic, heated, made me think that, at this point, we are very far from the first hackers in the 60s. As I get older I find that it is harder and harder to talk about technology with an hacking perspective, where there are no walls or pre-cooked ideas, and the limit is the exploration. For everything you say there is a best practice. For every idea there is a taboo. To this new setup I say LOLWUT, since I don’t feel represented by it, nor it represents hacking, at least in my vision. So the idea was to spend some technologically useless time in order to explore something of the 60s. My attention went immediately to one of the computer art pieces I love the most: Schotter, by Georg Nees (https://en.wikipedia.org/wiki/Georg_Nees). With the help of a plotter and ALGOL programs, Nees explored writing programs to generate art using caos (randomness) and repeating patterns. Schotter is remarkable because of the simplicity of the piece and the deep meaning that the observer can find looking at it. Under a surface of total calm and order, deep inside the disorder hides. Or, if you put it upside down it becomes like the sea during a tempest. However the surface may look impetuous, the deep sea remains calm. Is it possible to turn a piece of art into a database command? This was challenging because Redis is mostly used by a command line interface. Nowadays terminals are for sure fancier than the ones of the past, but yet to display decent graphics is hard. On the other side there is the huge advantage of the real time computation: a piece of art can be dynamic, changing every time it is generated. Before continuing, I want to show you the final result: img://antirez.com/misc/lolwut1.png While very low resolution the idea of the original piece is still there. To make this possible I used a trick that recently was used by multiple programs trying to display interesting things in a text console. It involves using the Braille unicode charset in order to create a pixel matrix which is more dense than the individual characters of the console. Specifically, for each character, it is possible to fit a 2x8 grid of pixels. The second part of the experiment was to make the art piece parametric: img://antirez.com/misc/lolwut2.png It is possible to generate different versions of the original piece, changing the number of squares and the output resolution. Finally the source code wanted to be an example of literate programming, being written in a form that resembles more a tutorial describing what everything does and why, instead of some opaque generator. You can find the code here: https://github.com/antirez/redis/blob/unstable/src/lolwut.c LOLWUT is also going to be a tradition starting from Redis 5. At each new major version of Redis what the command does will change completely, only a set of rules will be fixed: 1. It can’t do anything technologically useful. 2. It should be fast at doing what it does, so that it is safe to call LOLWUT on production instances. 3. The output should be entertaining in some way. I wrote the first one for Redis 5, for the next versions, if I find interest, I’ll ask somebody else that contributed to Redis to write the other LOLWUT versions, otherwise I’ll write it again myself (but I hope that’s not the case). LOLWUT should remember ourselves that the work we do, programming, did not start just in order to produce something useful. Initially it was mainly a matter of exploring possibilities. I hope that LOLWUT also will remind the Redis community that computers are about humans, and that it is not possible to reason in an aseptic way just thinking at the technological implications. There are people using systems, people building systems, and so forth. Comments

Alex Edwards

• A complete guide to working with Cookies in Go
  Sep 28, 2022

  In this post we're going to run through how to use cookies in your Go web application to persist data between HTTP requests for a specific client. We'll start simple, and slowly build up a working application which covers the following topics: Basic reading and writing of cookies Encoding special characters and maximum length Using tamper-proof (signed) cookies Using confidential (encrypted) and tamper-proof cookies Storing custom data types in cookies Hint: If you're new to web development and need a general introduction to what cookies are and how they work, I recommend reading this MDN article before continuing. If you just want the final code, rather than the explanations, you can find it in this gist. Basic use The first thing to know is that cookies in Go are represented by the http.Cookie type. This is a struct which looks like this: type Cookie struct { Name string Value string Path string Domain string Expires time.Time RawExpires string // MaxAge=0 means no 'Max-Age' attribute specified. // MaxAge0 means Max-Age attribute present and given in seconds MaxAge int Secure bool HttpOnly bool SameSite SameSite Raw string Unparsed []string } Name is the cookie name. It can contain any US-ASCII characters except ( ) @ , ; : \ " / [ ? ] = { } and space, tab and control characters. It is a mandatory field. Value contains the data that you want to persist. It can contain any US-ASCII characters except , ; \ " and space, tab and control characters. It is a mandatory field. Path, Domain, Expires, MaxAge, Secure, HttpOnly and SameSite map directly to the respective cookie attributes. All of these are optional fields. If set, the value of the SameSite field should be one of the SameSite constants from the net/http package. The RawExpires, Raw and Unparsed fields are only used when your Go program is acting as a client (rather than a server) and parsing the cookies from a HTTP response. Most of the time you won't need to use these fields. Cookies can be written in a HTTP response using the http.SetCookie() function, and read from a HTTP request using the *Request.Cookie() method. Let's jump in and use these things in a working example. If you'd like to follow along, please run the following commands to set up a basic project scaffold: $ mkdir example-project $ cd example-project/ $ go mod init example.com/example-project go: creating new go.mod: module example.com/example-project $ touch main.go In the main.go file we're going to create a simple web application with two endpoints: GET /set which writes a new cookie along with the HTTP response. GET /get which reads the cookie sent with the HTTP request and then echoes out the cookie value in the response. Go ahead and add the following code to main.go: File: main.gopackage main import ( "errors" "log" "net/http" ) func main() { // Start a web server with the two endpoints. mux := http.NewServeMux() mux.HandleFunc("/set", setCookieHandler) mux.HandleFunc("/get", getCookieHandler) log.Print("Listening...") err := http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func setCookieHandler(w http.ResponseWriter, r *http.Request) { // Initialize a new cookie containing the string "Hello world!" and some // non-default attributes. cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello world!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Use the http.SetCookie() function to send the cookie to the client. // Behind the scenes this adds a `Set-Cookie` header to the response // containing the necessary cookie data. http.SetCookie(w, &cookie) // Write a HTTP response as normal. w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Retrieve the cookie from the request using its name (which in our case is // "exampleCookie"). If no matching cookie is found, this will return a // http.ErrNoCookie error. We check for this, and return a 400 Bad Request // response to the client. cookie, err := r.Cookie("exampleCookie") if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } // Echo out the cookie value in the response body. w.Write([]byte(cookie.Value)) } Note: As an aside, we have set the Secure attribute on the cookie to true. This indicates to the client (usually a web browser) that the cookie should only be used for 'secure' connections. Generally speaking this means that the cookie should only be used for encrypted connections (i.e. HTTPS), but many modern browsers (including Firefox and Chrome) also consider unencrypted connections to localhost to be 'secure'. This means that the cookie should work on localhost even if our web application is only using HTTP. OK, let's try this out. Go ahead and run the application: $ go run . 2022/09/25 10:44:11 Listening... And then open http://localhost:3000/set in your web browser. You should see the "cookie set!" response and, if you have developer tools open, you should also see the Set-Cookie header containing the data in the HTTP response headers. Then if you visit http://localhost:3000/get, our exampleCookie cookie should be passed back along with the HTTP request, and our getCookieHandler will retrieve the cookie value and print it in the response. Like so: If you want, you can also make a request to http://localhost:3000/set using curl to see the contents of the Set-Cookie header. Like so: $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie="Hello world!"; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Sun, 25 Sep 2022 08:45:02 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! Encoding special characters and maximum length So far, so good! But there are a couple of important things to be aware of when writing cookies. As mentioned briefly above, cookie values must only contain a subset of the US-ASCII characters. If you try to use an unsupported character, Go will strip it out before setting the Set-Cookie header. Let's try this out by adapting our setCookieHandler to write a cookie value containing a non US-ASCII character like "Hello Zoë!" (notice the umlauted ë character): File: main.go package main ... func setCookieHandler(w http.ResponseWriter, r *http.Request) { cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } http.SetCookie(w, &cookie) w.Write([]byte("cookie set!")) } ... Then when you make a request to http://localhost:3000/set, you'll see that the cookie value has been stripped down to "Hello Zo!". $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie="Hello Zo!"; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Sun, 25 Sep 2022 09:00:03 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! A good way to avoid this kind of problem is to base64-encode your cookie values before writing them. Because the base64 character set is a subset of the US-ASCII characters supported in cookies, we can be confident that nothing will be stripped from the cookie value. Another thing to be aware of is that web browsers impose a maximum size limit on cookies. But this limit — and how the cookie size is calculated — depends on the browser version being used. To prevent problems, a good rule-of-thumb is to keep the total size of the cookie (including all attributes) to no more than 4096 bytes. If you try to send a cookie larger than 4096 bytes, Go will write the Set-Cookie header without any problems (it won't be truncated), but there is a risk that the client may truncate or reject the cookie. To help with these two potential problems, let's create an internal/cookies package containing a couple of helper functions: A Write() function which encodes a cookie value to base64 and checks that the total length of the cookie is no more than 4096 bytes before writing it. A Read() function which reads a cookie from the current request and decodes the cookie value from base64. $ mkdir -p internal/cookies $ touch internal/cookies/cookies.go File: internal/cookies/cookies.gopackage cookies import ( "encoding/base64" "errors" "net/http" ) var ( ErrValueTooLong = errors.New("cookie value too long") ErrInvalidValue = errors.New("invalid cookie value") ) func Write(w http.ResponseWriter, cookie http.Cookie) error { // Encode the cookie value using base64. cookie.Value = base64.URLEncoding.EncodeToString([]byte(cookie.Value)) // Check the total length of the cookie contents. Return the ErrValueTooLong // error if it's more than 4096 bytes. if len(cookie.String()) > 4096 { return ErrValueTooLong } // Write the cookie as normal. http.SetCookie(w, &cookie) return nil } func Read(r *http.Request, name string) (string, error) { // Read the cookie as normal. cookie, err := r.Cookie(name) if err != nil { return "", err } // Decode the base64-encoded cookie value. If the cookie didn't contain a // valid base64-encoded value, this operation will fail and we return an // ErrInvalidValue error. value, err := base64.URLEncoding.DecodeString(cookie.Value) if err != nil { return "", ErrInvalidValue } // Return the decoded cookie value. return string(value), nil } Then we can update our main.go file to use these new helpers, like so: File: main.gopackage main import ( "errors" "log" "net/http" "example.com/example-project/internal/cookies" // Import the internal/cookies package. ) ... func setCookieHandler(w http.ResponseWriter, r *http.Request) { // Initialize the cookie as normal. cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Write the cookie. If there is an error (due to an encoding failure or it // being too long) then log the error and send a 500 Internal Server Error // response. err := cookies.Write(w, cookie) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Use the Read() function to retrieve the cookie value, additionally // checking for the ErrInvalidValue error and handling it as necessary. value, err := cookies.Read(r, "exampleCookie") if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } w.Write([]byte(value)) } If you restart your web application and make a request to http://localhost:3000/set followed by http://localhost:3000/get in your browser, you should now successfully see the message "Hello Zoë!" in full. Likewise, if you make a request to http://localhost:3000/set using curl, you should see that the cookie value is SGVsbG8gWm_DqyE= — which is the base64 encoding of Hello Zoë!. $ curl -i localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie=SGVsbG8gWm_DqyE=; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Sun, 25 Sep 2022 09:14:18 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set $ echo "SGVsbG8gWm_DqyE=" | base64url --decode Hello Zoë! Tamper-proof (signed) cookies By default, you shouldn't trust cookie data. Because cookies are stored on the client, it's fairly straightforward for a user to edit them (in fact, many web browser extensions exist for exactly this purpose). So if you're performing actions in your web application based on the value of a cookie, it's important to first verify that the cookie hasn't been edited and contains the original name and value that you set. A good way to do this is to generate a HMAC signature of the cookie name and value, and then prepend this signature to the cookie value before sending it to the client. So that the final value is in this format: cookie.Value = "{HMAC signature}{original value}" When we receive the cookie back from the client, we can recalculate the HMAC signature from the cookie name and original value, and check that the recalculated HMAC signature matches the signature at the start of the received cookie. If they match, it confirms the integrity of the name and value — and we know that it hasn't been edited by the client. Let's update the internal/cookies/cookies.go file to include some WriteSigned() and ReadSigned() functions which do exactly that. File: internal/cookies/cookies.gopackage cookies import ( "crypto/hmac" "crypto/sha256" "encoding/base64" "errors" "net/http" ) ... func WriteSigned(w http.ResponseWriter, cookie http.Cookie, secretKey []byte) error { // Calculate a HMAC signature of the cookie name and value, using SHA256 and // a secret key (which we will create in a moment). mac := hmac.New(sha256.New, secretKey) mac.Write([]byte(cookie.Name)) mac.Write([]byte(cookie.Value)) signature := mac.Sum(nil) // Prepend the cookie value with the HMAC signature. cookie.Value = string(signature) + cookie.Value // Call our Write() helper to base64-encode the new cookie value and write // the cookie. return Write(w, cookie) } func ReadSigned(r *http.Request, name string, secretKey []byte) (string, error) { // Read in the signed value from the cookie. This should be in the format // "{signature}{original value}". signedValue, err := Read(r, name) if err != nil { return "", err } // A SHA256 HMAC signature has a fixed length of 32 bytes. To avoid a potential // 'index out of range' panic in the next step, we need to check sure that the // length of the signed cookie value is at least this long. We'll use the // sha256.Size constant here, rather than 32, just because it makes our code // a bit more understandable at a glance. if len(signedValue) Alright, let's update our main.go file to include a secret key and use the new WriteSigned() and ReadSigned() functions. The secret key should be generated using a cryptographically secure random number generator (CSRNG), should be unique to your application, and should ideally have at least 32 bytes of entropy. For the purpose of this example, we'll use a random 64 character hex string and decode it to give us a byte slice containing 32 random bytes. File: main.gopackage main import ( "encoding/hex" "errors" "log" "net/http" "example.com/example-project/internal/cookies" ) // Declare a global variable to hold the secret key. var secretKey []byte func main() { var err error // Decode the random 64-character hex string to give us a slice containing // 32 random bytes. For simplicity, I've hardcoded this hex string but in a // real application you should read it in at runtime from a command-line // flag or environment variable. secretKey, err = hex.DecodeString("13d6b4dff8f84a10851021ec8608f814570d562c92fe6b5ec4c9f595bcb3234b") if err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/set", setCookieHandler) mux.HandleFunc("/get", getCookieHandler) log.Print("Listening...") err = http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func setCookieHandler(w http.ResponseWriter, r *http.Request) { cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Use the WriteSigned() function, passing in the secret key as the final // argument. err := cookies.WriteSigned(w, cookie, secretKey) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Use the ReadSigned() function, passing in the secret key as the final // argument. value, err := cookies.ReadSigned(r, "exampleCookie", secretKey) if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } w.Write([]byte(value)) } If you visit http://localhost:3000/set in your web browser followed by http://localhost:3000/get, you should still successfully see the message "Hello Zoë!". If you like, you can also use a browser extension to change the cookie value (search for "cookie editor" in your browser extension store). If you do this and visit http://localhost:3000/get again, you should now receive a 400 Bad Request response and the "invalid cookie" message. Before we move on, let's also make a request to http://localhost:3000/set using curl: $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie=1lYrR9MfMsu6Dm39EgfbOuFTUbZm3_5tmWsF943HN4hIZWxsbyBab8OrIQ==; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Wed, 28 Sep 2022 09:28:55 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! In my case we can see that the signed cookie value is: 1lYrR9MfMsu6Dm39EgfbOuFTUbZm3_5tmWsF943HN4hIZWxsbyBab8OrIQ== Let's base64-decode this: $ echo "1lYrR9MfMsu6Dm39EgfbOuFTUbZm3_5tmWsF943HN4hIZWxsbyBab8OrIQ==" | base64url --decode �V+G�2˺m��:�SQ�f��m�k���7�Hello Zoë! The first part of the decoded value is the HMAC signature (which looks like gibberish), followed by our original cookie value in plaintext. Confidential (encrypted) and tamper-proof cookies The HMAC signing pattern above is great for times when you want to confirm that a cookie has not been edited by a client, and you're not worried about the client being able to read the cookie data (i.e. the cookie doesn't contain any secret or confidential information). But if you do want to prevent the client from being able to read the cookie data, we need to encrypt the data before writing it. A good way to encrypt the data in cookies is to use AES-GCM (AES with Galois/Counter Mode) encryption. AES-GCM is a type of authenticated encryption, which is good because it both encrypts and authenticates the data. The encryption ensures confidentiality of the data, and the authentication ensures the integrity of the data (i.e. that the data hasn't been changed). Effectively, encrypting our cookie data using AES-GCM is a relatively easy way to give us confidential, tamper-proof, cookies in a single step. Let's create two new helper functions, WriteEncrypted() and ReadEncrypted(), which use this. Like so: File: internal/cookies/cookies.gopackage cookies import ( "crypto/aes" "crypto/cipher" "crypto/hmac" "crypto/rand" "crypto/sha256" "encoding/base64" "errors" "fmt" "io" "net/http" "strings" ) ... func WriteEncrypted(w http.ResponseWriter, cookie http.Cookie, secretKey []byte) error { // Create a new AES cipher block from the secret key. block, err := aes.NewCipher(secretKey) if err != nil { return err } // Wrap the cipher block in Galois Counter Mode. aesGCM, err := cipher.NewGCM(block) if err != nil { return err } // Create a unique nonce containing 12 random bytes. nonce := make([]byte, aesGCM.NonceSize()) _, err = io.ReadFull(rand.Reader, nonce) if err != nil { return err } // Prepare the plaintext input for encryption. Because we want to // authenticate the cookie name as well as the value, we make this plaintext // in the format "{cookie name}:{cookie value}". We use the : character as a // separator because it is an invalid character for cookie names and // therefore shouldn't appear in them. plaintext := fmt.Sprintf("%s:%s", cookie.Name, cookie.Value) // Encrypt the data using aesGCM.Seal(). By passing the nonce as the first // parameter, the encrypted data will be appended to the nonce — meaning // that the returned encryptedValue variable will be in the format // "{nonce}{encrypted plaintext data}". encryptedValue := aesGCM.Seal(nonce, nonce, []byte(plaintext), nil) // Set the cookie value to the encryptedValue. cookie.Value = string(encryptedValue) // Write the cookie as normal. return Write(w, cookie) } func ReadEncrypted(r *http.Request, name string, secretKey []byte) (string, error) { // Read the encrypted value from the cookie as normal. encryptedValue, err := Read(r, name) if err != nil { return "", err } // Create a new AES cipher block from the secret key. block, err := aes.NewCipher(secretKey) if err != nil { return "", err } // Wrap the cipher block in Galois Counter Mode. aesGCM, err := cipher.NewGCM(block) if err != nil { return "", err } // Get the nonce size. nonceSize := aesGCM.NonceSize() // To avoid a potential 'index out of range' panic in the next step, we // check that the length of the encrypted value is at least the nonce // size. if len(encryptedValue) Then we can switch our main.go file to use these new helpers like so: File: main.gopackage main ... func setCookieHandler(w http.ResponseWriter, r *http.Request) { cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } err := cookies.WriteEncrypted(w, cookie, secretKey) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { value, err := cookies.ReadEncrypted(r, "exampleCookie", secretKey) if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } w.Write([]byte(value)) } Note: When using AES-GCM encryption, it's important that your secret key is exactly 32 bytes long. Otherwise you will get a runtime error like crypto/aes: invalid key size . Again, you can visit http://localhost:3000/set and http://localhost:3000/get in your browser, and you should still successfully see the message "Hello Zoë!". And if you edit the exampleCookie cookie using a browser extension, you should find that any subsequent requests result in an "invalid cookie" response. Let's take a look at the Set-Cookie header now using curl. $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie=hBGecbVJ2cI0yAwrbMYd5sv7qslxBJoGnk7LBLHVR9rKrqh1cTVs2IuWHZUOkl2fdYeIYmY=; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Wed, 28 Sep 2022 10:37:23 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! In my case the encrypted cookie value is: hBGecbVJ2cI0yAwrbMYd5sv7qslxBJoGnk7LBLHVR9rKrqh1cTVs2IuWHZUOkl2fdYeIYmY= If we base64-decode this value, we should now just see gibberish and our original "Hello Zoë!" value should no longer be visible. $ echo "hBGecbVJ2cI0yAwrbMYd5sv7qslxBJoGnk7LBLHVR9rKrqh1cTVs2IuWHZUOkl2fdYeIYmY=" | base64url --decode ��q�I��4�+l������q��N���G�ʮ�uq5l؋���]�u��bf Great! The encryption has worked! Storing custom data types So far we've just been storing simple string data in our cookies. But what if we want to store something more complicated, like the data for a user represented as a struct in Go? type User struct { Name string Age int } The good news is that the Go standard library includes the encoding/gob package, which we can use to encode/decode a Go value to and from a byte slice. It's kind of like "pickling" in Python, "marshaling" in Ruby, or "serializing" in PHP. To help demonstrate how to use this, let's update our main.go file to gob-encode a User struct and store it in a cookie: package main import ( "bytes" "encoding/gob" "encoding/hex" "errors" "fmt" "log" "net/http" "strings" "example.com/example-project/internal/cookies" ) var secret []byte // Declare the User type. type User struct { Name string Age int } func main() { // Importantly, we need to tell the encoding/gob package about the Go type // that we want to encode. We do this my passing *an instance* of the type // to gob.Register(). In this case we pass a pointer to an initialized (but // empty) instance of the User struct. gob.Register(&User{}) var err error secret, err = hex.DecodeString("13d6b4dff8f84a10851021ec8608f814570d562c92fe6b5ec4c9f595bcb3234b") if err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/set", setCookieHandler) mux.HandleFunc("/get", getCookieHandler) log.Print("Listening...") err = http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func setCookieHandler(w http.ResponseWriter, r *http.Request) { // Initialize a User struct containing the data that we want to store in the // cookie. user := User{Name: "Alice", Age: 21} // Initialize a buffer to hold the gob-encoded data. var buf bytes.Buffer // Gob-encode the user data, storing the encoded output in the buffer. err := gob.NewEncoder(&buf).Encode(&user) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } // Call buf.String() to get the gob-encoded value as a string and set it as // the cookie value. cookie := http.Cookie{ Name: "exampleCookie", Value: buf.String(), Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Write an encrypted cookie containing the gob-encoded data as normal. err = cookies.WriteEncrypted(w, cookie, secret) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Read the gob-encoded value from the encrypted cookie, handling any errors // as necessary. gobEncodedValue, err := cookies.ReadEncrypted(r, "exampleCookie", secret) if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } // Create a new instance of a User type. var user User // Create an strings.Reader containing the gob-encoded value. reader := strings.NewReader(gobEncodedValue) // Decode it into the User type. Notice that we need to pass a *pointer* to // the Decode() target here? if err := gob.NewDecoder(reader).Decode(&user); err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } // Print the user information in the response. fmt.Fprintf(w, "Name: %q\n", user.Name) fmt.Fprintf(w, "Age: %d\n", user.Age) } If you want, restart your application and visit localhost:3000/set followed by localhost:3000/get in your web browser. You should see a response similar to this: Note: In the example above, we've gob-encoded the cookie data and then written the cookie using WriteEncrypted(), but you could equally write the cookie using the Write() or WriteSigned() helper functions that we made earlier too.

• Flow: A tiny but powerful HTTP router for Go
  May 25, 2022

  Last year I wrote a new HTTP router for Go called Flow. I've been using it in production on this site and in a couple of other projects since, and I'm pretty happy with how it's working out so decided to share it a bit more widely. My aim with Flow was to bring together my favourite features from other popular routers that I frequently used. It has: A very small and readable codebase (approx. 160 LOC) with pattern-matching logic similar to matryer/way. Middleware management like chi — including the ability to create route 'groups' which use different middleware. Optional regexp support for tighter pattern matching, similar to chi and gorilla/mux. Automatic handling of OPTIONS requests, like julienschmidt/httprouter. Automatic handling of HEAD requests, like bmizerany/pat. An Allow header is automatically set on all OPTIONS and 405 Method Not Allowed responses, like julienschmidt/httprouter. Ability to map multiple HTTP methods to the same handler in one declaration, like gorilla/mux. Additionally: It has a very small API (see the Go docs) so there's not much to learn. It's designed to work nicely with http.Handler, http.HandlerFunc, and the standard Go middleware pattern. The handlers for 404 Not Found and 405 Method Not Allowed responses are customizable. Conflicting routes are permitted (e.g. /posts/:id and posts/new), with routes matched in the order that they are declared. It has zero dependencies. Below is a quick example of the syntax, and if you like the look of it you can check out the full README on GitHub. mux := flow.New() // The Use() method can be used to register middleware. Middleware declared at // the top level will used on all routes (including error handlers and OPTIONS // responses). mux.Use(exampleMiddleware1) // Routes can use multiple HTTP methods. mux.HandleFunc("/profile/:name", exampleHandlerFunc1, "GET", "POST") // Optionally, regular expressions can be used to enforce a specific pattern // for a named parameter. mux.HandleFunc("/profile/:name/:age|^[0-9]{1,3}$", exampleHandlerFunc2, "GET") // The wildcard ... can be used to match the remainder of a request path. // Notice that HTTP methods are also optional (if not provided, all HTTP // methods will match the route). mux.Handle("/static/...", exampleHandler) // You can create route 'groups'. mux.Group(func(mux *flow.Mux) { // Middleware declared within in the group will only be used on the routes // in the group. mux.Use(exampleMiddleware2) mux.HandleFunc("/admin", exampleHandlerFunc3, "GET") // Groups can be nested. mux.Group(func(mux *flow.Mux) { mux.Use(exampleMiddleware3) mux.HandleFunc("/admin/passwords", exampleHandlerFunc4, "GET") }) }) A note on performance I haven't done any benchmarking against other routers, so I can't speak about the relative performance of Flow. What I can say it has been plenty fast enough for all of my use-cases so far and not a hot spot when profiling my applications under load.

• How to use go run to manage tool dependencies
  May 10, 2022

  When you're working on a project it's common for there to be some developer tooling dependencies. These aren't code dependencies, but rather tools that you run as part of the development, testing, build or deployment processes. For example, you might use golang.org/x/text/cmd/gotext in conjunction with go:generate to generate message catalogs for translation, or honnef.co/go/tools/cmd/staticcheck to perform static analysis on your code before committing a change. This raises a couple of interesting questions — especially in a team environment. How do you make sure that everyone has the necessary tools installed on their machines? And that the tools they are using are all the same version? Until Go 1.17, the convention for managing this was to create a tools.go file in your project containing import statements for the different tools and a //go:build tools build constraint. If you're not already familiar with this approach, it's described in the official Go Wiki. But since Go 1.17 there is an alternative approach you can take. It has pros and cons compared to the tools.go approach, but it's worth knowing about and may be a good fit for some projects. It hinges on the fact that go run now allows you to execute a specific version of a remote package. From the 1.17 release notes: go run now accepts arguments with version suffixes (for example, go run example.com/cmd@v1.0.0). This causes go run to build and run packages in module-aware mode, ignoring the go.mod file in the current directory or any parent directory, if there is one. In other words, you can use go run package@version to execute a remote package when you are outside of a module, or inside of a module even if the package isn't listed in the go.mod file. It's also useful as a quick way to run an executable package without installing it. Instead of this: $ go install honnef.co/go/tools/cmd/staticcheck@v0.3.1 $ staticcheck ./... You can now just do this: $ go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... Important: When you execute go run package@version the necessary modules will be downloaded and cached on your machine in the module cache. So when you execute the same go run command later, the cache will be used (rather than everything being downloaded again) and it should complete faster. Using with go:generate Let's take a look at an example where we use the golang.org/x/tools/cmd/stringer tool in conjunction with go:generate to generate String() methods for some iota constants. If you'd like to follow along, please run the following commands: $ mkdir tools $ go mod init example.com/tools $ touch main.go And then add the following code to main.go: File: main.go package main import "fmt" //go:generate go run golang.org/x/tools/cmd/stringer@v0.1.10 -type=Level type Level int const ( Info Level = iota Error Fatal ) func main() { fmt.Printf("%s: Hello world!\n", Info) } The important thing here is the //go:generate line. When you run go generate on this file, it will in turn use go run to execute v0.1.10 of the golang.org/x/tools/cmd/stringer package. Let's try it out: $ go generate . go: downloading golang.org/x/tools v0.1.10 go: downloading golang.org/x/sys v0.0.0-20211019181941-9d821ace8654 go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 You should see that the necessary modules are downloaded and then the go:generate command finishes executing successfully — resulting in a new level_string.go file being generated and a working application. Like so: $ ls go.mod level_string.go main.go $ go run . Info: Hello world! Using in a Makefile You can also use the go run package@version pattern to execute tools from your scripts or Makefiles. To illustrate, let's create a Makefile with an audit task that executes a specific version of the staticcheck tool. $ touch Makefile File: Makefile .PHONY: audit audit: go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... If you run make audit, the necessary modules will be downloaded and the staticcheck tool should complete its checks successfully. $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... go: downloading honnef.co/go/tools v0.3.1 go: downloading golang.org/x/tools v0.1.11-0.20220316014157-77aa08bb151a go: downloading golang.org/x/exp/typeparams v0.0.0-20220218215828-6cf2b201936e go: downloading github.com/BurntSushi/toml v0.4.1 If you run it for a second time, you'll see that the module cache is used and it should finish much faster. $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... Pros and cons In terms of positives, go run package@version has a couple of nice advantages over the tools.go approach: It's simpler to set up and requires less code — no tools.go file is needed, there are no build constraints, and no aliased imports. It avoids polluting your dependency graph with things that your binaries do not actually depend on. In terms of negatives: If you have the same go run package@version command in multiple places throughout your codebase and want to upgrade to a newer version, then you need to update all of the commands manually (or use sed or find-and-replace). With the tools.go approach you only need to update your go.mod file by running go get package@newversion. With the tools.go approach it's possible to verify that cached code in your module cache hasn't been changed by running go mod verify. I'm not aware of an equivalent check for go run package@version (if you know of a way to do this, please let me know!). From my limited testing, it seems to be possible to edit the cached code in the module cache on your machine, and go run package@version will use this edited code without complaining. If you are working offline, then go run package@version may fail with a dial tcp: lookup proxy.golang.org: Temporary failure in name resolution error because it can't reach the Go module mirror — even if there is a copy already in your local module cache. Similar to this: $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... go: honnef.co/go/tools/cmd/staticcheck@v0.3.1: honnef.co/go/tools/cmd/staticcheck@v0.3.1: Get "https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.3.1.info": dial tcp: lookup proxy.golang.org: Temporary failure in name resolution make: *** [Makefile:4: audit] Error 1 As far as I can see this isn't a problem when you use the tools.go approach, although you can work around it fairly easily by setting the GOPROXY environment variable to direct while you are offline. Doing this will force go run to bypass the Go module mirror and use the cached module on your machine straight away. $ export GOPROXY=direct $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./...

• Change URL query params in Go
  Nov 28, 2021

  In this short post we're going to discuss how to add, modify or delete URL query string parameters in Go. To illustrate, we'll look at how to change this URL: https://example.com?name=alice&age=28&gender=female To this: https://example.com?name=alice&age=29&occupation=carpenter If you want to change the URL query string in place: // Use url.Parse() to parse a string into a *url.URL type. If your URL is // already a url.URL type you can skip this step. urlA, err := url.Parse("https://example.com?name=alice&age=28&gender=female") if err != nil { log.Fatal(err) } // Use the Query() method to get the query string params as a url.Values map. values := urlA.Query() // Make the changes that you want using the Add(), Set() and Del() methods. If // you want to retrieve or check for a specific parameter you can use the Get() // and Has() methods respectively. values.Add("occupation", "carpenter") values.Del("gender") values.Set("age", strconv.Itoa(29)) // Use the Encode() method to transform the url.Values map into a URL-encoded // string (like "age=29&name=alice...") and assign it back to the URL. Note // that the encoded values will be sorted alphabetically based on the parameter // name. urlA.RawQuery = values.Encode() fmt.Printf("urlA: %s", urlA.String()) Running this will print out: urlA: https://example.com?age=29&name=alice&occupation=carpenter If you want to create a clone of the URL but with a different query string, while leaving the original URL unchanged, you need to create a copy of the original url.URL struct first. There are a couple of ways to do this. You can either re-parse the URL, or you can dereference the original url.URL and make a copy, like so: // This is equivalent to: var newUrl url.URL = *originalUrl newUrl := *originalUrl When you do this, you create a new newURL variable of type url.URL which is initialized to the (dereferenced) value of *originalURL. This means that newURL has a different address in memory to originalURL. Putting this together, the pattern for creating a new URL with different parameters is: urlA, err := url.Parse("https://example.com?name=alice&age=28&gender=female") if err != nil { log.Fatal(err) } // Make a copy of the original url.URL. urlB := *urlA // Make the param changes to the new url.URL type... values := urlB.Query() values.Add("occupation", "carpenter") values.Del("gender") values.Set("age", strconv.Itoa(29)) urlB.RawQuery = values.Encode() fmt.Printf("urlA: %s\n", urlA.String()) // This will be unchanged. fmt.Printf("urlB: %s\n", urlB.String()) // This will have the new params. Running this will print out: urlA: https://example.com?name=alice&age=28&gender=female urlB: https://example.com?age=29&name=alice&occupation=carpenter As a side note, you can use this technique any time you want to 'clone' a URL and make changes to it. For example to create a clone of a URL with a different path, you can do this: urlA, err := url.Parse("https://example.com/foo") if err != nil { log.Fatal(err) } urlB := *urlA urlB.Path = "/bar" fmt.Printf("%s\n", urlA.String()) // Prints https://example.com/foo fmt.Printf("%s\n", urlB.String()) // Prints https://example.com/bar

• I18n in Go: Managing translations
  Aug 25, 2021

  Recently I've been building a fully internationalized (i18n) and localized (l10n) web application for the first time with Go's golang.org/x/text packages. I've found that the packages and tools that live under golang.org/x/text are really effective and well designed, although it's been a bit of a challenge to figure out how to put it all together in a real application. Note: Just in case you're not already aware, the packages that live under golang.org/x are part of the official Go Project but outside the main Go standard library tree. They are held to looser standards that the standard library packages, which means they aren't subject to the Go compatibility promise (i.e. their APIs might change), and documentation may not always be complete. In this tutorial I want to explain how you can use golang.org/x/text packages to manage translations in your application. Specifically: How to use the golang.org/x/text/language and golang.org/x/text/message packages to print translated messages from your Go code. How to use the gotext tool to automatically extract messages for translation from your code into JSON files. How to use gotext to parse translated JSON files and create a catalog containing translated messages. How to manage variables in messages and provided pluralized versions of translations. What we'll be building To help put this into context, we're going to create a simple pre-launch website for an imaginary online bookstore. We'll start off slowly and build up the code step-by-step. Our application will have just a single home page, and we'll localize the page content based on a locale identifier at the start of the URL path. We'll set up our application to support three different locales: the United Kingdom, Germany, and the French-speaking part of Switzerland. URL Localized for localhost:4018/en-gb United Kingdom localhost:4018/de-de Germany localhost:4018/fr-ch Switzerland (French-speaking) We're going to follow a common convention and use BCP 47 language tags as the locale identifier in our URLs. Simplifying things hugely for the sake of this tutorial, BCP 47 language tags typically take the format {language}-{region}. The language part is a ISO 639-1 code and the region is a two-letter country code from ISO_3166-1. It's conventional to uppercase the region (like en-GB), but BCP 47 tags are technically case-insensitive and it's OK for us to use all-lowercase versions in our URLs. Scaffolding a web application If you'd like to follow along with the application build, go ahead and run the following commands to setup a new project directory. $ mkdir bookstore $ cd bookstore $ go mod init bookstore.example.com go: creating new go.mod: module bookstore.example.com At this point, you should have a go.mod file in the root of the project directory with the module path bookstore.example.com. Next create a new cmd/www directory to hold the code for the bookstore web application, and add main.go and handlers.go files like so: $ mkdir -p cmd/www $ touch cmd/www/main.go cmd/www/handlers.go Your project directory should now look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go └── go.mod Let's begin in the cmd/www/main.go file and add the code to declare our application routes and start a HTTP server. Because our application URL paths will always use a (dynamic) locale as a prefix — like /en-gb/bestsellers or /fr-ch/bestsellers — it's simplest if our application uses a third-party router which supports dynamic values in URL path segments. I'm going to use pat, but feel free to use an alternative like chi or gorilla/mux if you prefer. Note: If you're not sure which router to use in your project, you might like to take a look at my comparison of Go routers blog post. OK, open up the main.go file and add the following code: File: cmd/www/main.go package main import ( "log" "net/http" "github.com/bmizerany/pat" ) func main() { // Initialize a router and add the path and handler for the homepage. mux := pat.New() mux.Get("/:locale", http.HandlerFunc(handleHome)) // Start the HTTP server using the router. log.Print("starting server on :4018...") err := http.ListenAndServe(":4018", mux) log.Fatal(err) } Then in the cmd/www/handlers.go file, add a handleHome() function which extracts the locale identifer from the URL path and echoes it in the HTTP response. File: cmd/www/handlers.go package main import ( "fmt" "net/http" ) func handleHome(w http.ResponseWriter, r *http.Request) { // Extract the locale from the URL path. This line of code is likely to // be different for you if you are using an alternative router. locale := r.URL.Query().Get(":locale") // If the locale matches one of our supported values, echo the locale // in the response. Otherwise send a 404 Not Found response. switch locale { case "en-gb", "de-de", "fr-ch": fmt.Fprintf(w, "The locale is %s\n", locale) default: http.NotFound(w, r) } } Once that's done, run go mod tidy to tidy your go.mod file and download any necessary dependencies, and then run the web application. $ go mod tidy go: finding module for package github.com/bmizerany/pat go: found github.com/bmizerany/pat in github.com/bmizerany/pat v0.0.0-20210406213842-e4b6760bdd6f $ go run ./cmd/www/ 2021/08/21 21:22:57 starting server on :4018... If you make some requests to the application using curl, you should find that the appropriate locale is echoed back to you like so: $ curl localhost:4018/en-gb The locale is en-gb $ curl localhost:4018/de-de The locale is de-de $ curl localhost:4018/fr-ch The locale is fr-ch $ curl localhost:4018/da-DK 404 page not found Extracting and translating text content Now that we've laid the groundwork for our web application, let's get into the core of this tutorial and update the handleHome() function so that it renders a "Welcome!" message translated for the specific locale. In this project we'll use British English (en-GB) as the default 'source' or 'base' language in our application, but we'll want to render a translated version of the welcome message in German and French for the other locales. To do this, we'll need to import the golang.org/x/text/language and golang.org/x/text/message packages and update our handleHome() function to do the following two things: Construct a language.Tag which identifies the target language that we want to translate the message in to. The language package contains some pre-defined tags for common language variants, but I find that it's easier to use the language.MustParse() function to create a tag. This let's you create a language.Tag for any valid BCP 47 value, like language.MustParse("fr-CH"). Once you have a language tag, you can use the message.NewPrinter() function to create a message.Printer instance that prints out messages in that specific language. If you're following along, please go ahead and update your cmd/www/handlers.go file to contain the following code: File: cmd/www/handlers.go package main import ( "net/http" "golang.org/x/text/language" "golang.org/x/text/message" ) func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") // Declare variable to hold the target language tag. var lang language.Tag // Use language.MustParse() to assign the appropriate language tag // for the locale. switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } // Initialize a message.Printer which uses the target language. p := message.NewPrinter(lang) // Print the welcome message translated into the target language. p.Fprintf(w, "Welcome!\n") } Again, run go mod tidy to download the necessary dependencies… $ go mod tidy go: finding module for package golang.org/x/text/message go: finding module for package golang.org/x/text/language go: downloading golang.org/x/text v0.3.7 go: found golang.org/x/text/language in golang.org/x/text v0.3.7 go: found golang.org/x/text/message in golang.org/x/text v0.3.7 And then run the application: $ go run ./cmd/www/ 2021/08/21 21:33:52 starting server on :4018... When you make a request to any of the supported URLs, you should now see the (untranslated) welcome message like this: $ curl localhost:4018/en-gb Welcome! $ curl localhost:4018/de-de Welcome! $ curl localhost:4018/fr-ch Welcome! So in all cases we're seeing the "Welcome!" message in our en-GB source language. That's because we still need to provide Go's message package with the actual translations that we want to use. Without the actual translations, it falls back to displaying the message in the source language. There are a number of ways to provide Go's message package with translations, but for most non-trivial applications it's probably sensible to use some automated tooling to help you manage the task. Fortunately, Go provides the gotext tool to assist with this. Note: The gotext tool we're using is the one from golang.org/x/text/cmd/gotext. It shouldn't be confused with the github.com/leonelquinteros/gotext package (which is designed to work with GNU gettext utilities and PO/MO files). If you're following along, please use go install to install the gotext executable on your machine: $ go install golang.org/x/text/cmd/gotext@latest All being well, the tool should be installed to your $GOBIN directory on your system path and you can run it like so: $ which gotext /home/alex/go/bin/gotext $ gotext gotext is a tool for managing text in Go source code. Usage: gotext command [arguments] The commands are: update merge translations and generate catalog extract extracts strings to be translated from code rewrite rewrites fmt functions to use a message Printer generate generates code to insert translated messages Use "gotext help [command]" for more information about a command. Additional help topics: Use "gotext help [topic]" for more information about that topic. I really like the gotext tool — it's functionality is excellent — but there are a couple of important things to point out before we carry on. The first thing is that go text is designed to work in conjunction with go generate, not as a standalone command-line tool. You can run it as a standalone tool, but weird things happen and it's a lot smoother if you use it in the way it's intended. The other thing is that documentation and help functionality is basically non-existent. The best guidance on how to use it are the examples in the repository and, probably, this article that you're reading right now. There is an open issue about the lack of help functionality, and hopefully this is something that will improve in the future. In this tutorial, we're going to store the all the code relating to translations in a new internal/translations package. We could keep all the translation code for our web application under cmd/www instead, but in my (limited) experience I've found that using a separate internal/translations package is better. It helps separate concerns and also makes it possible to reuse the same translations across different applications in the same project. YMMV. If you're following along, go ahead and create that new directory and a translations.go file like so: $ mkdir -p internal/translations $ touch internal/translations/translations.go At this point, your project structure should look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go ├── go.mod ├── go.sum └── internal └── translations └── translations.go Next, let's open up the internal/translations/translations.go file and add a go generate command which uses gotext to extract the messages for translation from our application. File: internal/translations/translations.go package translations //go:generate gotext -srclang=en-GB update -out=catalog.go -lang=en-GB,de-DE,fr-CH bookstore.example.com/cmd/www There's a lot going on in this command, so let's quickly break it down. The -srclang flag contains the BCP 47 tag for the source (or 'base') language that we are using in the application. In our case, the source language is en-GB. update is thegotext function that we want to execute. As well as update there are extract, rewrite and generate functions, but in the translation workflow for a web application the only one you actually need is update. The -out flag contains the path that you want the message catalog to be output to. This path should be relative to the file containing the go generate command. In our case, we've set the value to catalog.go, which means that the message catalog will be output to a new internal/translations/catalog.go file. We'll talk more about message catalogs and explain what they are shortly. The -lang flag contains a comma-separated list of the BCP 47 tags that you want to create translations for. You don't need to include the source language here, but (as we'll demonstrate later in this article) it can be helpful for dealing with pluralization of text content. Lastly, we have the fully-qualified module path for the package(s) that you want to create translations for (in this case bookstore.example.com/cmd/www). You can list multiple packages if necessary, separated by a whitespace character. When we execute this go generate command, gotext will walk the code for the cmd/www application and look for all calls to a message.Printer†. It then extracts the relevant message strings and outputs them to some JSON files for translation. † Important: It's critical to note when gotext walks your code it actually only looks for calls to message.Printer.Printf(), Fprintf() and Sprintf() — basically the three methods that end with an f. It ignores all other methods such as Sprint() or Println(). You can see this behavior in the gotext implementation here. OK, let's put this into action and call go generate on our translations.go file. In turn, this will execute the gotext command that we included at the top of that file. $ go generate ./internal/translations/translations.go de-DE: Missing entry for "Welcome!". fr-CH: Missing entry for "Welcome!". Cool, this looks like we're getting somewhere. We've got some useful feedback to indicate that we are missing the necessary German and French translations for our "Welcome!" message. If you take a look at the directory structure for your project, it should now look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go ├── go.mod ├── go.sum └── internal └── translations ├── catalog.go ├── locales │ ├── de-DE │ │ └── out.gotext.json │ ├── en-GB │ │ └── out.gotext.json │ └── fr-CH │ └── out.gotext.json └── translations.go We can see that the go generate command has automatically generated an internal/translations/catalog.go file for us (which we'll look at in a minute), and a locales folder containing out.gotext.json files for each of our target languages. Let's take a look at the internal/translations/locales/de-DE/out.gotext.json file: File: internal/translations/locales/de-DE/out.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "" } ] } In this JSON file, the relevant BCP 47 language tag is defined at the top of the file, followed by a JSON array of the messages which require translation. The message value is the text for translation in the source language, and the (currently empty) translation value is where we should enter appropriate German translation. It's important to emphasize that you don't edit this file in place. Instead, the workflow for adding a translation goes like this: You generate the out.gotext.json files containing the messages which need to be translated (which we've just done). You send these files to a translator, who edits the JSON to include the necessary translations. They then send the updated files back to you. You then save these updated files with the name messages.gotext.json in the folder for the appropriate language. For demonstration purposes, let's quickly simulate this workflow by copying the out.gotext.json files to messages.gotext.json files, and updating them to include the translated messages like so: $ cp internal/translations/locales/de-DE/out.gotext.json internal/translations/locales/de-DE/messages.gotext.json $ cp internal/translations/locales/fr-CH/out.gotext.json internal/translations/locales/fr-CH/messages.gotext.json File: internal/translations/locales/de-DE/messages.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" } ] } File: internal/translations/locales/fr-CH/messages.gotext.json { "language": "fr-CH", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Bienvenu !" } ] } If you like, you can also take a look at the out.gotext.json file for our en-GB source language. You'll see that the translation value for the message has been auto-filled for us. File: internal/translations/locales/en-GB/messages.gotext.json { "language": "en-GB", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Welcome!", "translatorComment": "Copied from source.", "fuzzy": true } ] } The next step is to run our go generate command again. This time, it should execute without any warning messages about missing translations. $ go generate ./internal/translations/translations.go Now it's a good time to take a look at the internal/translations/catalog.go file, which is automatically generated for us by the gotext update command. This file contains a message catalog, which is — very roughly speaking — a mapping of messages and their relevant translations for each target language. Let's take a quick look inside the file: File: internal/translations/catalog.go // Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT. package translations import ( "golang.org/x/text/language" "golang.org/x/text/message" "golang.org/x/text/message/catalog" ) type dictionary struct { index []uint32 data string } func (d *dictionary) Lookup(key string) (data string, ok bool) { p, ok := messageKeyToIndex[key] if !ok { return "", false } start, end := d.index[p], d.index[p+1] if start == end { return "", false } return d.data[start:end], true } func init() { dict := map[string]catalog.Dictionary{ "de_DE": &dictionary{index: de_DEIndex, data: de_DEData}, "en_GB": &dictionary{index: en_GBIndex, data: en_GBData}, "fr_CH": &dictionary{index: fr_CHIndex, data: fr_CHData}, } fallback := language.MustParse("en-GB") cat, err := catalog.NewFromMap(dict, catalog.Fallback(fallback)) if err != nil { panic(err) } message.DefaultCatalog = cat } var messageKeyToIndex = map[string]int{ "Welcome!\n": 0, } var de_DEIndex = []uint32{ // 2 elements 0x00000000, 0x00000011, } // Size: 32 bytes const de_DEData string = "\x04\x00\x01\n\f\x02Willkommen!" var en_GBIndex = []uint32{ // 2 elements 0x00000000, 0x0000000e, } // Size: 32 bytes const en_GBData string = "\x04\x00\x01\n\t\x02Welcome!" var fr_CHIndex = []uint32{ // 2 elements 0x00000000, 0x00000010, } // Size: 32 bytes const fr_CHData string = "\x04\x00\x01\n\v\x02Bienvenu !" // Total table size 143 bytes (0KiB); checksum: 385F6E56 I don't want to dwell on the details here, because it's OK for use to treat this file as something of a 'black box', and — as warned by the comment at the top of the file — we shouldn't make any changes to it directly. But the most important thing to point out is that this file contains an init() function which, when called, initializes a new message catalog containing all our translations and mappings. It then sets this as the default message catalog by assigning it to the message.DefaultCatalog global variable. When we call one of the message.Printer functions, the printer will lookup the relevant translation from the default message catalog for printing. This is really nice, because it means that all our translations are stored in memory at runtime, and any lookups are very fast and efficient. So, if we take a step back for a moment, we can see that the gotext update command that we're using with go generate actually does two things. One — it walks the code in our cmd/www application and extracts the necessary strings for translation into the out.gotext.json files; and two — it also parses any messages.gotext.json files (if present) and updates the message catalog accordingly. The final step in getting this working is to import the internal/translations package in our cmd/www/handlers.go file. This will ensure that the init() function in internal/translations/translations.go is called, and the default message catalog is updated to be the one containing our translations. Because we won't actually be referencing anything in the internal/translations package directly, we'll need to alias the import path to the blank identifer _ to prevent the Go compiler from complaining. Go ahead and do that now: File: cmd/www/handlers.go package main import ( "net/http" // Import the internal/translations package, so that its init() // function is called. _ "bookstore.example.com/internal/translations" "golang.org/x/text/language" "golang.org/x/text/message" ) func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") var lang language.Tag switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") } Alright, let's try this out! When your restart the application and try making some requests, you should now see the "Welcome!" message translated into the appropriate language. $ curl localhost:4018/en-GB Welcome! $ curl localhost:4018/de-de Willkommen! $ curl localhost:4018/fr-ch Bienvenu ! Using variables in translations Now that we've got the basic translations working in our application, let's move on to something a bit more advanced and look at how to manage translations with interpolated variables in them. To demonstrate, we'll update the HTTP response from our handleHome() function to include a "{N} books available" line, where {N} is an integer containing the number of books in our imaginary bookstore. File: cmd/www/handlers.go package main ... func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") var lang language.Tag switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } // Define a variable to hold the number of books. In a real application // this would probably be retrieved by making a database query or // something similar. var totalBookCount = 1_252_794 p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") // Use the Fprintf() function to include the new message in the HTTP // response, with the book count as in interpolated integer value. p.Fprintf(w, "%d books available\n", totalBookCount) } Save the changes, then use go generate to output some new out.gotext.json files. You should see warning messages for the new missing translations like so: $ go generate ./internal/translations/translations.go de-DE: Missing entry for "{TotalBookCount} books available". fr-CH: Missing entry for "{TotalBookCount} books available". Let's take a look at the de-DE/out.gotext.json file: File: internal/translations/locales/de-DE/out.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": "", "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } The first thing to point out here is that the translation for our "Welcome!" message has been persisted across the workflow and is already present in the out.gotext.json file. This is obviously really important, because it means that when we send the file to the translator they won't need to provide the translation again. The second thing is that there is now an entry for our new message. We can see that this has the form "{TotalBookCount} books available", with the (capitalized) variable name from our Go code being used as the placeholder parameter. You should keep this in mind when writing your code, and try to use sensible and descriptive variable names that will make sense to your translators. The placeholders array also provides additional information about each placeholder value, the most useful part probably being the type value (which in this case tells the translator that the TotalBookCount value is an integer). So the next step is to send these new out.gotext.json files off to a translator for translation. Again, we'll simulate that here by copying them to messages.gotext.json files and adding the translations like so: $ cp internal/translations/locales/de-DE/out.gotext.json internal/translations/locales/de-DE/messages.gotext.json $ cp internal/translations/locales/fr-CH/out.gotext.json internal/translations/locales/fr-CH/messages.gotext.json File: internal/translations/locales/de-DE/messages.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": "{TotalBookCount} Bücher erhältlich", "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } File: internal/translations/locales/fr-CH/messages.gotext.json { "language": "fr-CH", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Bienvenu !" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": "{TotalBookCount} livres disponibles", "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } Make sure that both messages.gotext.json files are saved, and then run go generate to update our message catalog. This should run without any warnings. $ go generate ./internal/translations/translations.go When you restart the cmd/www application and make some HTTP requests again, you should now see the new translated messages like so: $ curl localhost:4018/en-GB Welcome! 1,252,794 books available $ curl localhost:4018/de-de Willkommen! 1.252.794 Bücher erhältlich $ curl localhost:4018/fr-ch Bienvenu ! 1 252 794 livres disponibles Now this is really cool. As we'll as the translations being applied by our message.Printer, it's also smart enough to output the interpolated integer value with the correct number formatting for each language. We can see here that our en-GB locale uses the "," character as a thousands separator, whereas de-DE uses "." and fr-CH uses the whitespace " ". A similar thing is done for decimal separators too. Dealing with pluralization's This is working nicely, but what happens if there is only 1 book available in our bookstore? Let's update the handleHome() function so that the totalBookCount value is 1: File: cmd/www/handlers.go package main ... func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") var lang language.Tag switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } // Set the total book count to 1. var totalBookCount = 1 p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") p.Fprintf(w, "%d books available\n", totalBookCount) } (I know this is a bit of a tenuous example, but it helps illustrate Go's pluralization functionality without much extra code, so bear with me!) You can probably imagine what happens when we restart the application and make a request to localhost:4018/en-gb now. $ curl localhost:4018/en-gb Welcome! 1 books available That's right, we see the message "1 books available", which isn't correct English because of the plural noun books. It would be better if this message read 1 book available or — even better — One book available instead. Happily, it's possible for us to specify alternative translations based on the value of an interpolated variable in our messages.gotext.json files. Let's start by demonstrating this for our en-GB locale. If you're following along, copy the en-GB/out.gotext.json file to en-GB/messages.gotext.json: $ cp internal/translations/locales/en-GB/out.gotext.json internal/translations/locales/en-GB/messages.gotext.json And then update it like so: File: internal/translations/locales/en-GB/messages.gotext.json { "language": "en-GB", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Welcome!", "translatorComment": "Copied from source.", "fuzzy": true }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": { "select": { "feature": "plural", "arg": "TotalBookCount", "cases": { "=1": { "msg": "One book available" }, "other": { "msg": "{TotalBookCount} books available" } } } }, "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } Now, rather than the translation value being a simple string we have set it to a JSON object that instructs the message catalog to use different translations depending on the value of the TotalBookCount placeholder. The key part here is the cases value, which contains the translations to use for different values of the placeholder. The supported case rules are: Case Description "=x" Where x is an integer that equals the value of the placeholder " Where x is an integer that is larger than the value of the placeholder "other" All other cases (a bit like default in a Go switch statement) Note: If you look at the documentation for the golang.org/x/text/feature/plural package (which is what gotext uses behind the scenes when generating the message catalog), you'll see that it also mentions the case rules "zero", "one", "two", "few", and "many". However, these rules aren't supported for all possible target languages, and you may get an error like gotext: generation failed: error: plural: form "many" not supported for language "de-DE" if you try to use them. It seems to be safer to stick with the three case rules in the table above. Additionally, it's important to be aware that the range of allowed values for x in the "=x" and " case rules is 0 to 32767. Trying to use something outside of that range will result in an error. There's an open issue about these behaviors here. Let's complete work this by updating the messages.gotext.json files for our de-DE and fr-CH languages to include the appropriate pluralized variations, like so: File: internal/translations/locales/de-DE/messages.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": { "select": { "feature": "plural", "arg": "TotalBookCount", "cases": { "=1": { "msg": "Ein Buch erhältlich" }, "other": { "msg": "{TotalBookCount} Bücher erhältlich" } } } }, "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } File: internal/translations/locales/fr-CH/messages.gotext.json { "language": "fr-CH", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Bienvenu !" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": { "select": { "feature": "plural", "arg": "TotalBookCount", "cases": { "=1": { "msg": "Un livre disponible" }, "other": { "msg": "{TotalBookCount} livres disponibles" } } } }, "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } Once those files are saved, use go generate again to update the message catalog: $ go generate ./internal/translations/translations.go And if you restart the web application and make some HTTP requests, you should now see the appropriate message for 1 book: $ curl localhost:4018/en-GB Welcome! One book available $ curl localhost:4018/de-de Willkommen! Ein Buch erhältlich $ curl localhost:4018/fr-ch Bienvenu ! Un livre disponible If you like, you can revert the totalBookCount variable back to a larger number... File: cmd/www/handlers.go package main ... func handleHome(w http.ResponseWriter, r *http.Request) { ... // Revert the total book count. var totalBookCount = 1_252_794 p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") p.Fprintf(w, "%d books available\n", totalBookCount) } And when you restart the application and make another request, you should see the "other" version of our message: $ curl localhost:4018/de-de Willkommen! 1.252.794 Bücher erhältlich Creating a localizer abstraction In the final part of this article we're going to create a new internal/localizer package which abstracts all our code for dealing with languages, printers and translations. If you're following along, go ahead and create a new internal/localizer directory containing a localizer.go file. $ mkdir -p internal/localizer $ touch internal/localizer/localizer.go At this point, your project structure should look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go ├── go.mod ├── go.sum └── internal ├── localizer │ └── localizer.go └── translations ├── catalog.go ├── locales │ ├── de-DE │ │ ├── messages.gotext.json │ │ └── out.gotext.json │ ├── en-GB │ │ ├── messages.gotext.json │ │ └── out.gotext.json │ └── fr-CH │ ├── messages.gotext.json │ └── out.gotext.json └── translations.go And then add the following code to the new localizer.go file: File: internal/localizer/localizer.go package localizer import ( // Import the internal/translations so that it's init() function // is run. It's really important that we do this here so that the // default message catalog is updated to use our translations // *before* we initialize the message.Printer instances below. _ "bookstore.example.com/internal/translations" "golang.org/x/text/language" "golang.org/x/text/message" ) // Define a Localizer type which stores the relevant locale ID (as used // in our URLs) and a (deliberately unexported) message.Printer instance // for the locale. type Localizer struct { ID string printer *message.Printer } // Initialize a slice which holds the initialized Localizer types for // each of our supported locales. var locales = []Localizer{ { // Germany ID: "de-de", printer: message.NewPrinter(language.MustParse("de-DE")), }, { // Switzerland (French speaking) ID: "fr-ch", printer: message.NewPrinter(language.MustParse("fr-CH")), }, { // United Kingdom ID: "en-gb", printer: message.NewPrinter(language.MustParse("en-GB")), }, } // The Get() function accepts a locale ID and returns the corresponding // Localizer for that locale. If the locale ID is not supported then // this returns `false` as the second return value. func Get(id string) (Localizer, bool) { for _, locale := range locales { if id == locale.ID { return locale, true } } return Localizer{}, false } // We also add a Translate() method to the Localizer type. This acts // as a wrapper around the unexported message.Printer's Sprintf() // function and returns the appropriate translation for the given // message and arguments. func (l Localizer) Translate(key message.Reference, args ...interface{}) string { return l.printer.Sprintf(key, args...) } Note: Notice here that we're initializing a single message.Printer for each locale at startup, and these will be used concurrently by our web application handlers. Although the golang.org/x/text/message documentation doesn't say that message.Printer is safe for concurrent use, I checked with Marcel van Lohuizen (the lead developer of the golang.org/x/text packages) and he confirmed that message.Printer is intended to be used concurrently and is concurrency safe (so long as access to any write destination is synchronized). Next let's update the cmd/www/handlers.go file to use our new Localizer type, and — while we're at it — let's also make our handleHome() function render an additional "Launching soon!" message. File: cmd/www/handlers.go package main import ( "fmt" // New import "net/http" "bookstore.example.com/internal/localizer" // New import ) func handleHome(w http.ResponseWriter, r *http.Request) { // Initialize a new Localizer based on the locale ID in the URL. l, ok := localizer.Get(r.URL.Query().Get(":locale")) if !ok { http.NotFound(w, r) return } var totalBookCount = 1_252_794 // Update these to use the new Translate() method. fmt.Fprintln(w, l.Translate("Welcome!")) fmt.Fprintln(w, l.Translate("%d books available", totalBookCount)) // Add an additional "Launching soon!" message. fmt.Fprintln(w, l.Translate("Launching soon!")) } It's worth pointing out that our use of the Translate() method here isn't just some syntactic sugar. You might remember earlier that I wrote the following warning: It's critical to note when gotext walks your code it actually only looks for calls to message.Printer.Printf(), Fprintf() and Sprintf() — basically the three methods that end with an f. It ignores all other methods such as Sprint() or Println(). By having all our translations go through the Translate() method — which uses Sprintf() behind-the-scenes — we avoid the scenario where you accidentally use a method like Sprint() or Println() and gotext doesn't extract the message to the out.gotext.json files. Let's try this out and run go generate again: $ go generate ./internal/translations/translations.go de-DE: Missing entry for "Launching soon!". fr-CH: Missing entry for "Launching soon!". So this is really smart. We can see that gotext has been clever enough to walk our entire codebase and identify what strings need to be translated, even when we abstract the message.Printer.Sprintf() call to a helper function in a different package. This is awesome, and one of the things that I really appreciate about the gotext tool. If you're following along, please go ahead and copy the out.gotext.json files to message.gotext.json files, and add the necessary translations for the new "Launching soon!" message. Then remember to run go generate again and restart the web application. When you make some HTTP requests again now, your responses should look similar to this: $ curl localhost:4018/en-gb Welcome! 1,252,794 books available Launching soon! $ curl localhost:4018/de-de Willkommen! 1.252.794 Bücher erhältlich Bald verfügbar! $ curl localhost:4018/fr-ch Bienvenu ! 1 252 794 livres disponibles Bientôt disponible ! Additional information Conflicting routes At this start of this post I'd deliberately didn't recommending using httprouter, despite it being an excellent and popular router. This is because using a dynamic locale as the first part of a URL path is likely to result in conflicts with other application routes which don't require a locale prefix, like /static/css/main.css or /admin/login. The httprouter package doesn't allow conflicting routes, which makes using it awkward in this scenario. If you do want to use httprouter, or want to avoid conflicting routes in your application, you could pass the locale as a query string parameter instead like /category/travel?locale=gb.

• How to correctly use Basic Authentication in Go
  Jun 21, 2021

  When searching for examples of HTTP basic authentication with Go, every result I found unfortunately contained code which was either out-of-date (i.e. doesn't use the r.BasicAuth() functionality that was introduced in Go 1.4) or failed to mitigate the risk of timing attacks. So in this post, I'd like to discuss how to use it correctly in your Go applications. We'll start with a bit of background information, but if you're not interested in that you can skip straight to the code. What is basic authentication? When should I use it? As a developer, you're probably already familiar with the prompt that web browsers show when you visit a URL that is protected with basic authentication. When you input a username and password into this prompt, the web browser will send a HTTP request to the server containing an Authorization header — similar to this: Authorization: Basic YWxpY2U6cGE1NXdvcmQ= The Authorization header value is made up of the string Basic followed by the username and password in the format username:password and base-64 encoded. In this specific example, YWxpY2U6cGE1NXdvcmQ= is the base-64 encoding of the value alice:pa55word. When the server receives this request, it can decode the username and password from the Authorization header and check that they are valid. If the credentials are not valid, the server can return a 401 Unauthorized response and the browser can redisplay the prompt. Basic authentication can be used in lots of different scenarios, but it's often a good fit for when you have a low-value resource and want a quick and easy way to protect it from prying eyes. To help keep things secure you should: Only ever use it over HTTPS connections. If you don't use HTTPS, the Authorization header can potentially be intercepted and decoded by an attacker, who can then use the username and password to gain access to your protected resources. Use a strong password that is difficult for attackers to guess or brute-force. Consider adding rate limiting to your application, to make it harder for an attacker to brute-force the credentials. On the client side, basic auth is supported out-of-the-box by most programming languages and command-line tools such as curl and wget, as well as web browsers. Protecting a web application Probably the simplest way to use basic authentication in your application is to create some middleware. In this middleware we want to do two things: Extract the provided username and password from the request Authorization header, if it exists. The best way to do this is with the r.BasicAuth() method. Compare the provided username and password against the values that you expect. If the username and password are not correct, or the request didn't contain a valid Authorization header, then the middleware should send a 401 Unauthorized response and set a WWW-Authenticate header to inform the client that basic authentication should be used to gain access. Otherwise, the middleware should allow the request to proceed and call the next handler in the chain. When comparing the provided username and password against the expected values, to eliminate the risk of a timing attack you should use Go's subtle.ConstantTimeCompare() function instead of the == operator. Note: In Go (like most languages) the normal == comparison operator will return as soon as it finds a difference between two strings. So if the first character is different, it will return after just looking at one character. In theory, this opens the opportunity for a timing attack where an attacker could make lots of requests to your application, and look at discrepancies in the average response time. The time it takes for them receive a 401 Unauthorized response effectively tells them how many characters they got right. With enough requests, they could build up a picture of the complete username and password. Realistically though, string comparison is so fast that network jitter will obscure any differences in timing, meaning that it is probably impossible to pull off this attack successfully. But... there is some evidence that remote timing attacks are feasible, and given that we can quite easily eliminate this risk completely by using subtle.ConstantTimeCompare(), I think it makes sense to do so. It's also important to be aware that using subtle.ConstantTimeCompare() can leak information about username and password length. To prevent this, we should hash both the provided and expected username and password values using a fast cryptographic hash function like SHA-256 before comparing them. This ensures that both the provided and expected values that we are comparing are equal in length and prevents subtle.ConstantTimeCompare() itself from returning early. Putting that together, the pattern for implementing some middleware looks like this: func basicAuth(next http.HandlerFunc) http.HandlerFunc { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Extract the username and password from the request // Authorization header. If no Authorization header is present // or the header value is invalid, then the 'ok' return value // will be false. username, password, ok := r.BasicAuth() if ok { // Calculate SHA-256 hashes for the provided and expected // usernames and passwords. usernameHash := sha256.Sum256([]byte(username)) passwordHash := sha256.Sum256([]byte(password)) expectedUsernameHash := sha256.Sum256([]byte("your expected username")) expectedPasswordHash := sha256.Sum256([]byte("your expected password")) // Use the subtle.ConstantTimeCompare() function to check if // the provided username and password hashes equal the // expected username and password hashes. ConstantTimeCompare // will return 1 if the values are equal, or 0 otherwise. // Importantly, we should do the work to evaluate both the // username and password before checking the return values to // avoid leaking information. usernameMatch := (subtle.ConstantTimeCompare(usernameHash[:], expectedUsernameHash[:]) == 1) passwordMatch := (subtle.ConstantTimeCompare(passwordHash[:], expectedPasswordHash[:]) == 1) // If the username and password are correct, then call // the next handler in the chain. Make sure to return // afterwards, so that none of the code below is run. if usernameMatch && passwordMatch { next.ServeHTTP(w, r) return } } // If the Authentication header is not present, is invalid, or the // username or password is wrong, then set a WWW-Authenticate // header to inform the client that we expect them to use basic // authentication and send a 401 Unauthorized response. w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`) http.Error(w, "Unauthorized", http.StatusUnauthorized) }) } Important: If you're looking at the code above and thinking "I thought you should never use SHA-256 for hashing passwords...", it's important to emphasize that the username and password are not being hashed for the purpose of storage, they are only being hashed in order to get two equal-length byte slices that can be compared in constant-time. Low collision risk is the important thing here, and a hash like SHA-256 is a good fit for this purpose. You might also be wondering what the realm value is and why we are setting it to "restricted" in the WWW-Authenticate response header. Basically, the realm value is a string which allows you to create partitions of protected space in your application. So, for example, an application could have a "documents" realm and an "admin area" realm, which require different credentials. A web browser (or other type of client) can cache and automatically reuse the same username and password for any requests within the same realm, so that the prompt doesn't need to be shown for every single request. If you don't require multiple partitions for your application, you can set the realm to a single hardcoded value like "restricted", like we have in the code above. For the sake of security and/or flexibility, you may also prefer to store the expected username and password values in environment variables or pass them as command-line flag values when starting the application, rather than hard-coding them into your application. A working example Let's take a quick look at this in the context of a small — but fully functioning — web application. If you'd like to follow along, create a new basic-auth-example directory on your computer, add a main.go file, initialize a module, and create a pair of locally-trusted TLS certificates using the mkcert tool. Like so: $ mkdir basic-auth-example $ cd basic-auth-example $ touch main.go $ go mod init example.com/basic-auth-example go: creating new go.mod: module example.com/basic-auth-example $ mkcert localhost Created a new certificate valid for the following names 📜 - "localhost" The certificate is at "./localhost.pem" and the key at "./localhost-key.pem" ✅ It will expire on 21 September 2023 🗓 $ ls go.mod localhost-key.pem localhost.pem main.go Then add the following code to the main.go file, so that the application reads the expected username and password from environment variables and uses the middleware pattern that we described above. package main import ( "crypto/sha256" "crypto/subtle" "fmt" "log" "net/http" "os" "time" ) type application struct { auth struct { username string password string } } func main() { app := new(application) app.auth.username = os.Getenv("AUTH_USERNAME") app.auth.password = os.Getenv("AUTH_PASSWORD") if app.auth.username == "" { log.Fatal("basic auth username must be provided") } if app.auth.password == "" { log.Fatal("basic auth password must be provided") } mux := http.NewServeMux() mux.HandleFunc("GET /unprotected", app.unprotectedHandler) mux.HandleFunc("GET /protected", app.basicAuth(app.protectedHandler)) srv := &http.Server{ Addr: ":4000", Handler: mux, IdleTimeout: time.Minute, ReadTimeout: 10 * time.Second, WriteTimeout: 30 * time.Second, } log.Printf("starting server on %s", srv.Addr) err := srv.ListenAndServeTLS("./localhost.pem", "./localhost-key.pem") log.Fatal(err) } func (app *application) protectedHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "This is the protected handler") } func (app *application) unprotectedHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "This is the unprotected handler") } func (app *application) basicAuth(next http.HandlerFunc) http.HandlerFunc { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { username, password, ok := r.BasicAuth() if ok { usernameHash := sha256.Sum256([]byte(username)) passwordHash := sha256.Sum256([]byte(password)) expectedUsernameHash := sha256.Sum256([]byte(app.auth.username)) expectedPasswordHash := sha256.Sum256([]byte(app.auth.password)) usernameMatch := (subtle.ConstantTimeCompare(usernameHash[:], expectedUsernameHash[:]) == 1) passwordMatch := (subtle.ConstantTimeCompare(passwordHash[:], expectedPasswordHash[:]) == 1) if usernameMatch && passwordMatch { next.ServeHTTP(w, r) return } } w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`) http.Error(w, "Unauthorized", http.StatusUnauthorized) }) } You should then be able to start the application, using a pair of temporary AUTH_USERNAME and AUTH_PASSWORD environment variables. Like so: $ AUTH_USERNAME=alice AUTH_PASSWORD=p8fnxeqj5a7zbrqp go run . 2021/06/20 16:09:21 starting server on :4000 At this point, if you open your web browser and visit https://localhost:4000/protected you should be greeted by the basic authentication prompt. Alternatively, you can make some requests using curl to verify that the authentication checks are working correctly. $ curl -i https://localhost:4000/unprotected HTTP/2 200 content-type: text/plain; charset=utf-8 content-length: 32 date: Sun, 20 Jun 2021 14:09:56 GMT This is the unprotected handler $ curl -i https://localhost:4000/protected HTTP/2 401 content-type: text/plain; charset=utf-8 www-authenticate: Basic realm="restricted", charset="UTF-8" x-content-type-options: nosniff content-length: 13 date: Sun, 20 Jun 2021 14:09:59 GMT Unauthorized $ curl -i -u alice:p8fnxeqj5a7zbrqp https://localhost:4000/protected HTTP/2 200 content-type: text/plain; charset=utf-8 content-length: 30 date: Sun, 20 Jun 2021 14:10:14 GMT This is the protected handler $ curl -i -u alice:wrongPa55word https://localhost:4000/protected HTTP/2 401 content-type: text/plain; charset=utf-8 www-authenticate: Basic realm="restricted", charset="UTF-8" x-content-type-options: nosniff content-length: 13 date: Sun, 20 Jun 2021 14:15:30 GMT Unauthorized Making a request to a protected resource Finally, if you need to access a protected resource from your Go code as a client, all you need to do is call the r.SetBasicAuth() method on your request before executing it. Like so: package main import ( "fmt" "io" "log" "net/http" "time" ) func main() { client := http.Client{Timeout: 5 * time.Second} req, err := http.NewRequest(http.MethodGet, "https://localhost:4000/protected", http.NoBody) if err != nil { log.Fatal(err) } req.SetBasicAuth("alice", "p8fnxeqj5a7zbrqp") res, err := client.Do(req) if err != nil { log.Fatal(err) } defer res.Body.Close() resBody, err := io.ReadAll(res.Body) if err != nil { log.Fatal(err) } fmt.Printf("Status: %d\n", res.StatusCode) fmt.Printf("Body: %s\n", string(resBody)) }

• Surprises and gotchas when working with JSON
  Sep 24, 2020

  This is a list of things about Go's encoding/json package which, over the years, have either confused or surprised me when I first encountered them. Many of these things are mentioned in the official package documentation if you read it carefully enough, so in theory they shouldn't come as a surprise. But a few of them aren't mentioned in the documentation at all — or at least, they aren't pointed out explicitly — and are worth being aware of! Map entries are sorted alphabetically Byte slices are encoded as base-64 strings Nil and empty slices are encoded differently Integer, time.Time and net.IP values can be used as map keys Angle brackets and ampersands in strings are escaped Trailing zeroes are removed from floats Using omitempty on an zero-valued struct doesn't work Using omitempty on a zero-value time.Time doesn't work There is a 'string' struct tag Non-ASCII punctuation characters aren't supported in struct tags Decoding a JSON number into an interface{} yields a float64 Don't use More() to check if there are remaining JSON objects in a stream String values returned by custom MarshalJSON() methods must be quoted Map entries are sorted alphabetically When encoding a Go map to JSON, the entries will be sorted alphabetically based on the map key. For example, the following map: m := map[string]int{ "z": 123, "0": 123, "a": 123, "_": 123, } Will be encoded to the JSON: {"0":123,"_":123,"a":123,"z":123} Byte slices are encoded as base-64 strings Any []byte slices will be converted to a base64-encoded string when encoding them to JSON. The base64 string uses padding and the standard encoding characters, as defined in RFC 4648. For example, the following map: m := map[string][]byte{ "foo": []byte("bar baz"), } Will be encoded to the JSON: {"foo":"YmFyIGJheg=="} Nil and empty slices are encoded differently Nil slices in Go will be encoded to the null JSON value. In contrast, an empty (but not nil) slice will be encoded as an empty JSON array. For example: var nilSlice []string emptySlice := []string{} m := map[string][]string{ "nilSlice": nilSlice, "emptySlice": emptySlice, } Will be encoded to the JSON: {"emptySlice":[],"nilSlice":null} Integer, time.Time and net.IP values can be used as map keys It's possible to encode a map which has integer values as the map keys. These integers will be automatically converted to strings in the resulting JSON (because the keys in a JSON object must always be strings). For example: m := map[int]string{ 123: "foo", 456_000: "bar", } Will be encoded to the JSON: {"123":"foo","456000":"bar"} In addition, Go allows you to encode maps with keys that implement the encoding.TextMarshaler interface. This means that you can also use time.Time and net.IP values as map keys out-of-the-box. For example: t1 := time.Now() t2 := t1.Add(24 * time.Hour) m := map[time.Time]string{ t1: "foo", t2: "bar", } Will be encoded to the JSON: {"2009-11-10T23:00:00Z":"foo","2009-11-11T23:00:00Z":"bar"} Note that trying to encode a map with any other type of key will result in a json.UnsupportedTypeError error. Angle brackets and ampersands in strings are escaped If a string contains angle brackets<> these will be escaped to \u003c and \u003e in the JSON output. Likewise the & character will be escaped to \u0026. This is to prevent some web browsers from accidentally interpreting the JSON as HTML. For example: s := []string{ "", "bar & baz", } Will be encoded to the JSON: ["\u003cfoo\u003e","bar \u0026 baz"] If you need to prevent these characters being escaped, you should use a json.Encoder instance and call SetEscapeHTML(false). An example is here. Trailing zeroes are removed from floats When encoding a floating-point number with a fractional part that ends in zero(es), any trailing zeroes will not appear in the JSON. For example: s := []float64{ 123.0, 456.100, 789.990, } Will be encoded to the JSON: [123,456.1,789.99] Using omitempty on an zero-valued struct doesn't work The omitempty directive never considers a struct type to be empty — even if all the struct fields have their zero value, and you use omitempty on those fields too. It will always appear as an object in the encoded JSON. For example: s := struct { Foo struct { Bar string `json:",omitempty"` } `json:",omitempty"` }{} Will be encoded to the JSON: {"Foo":{}} There’s a long-standing proposal which discusses changing this behavior, but the Go 1 compatibility promise means that it's unlikely to happen any time soon. Instead, you can get around this by making the field a pointer to a struct, which works because omitempty considers nil pointers to be empty. For example: s := struct { Foo *struct { Bar string `json:",omitempty"` } `json:",omitempty"` }{} Using omitempty on a zero-value time.Time doesn't work Using omitempty on a zero-value time.Time field won't hide it in the encoded JSON. This is because the time.Time type is a struct behind the scenes and, as mentioned above, omitempty never considers a struct type to be empty. Instead, the string "0001-01-01T00:00:00Z" will appear in the JSON (which is the value returned by calling the MarshalJSON() method on an zero-value time.Time. For example: s := struct { Foo time.Time `json:",omitempty"` }{} Will be encoded to the JSON: {"Foo":"0001-01-01T00:00:00Z"} There is a 'string' struct tag Go provides a string struct tag directive which forces the data in an individual field to be encoded as a string in the resulting JSON. For example, if you want to force an integer to be represented as a string instead of an JSON number you can use the string directive like so: s := struct { Foo int `json:",string"` }{ Foo: 123, } And this will be encoded to the JSON: {"Foo":"123"} Note that the string struct tag directive will only work on fields which contain float, integer or bool types. For any other type it will have no effect. Non-ASCII punctuation characters aren't supported in struct tags When using struct tags to change key names in JSON, any tags containing non-ASCII punctuation characters will be ignored. Notably this means that you can't use en or em dashes, or most currency signs, in struct tags. For example: s := struct { CostUSD string `json:"cost $"` // OK CostEUR string `json:"cost €"` // Contains the non-ASCII punctuation character €. Will be ignored. }{ CostUSD: "100.00", CostEUR: "100.00", } Will be encoded to the following JSON (notice that the struct tag renaming the CostEUR field has been ignored): {"cost $":"100.00","CostEUR":"100.00"} Likewise, any struct tags containing non-ASCII punctuation characters will be ignored when decoding values from a JSON object into a struct, and the struct field will be left with its zero value. For example the following code: js := []byte(`{"cost $":"100.00","cost €":"100.00"}`) s := struct { CostUSD string `json:"cost $"` CostEUR string `json:"cost €"` }{} err := json.Unmarshal(js, &s) if err != nil { log.Fatal(err) } fmt.Printf("%+v", s) Will print out: {CostUSD:100.00 CostEUR:} This can be annoying in situations where you need to decode a JSON object that has keys containing non-ASCII characters, and you can't change the JSON. To work around this limitation, you can decode to a map as an intermediary step, and then copy the data from the map to the struct. For example the following code: js := []byte(`{"cost $":"100.00","cost €":"100.00"}`) var aux map[string]string err := json.Unmarshal([]byte(js), &aux) if err != nil { log.Fatal(err) } s := struct { CostUSD string `json:"cost $"` CostEUR string `json:"cost €"` }{ CostUSD: aux["cost $"], CostEUR: aux["cost €"], } fmt.Printf("%+v", s) Will print out: {CostUSD:100.00 CostEUR:100.00} Decoding a JSON number into an interface{} yields a float64 When decoding a JSON number into an interface{}, the value will have the underlying type float64 — even if it is an integer in the original JSON. If you want to get the value as an integer (instead of a float64) the most robust approach is to decode the JSON using a json.Decoder instance with the UseNumber() method set on it. This will decode all JSON numbers to the underlying type json.Number instead of float64, and you can then access the number as an integer using its Int64() method. For example: js := `{"foo": 123, "bar": true}` var m map[string]interface{} dec := json.NewDecoder(strings.NewReader(js)) dec.UseNumber() err := dec.Decode(&m) if err != nil { log.Fatal(err) } i, err := m["foo"].(json.Number).Int64() if err != nil { log.Fatal(err) } fmt.Printf("foo: %d", i) Will print: foo: 123 Don't use More() to check if there are remaining JSON objects in a stream When processing a stream of JSON objects with json.Decoder, don't use the More() method to check if there is a remaining object in the stream. Depsite its name, More() is not designed for this purpose†, and trying to use it in this way may cause some subtle problems. †The More() method is intended to be used in conjunction with Token(), and exists specifically to check if there is another element in the array or object currently being parsed. For example, if you use it when decoding an invalid JSON stream like {"name": "alice"}{"name": "bob"}] (notice the additional square bracket at the end) it won't result in an error (when it should!). Like so: js := `{"name": "alice"}{"name": "bob"}]` dec := json.NewDecoder(strings.NewReader(js)) for { var user map[string]string err := dec.Decode(&user) if err != nil { log.Fatal(err) } fmt.Printf("%v\n", user) // Don't do this! if !dec.More() { break } } This code will run without error and output: map[name:alice] map[name:bob] The correct technique to see if a stream contains another JSON object is to check for an io.EOF error, which will be returned when there are no more objects to process in the stream. Like so: js := `{"name": "alice"}{"name": "bob"}]` dec := json.NewDecoder(strings.NewReader(js)) for { var user map[string]string err := dec.Decode(&user) if err != nil { if errors.Is(err, io.EOF) { break } log.Fatal(err) } fmt.Printf("%v\n", user) } Running this will correctly result in an error, as we would expect given the invalid input: map[name:alice] map[name:bob] 2009/11/10 23:00:00 invalid character ']' looking for beginning of value String values returned by custom MarshalJSON() methods must be quoted If you are creating a custom MarshalJSON() method which returns a string value, you must wrap the string in double quotes before returning it, otherwise it won't be interpreted as a JSON string and will result in a runtime error. For example: type Age int func (age Age) MarshalJSON() ([]byte, error) { encodedAge := fmt.Sprintf("%d years", age) encodedAge = strconv.Quote(encodedAge) // Wrap the string in quotes before returning. return []byte(encodedAge), nil } func main() { users := map[string]Age{ "alice": 21, "bob": 84, } js, err := json.Marshal(users) if err != nil { log.Fatal(err) } fmt.Printf("%s", js) } Will result in the following JSON being printed: {"alice":"21 years","bob":"84 years"} If, in the code above, you didn't quote the return value from MarshalJSON() you will get the error: 2009/11/10 23:00:00 json: error calling MarshalJSON for type main.Age: invalid character 'y' after top-level value

• How to manage database timeouts and cancellations in Go
  Apr 20, 2020

  One of the great features of Go's database/sql package is that it's possible to cancel database queries while they are still running via a context.Context instance. On the face of it, usage of this functionality is quite straightforward (here's a basic example). But once you start digging into the details there's a lot a nuance and quite a few gotchas... especially if you are using this functionality in the context of a web application or API. So in this post I want to explain how to cancel database queries in a web application, what behavioral quirks and edge cases it is important to be aware of, and try to provide answers to the questions that you might have when working through all this. But first off, why would you want to cancel a database query? Two scenarios spring to mind: When a query is taking a lot longer to complete than expected. If this happens, it suggests a problem — either with that particular query or your database or application more generally. In this scenario, you would probably want to cancel the query after a set period of time (so that resources are freed-up and the database connection is returned to the sql.DB connection pool for reuse), log an error for further investigation, and return a 500 Internal Server Error response to the client. When a client goes away unexpectedly before the query completes. This could happen for a number of reasons, such as a user closing a browser tab or terminating a process. In this scenario, nothing has really gone 'wrong', but there is no client left to return a response to so you may as well cancel the query and free-up the resources. Mimicking a long-running query Let's start with the first scenario. To demonstrate this, I'll make a very basic web application with a handler that executes a SELECT pg_sleep(10) SQL query against a PostgreSQL database using the pq driver. The pg_sleep(10) function will make the query sleep for 10 seconds before returning, essentially mimicking a slow-running query. package main import ( "database/sql" "fmt" "log" "net/http" _ "github.com/lib/pq" ) var db *sql.DB func slowQuery() error { _, err := db.Exec("SELECT pg_sleep(10)") return err } func main() { var err error db, err = sql.Open("postgres", "postgres://user:pa$$word@localhost/example_db") if err != nil { log.Fatal(err) } if err = db.Ping(); err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/", exampleHandler) log.Print("Listening...") err = http.ListenAndServe(":5000", mux) if err != nil { log.Fatal(err) } } func exampleHandler(w http.ResponseWriter, r *http.Request) { err := slowQuery() if err != nil { serverError(w, err) return } fmt.Fprintln(w, "OK") } func serverError(w http.ResponseWriter, err error) { log.Printf("ERROR: %s", err.Error()) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) } If you were to run this code, then make a GET / request to the application you should find that the request hangs for 10 seconds before you finally get an "OK" response. Like so: $ curl -i localhost:5000/ HTTP/1.1 200 OK Date: Fri, 17 Apr 2020 07:46:40 GMT Content-Length: 3 Content-Type: text/plain; charset=utf-8 OK Note: The structure of the application code above is deliberately over-simplified. In a real project I would recommend using dependency injection to make the sql.DB connection pool and logger available to your handlers, instead of using global variables. Adding a context timeout OK, now that we've got some code that mimics a long-running query, let's enforce a timeout on the query so it is automatically canceled if it doesn't complete within 5 seconds. To do this we need to: Use the context.WithTimeout() function to create a context.Context instance with a 5-second timeout duration. Execute the SQL query using the ExecContext() method, passing the context.Context instance as a parameter. I'll demonstrate: package main import ( "context" // New import "database/sql" "fmt" "log" "net/http" "time" // New import _ "github.com/lib/pq" ) var db *sql.DB func slowQuery(ctx context.Context) error { // Create a new child context with a 5-second timeout, using the // provided ctx parameter as the parent. ctx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() // Pass the child context (the one with the timeout) as the first // parameter to ExecContext(). _, err := db.ExecContext(ctx, "SELECT pg_sleep(10)") return err } ... func exampleHandler(w http.ResponseWriter, r *http.Request) { // Pass the request context to slowQuery(), so it can be used as the // parent context. err := slowQuery(r.Context()) if err != nil { serverError(w, err) return } fmt.Fprintln(w, "OK") } ... There are a few things about this that I'd like to emphasize and explain: Note that we pass r.Context() (the request context) to slowQuery() to use as the parent context. As we'll see in the next section, this is important because it means that any cancellation signal on the request context will be able to 'bubble down' to the context that we use in ExecContext(). The defer cancel() line is important because it ensures that the resources associated with our child context (the one with the timeout) will be released before the slowQuery() function returns. If we don't call cancel() it may cause a memory leak: the resources won't be released until either the parent r.Context() is canceled or the 5-second timeout is hit (whichever happens first). The timeout countdown begins from the moment that the child context is created using context.WithTimeout(). If you want more control over this you could use the alternative context.WithDeadline() function, which allows you to set an explicit time.Time value for when the context should timeout instead. OK, let's try this out. If you run the application again and make a GET / request, after a 5-second delay you should get a response like this: $ curl -i localhost:5000/ HTTP/1.1 500 Internal Server Error Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Fri, 17 Apr 2020 08:21:14 GMT Content-Length: 28 Sorry, something went wrong And if you go back to the terminal window running the application you should see a log message similar to this: $ go run . 2020/04/17 10:21:07 Listening... 2020/04/17 10:21:14 ERROR: pq: canceling statement due to user request That log message might seem a bit odd... until you realize that the error message is actually coming from PostgreSQL. In that light it makes sense: our web application is the user and we're canceling the query after 5 seconds. So this is actually really good; things are working as we want. Specifically, after 5 seconds the context timeout is reached and the pq driver sends a cancellation signal to PostgreSQL†. PostgreSQL then terminates the running query (thereby freeing-up resources). The client is sent a 500 Internal Server Error response, and the error message is logged so we know that something has gone wrong. † More precisely, our child context (the one with the 5-second timeout) has a Done channel, and when the timeout is reached it will close the Done channel. While the SQL query is running, our database driver pq is also running a background goroutine which listens on this Done channel. If the channel is closed, then it sends a cancellation signal to PostgreSQL. PostgreSQL terminates the query, and then sends the error message that we see above as a response to the original pq goroutine. That error message is then returned to our slowQuery() function. Dealing with closed connections OK, let's try one more thing. Let's use curl to make a GET / request and then very quickly (within 5 seconds) press Ctrl+C to cancel the request. If you look at the logs for the application again, you should see another log line with exactly the same error message that we saw before. $ go run . 2020/04/17 10:21:07 Listening... 2020/04/17 10:21:14 ERROR: pq: canceling statement due to user request 2020/04/17 10:41:18 ERROR: pq: canceling statement due to user request So what's happening here? In this case, the request context (which we use as the parent in our code above) is canceled because the client closed the connection. From the net/http docs: For incoming server requests, the [request] context is canceled when the client's connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns. This cancellation signal bubbles down to our child context, it's Done channel is closed, and the pq driver terminates the running query in exactly the same way as before. With that in mind, it's not surprising that we see the same error message... From a PostgreSQL point of view exactly the same thing is happening as when the timeout was reached. But from the perspective of our web application the scenario is very different. A client connection being closed can happen for many different, innocuous, reasons. It's not really an error from our application's point of view, although it is probably sensible to log it as a warning (if we start to see elevated rates, it could be a sign that something is wrong). Fortunately, it's possible to tell these two scenarios apart by calling the ctx.Err() method on our child context. If the context was canceled (due to a client closing the connection), then ctx.Err() will return context.Canceled. If the timeout was reached, then it will return context.DeadlineExceeded. If both the deadline is reached and the context is canceled, then ctx.Err() will surface whichever happened first. There's another important thing to point out here: it's possible that a timeout/cancellation will happen before the PostgreSQL query even starts. For example you might have set MaxOpenConns() on your sql.DB connection pool, and if that open connection limit is reached and all connections are in-use, then the query will be 'queued' by sql.DB until a connection becomes available. In this scenario — or any other which causes a delay — it's quite possible that the timeout/cancellation will occur before a free database connection even becomes available. In this case ExecContext() will directly return the ctx.Err() value as the error response (instead of the "pq: canceling statement due to user request" error that we see above). If you're using the QueryContext() method then it's also possible that the timeout/cancellation will occur when processing the data with Scan(). If this happens, then Scan() will directly return the ctx.Err() value as an error. As far as I can see this behavior isn't mentioned in the database/sql docs, but I can confirm that this is the case with Go 1.14 and the comments on issue #28842 suggest that it is intentional. Putting all that together, a sensible approach is to check for the error "pq: canceling statement due to user request" and then wrap this with the error from ctx.Err() before returning from our slowQuery() function. Then in our handler, we can use the errors.Is() function to check if the error from slowQuery() is equal to (or wraps) context.Canceled and manage it accordingly. Like so: package main import ( "context" "database/sql" "errors" // New import "fmt" "log" "net/http" "time" _ "github.com/lib/pq" ) var db *sql.DB func slowQuery(ctx context.Context) error { ctx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() _, err := db.ExecContext(ctx, "SELECT pg_sleep(10)") // If we get a "pq: canceling statement..." error wrap it with the // context error before returning. if err != nil && err.Error() == "pq: canceling statement due to user request" { return fmt.Errorf("%w: %v", ctx.Err(), err) } return err } ... func exampleHandler(w http.ResponseWriter, r *http.Request) { err := slowQuery(r.Context()) if err != nil { // Check if the returned error equals or wraps context.Canceled and // record a warning if it does. switch { case errors.Is(err, context.Canceled): serverWarning(err) default: serverError(w, err) } return } fmt.Fprintln(w, "OK") } func serverWarning(err error) { log.Printf("WARNING: %s", err.Error()) } ... If you were to run this application again now and make two different GET / requests — one that times out, and the other that you cancel — you should see clearly different messages in the application log, like so: $ go run . 2020/04/17 13:09:25 Listening... 2020/04/17 13:09:45 ERROR: context deadline exceeded: pq: canceling statement due to user request 2020/04/17 13:09:47 WARNING: context canceled: pq: canceling statement due to user request Other context-aware methods The database/sql package provides context-aware variants for most actions on sql.DB, including PingContext(), QueryContext(), and QueryRowContext(). We can (and should!) update the main() function in the code above to use PingContext() instead of Ping(). In this case there is no request context to use as the parent, so we need to create an empty parent context with context.Background() instead. Like so: ... func main() { var err error db, err = sql.Open("postgres", "postgres://user:pa$$word@localhost/example_db") if err != nil { log.Fatal(err) } // Create a context with a 10-second timeout, using the empty // context.Background() as the parent. ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() // Use this when testing the connection pool. if err = db.PingContext(ctx); err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/", exampleHandler) log.Print("Listening...") err = http.ListenAndServe(":5000", mux) if err != nil { log.Fatal(err) } } ... Can I set a global timeout for all requests? Sure, you could create and use some middleware on your routes which adds a timeout to the current request context, similar to this: func setTimeout(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second) defer cancel() // This gives you a copy of the request with a the request context // changed to the new context with the 5-second timeout created // above. r = r.WithContext(ctx) next.ServeHTTP(w, r) }) } If you take this approach there are a few of things to be aware of: The timeout starts from the moment the context is created, so any code running in your handlers before the database query counts towards the timeout. If you have multiple queries being executed in a handler, then they all have to complete within that one time. The timeout will continue to apply even if you derive a child context with a different timeout duration. So while you can enforce an earlier timeout in a child context, you can't make it longer. What about http.TimeoutHandler? Go provides a http.TimeoutHandler() middleware function which you can use to wrap your handlers or router/servemux. This works similar to the middleware above in the sense that it sets a timeout on the request context... so the warnings above also apply when using this. However, http.TimeoutHandler() also sends the client a 503 Service Unavailable response and a HTML error message. So, if you're using this in your application, you shouldn't (or at least, you don't need to) send the client an error response yourself when encountering a context.DeadlineExceeded error. How about transactions? How does context work in those? The database/sql package provides a BeginTx() method which you can use to initiate a context-aware transaction. A code example can be seen here. It's important to understand that the context you provide to BeginTx() applies to the whole transaction. In the event of a timeout/cancellation on the context, then the queries in the transaction will automatically be rolled-back. It's perfectly fine to pass the same context as a parameter for all the queries in the transaction, in which case it ensures that they all (as a whole) complete before any timeout/cancellation . Alternatively, if you want per-query timeouts you can create different child contexts with different timeouts for each in the queries in the transaction. But you must derive these child contexts from the context you passed to BeginTX(). Otherwise there is a risk that the BeginTX() context timeout/cancellation occurs and the automatic rollback happens, but your code still may try to execute the query with a still-live context. If that happened you would receive the error "sql: transaction has already been committed or rolled back". What about background processing? When doing background-processing in a different goroutine, bear in mind that if a parent context is canceled, the cancellation signal 'bubbles down' to its children. And also bear in mind what I quoted earlier about request context cancellation: For incoming server requests, the [request] context is canceled ... when the ServeHTTP method returns. Combine those two things, and it means that if you use a context which is a child of the request context in the background-process, the background-process will get a cancellation signal when the HTTP response is sent for the initial request. If you don't want that to be the case (and you probably don't), then you should create a brand-new context for the background-process using context.Background() and copy over any values that you need... or just pass them as regular parameters instead. If a context is canceled, can I be confident that it's due to a closed connection? Yes — so long as it's within the main goroutine for the request, it's a child of the request context, and you haven't manually canceled it yourself yet using defer cancel(). Otherwise, no. Is the behavior the same with other databases and drivers? I'm not sure. I've only used these features extensively with PostgreSQL and the pq driver. I imagine that things will be roughly the same with other databases and drivers, but you'll need to check. Anything else I should know? Yep. This is a strange one and it's not officially documented yet, but if a client makes a request with a non-empty request body then closes the connection, the context won't be canceled until after you have read the request body. This doesn’t apply to requests without a request body, where the cancellation signal will be received immediately. You should also be aware of the WriteTimeout setting on your http.Server (if you have set one). Your context timeouts should always be shorter than your WriteTimeout value, otherwise the WriteTimeout will be hit first, the connection will be closed, and the client won’t get any response.

• How to parse a JSON request body in Go
  Oct 21, 2019

  Let's say that you're building a JSON API with Go. And in some of the handlers — probably as part of a POST or PUT request — you want to read a JSON object from the request body and assign it to a struct in your code. After a bit of research, there's a good chance that you'll end up with some code that looks similar to the personCreate handler here: // File: main.go package main import ( "encoding/json" "fmt" "log" "net/http" ) type Person struct { Name string Age int } func personCreate(w http.ResponseWriter, r *http.Request) { // Declare a new Person struct. var p Person // Try to decode the request body into the struct. If there is an error, // respond to the client with the error message and a 400 status code. err := json.NewDecoder(r.Body).Decode(&p) if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } // Do something with the Person struct... fmt.Fprintf(w, "Person: %+v", p) } func main() { mux := http.NewServeMux() mux.HandleFunc("/person/create", personCreate) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } If you're putting together a quick prototype, or building an API for personal/internal use only, then the code in the personCreate handler is probably OK. But if you're building an API for public use in production then there are a few issues with this to be aware of, and things that can be improved. Not all errors returned by Decode() are caused by a bad request from the client. Specifically, Decode() can return a json.InvalidUnmarshalError error — which is caused by an unmarshalable target destination being passed to Decode(). If that happens, then it indicates a problem with our application — not the client request — so really the error should be logged and a 500 Internal Server Error response sent to the client instead. The error messages returned by Decode() aren't ideal for sending to a client. Some are arguably too detailed and expose information about the underlying program (like "json: cannot unmarshal number into Go struct field Person.Name of type string"). Others aren't descriptive enough (like "unexpected EOF") and some are just plain confusing (like "invalid character 'A' looking for beginning of object key string"). There also isn't consistency in the formatting or language used. A client can include extra unexpected fields in their JSON, and these fields will be silently ignored without the client receiving any error. We can fix this by using the decoder's DisallowUnknownFields() method. There's no upper limit on the size of the request body that will be read by the Decode() method. Limiting this would help prevent our server resources being wasted if a malcious client sends a very large request body, and it's something we can easily do by using the http.MaxBytesReader() function. There's no check for a Content-Type: application/json header in the request. Of course, this header may not always be present, and mistakes and malicious clients mean that it isn't a guarantee of the actual content type. But checking for an incorrect Content-Type header would allow us to 'fail fast' and send a helpful error message without spending unnecessary resources on parsing the body. The Decode() method is designed to decode streams of JSON objects. This means a request body like '{"Name": "Bob"}{"Name": "Carol": "Age": 54}' or '{"Name": "Dave"}{}' is considered valid and won't result in the client receiving an error message. But in each case, only the first JSON object in the request body will actually be parsed. There are two solutions here. We can either check the decoder's More() method after decoding to see if there any any other JSON objects in the request body. Or we could avoid using Decode() altogether and read the body into a byte slice and pass it to json.Unmarshal(), which would return an error if the body contains multiple JSON objects. The downside of using json.Unmarshal() is that there is no way to disallow extra unexpected fields in the JSON, so we can't address point 3 above. An Improved Handler Let's implement an alternative version of the personCreate handler which addresses all of these issues. You'll notice here that we're using the new errors.Is() and errors.As() functions, which have been introduced in Go 1.13, to help intercept the errors from Decode(). // File: main.go package main import ( "encoding/json" "errors" "fmt" "io" "log" "net/http" "strings" "github.com/golang/gddo/httputil/header" ) type Person struct { Name string Age int } func personCreate(w http.ResponseWriter, r *http.Request) { // If the Content-Type header is present, check that it has the value // application/json. Note that we are using the gddo/httputil/header // package to parse and extract the value here, so the check works // even if the client includes additional charset or boundary // information in the header. if r.Header.Get("Content-Type") != "" { value, _ := header.ParseValueAndParams(r.Header, "Content-Type") if value != "application/json" { msg := "Content-Type header is not application/json" http.Error(w, msg, http.StatusUnsupportedMediaType) return } } // Use http.MaxBytesReader to enforce a maximum read of 1MB from the // response body. A request body larger than that will now result in // Decode() returning a "http: request body too large" error. r.Body = http.MaxBytesReader(w, r.Body, 1048576) // Setup the decoder and call the DisallowUnknownFields() method on it. // This will cause Decode() to return a "json: unknown field ..." error // if it encounters any extra unexpected fields in the JSON. Strictly // speaking, it returns an error for "keys which do not match any // non-ignored, exported fields in the destination". dec := json.NewDecoder(r.Body) dec.DisallowUnknownFields() var p Person err := dec.Decode(&p) if err != nil { var syntaxError *json.SyntaxError var unmarshalTypeError *json.UnmarshalTypeError switch { // Catch any syntax errors in the JSON and send an error message // which interpolates the location of the problem to make it // easier for the client to fix. case errors.As(err, &syntaxError): msg := fmt.Sprintf("Request body contains badly-formed JSON (at position %d)", syntaxError.Offset) http.Error(w, msg, http.StatusBadRequest) // In some circumstances Decode() may also return an // io.ErrUnexpectedEOF error for syntax errors in the JSON. There // is an open issue regarding this at // https://github.com/golang/go/issues/25956. case errors.Is(err, io.ErrUnexpectedEOF): msg := fmt.Sprintf("Request body contains badly-formed JSON") http.Error(w, msg, http.StatusBadRequest) // Catch any type errors, like trying to assign a string in the // JSON request body to a int field in our Person struct. We can // interpolate the relevant field name and position into the error // message to make it easier for the client to fix. case errors.As(err, &unmarshalTypeError): msg := fmt.Sprintf("Request body contains an invalid value for the %q field (at position %d)", unmarshalTypeError.Field, unmarshalTypeError.Offset) http.Error(w, msg, http.StatusBadRequest) // Catch the error caused by extra unexpected fields in the request // body. We extract the field name from the error message and // interpolate it in our custom error message. There is an open // issue at https://github.com/golang/go/issues/29035 regarding // turning this into a sentinel error. case strings.HasPrefix(err.Error(), "json: unknown field "): fieldName := strings.TrimPrefix(err.Error(), "json: unknown field ") msg := fmt.Sprintf("Request body contains unknown field %s", fieldName) http.Error(w, msg, http.StatusBadRequest) // An io.EOF error is returned by Decode() if the request body is // empty. case errors.Is(err, io.EOF): msg := "Request body must not be empty" http.Error(w, msg, http.StatusBadRequest) // Catch the error caused by the request body being too large. Again // there is an open issue regarding turning this into a sentinel // error at https://github.com/golang/go/issues/30715. case err.Error() == "http: request body too large": msg := "Request body must not be larger than 1MB" http.Error(w, msg, http.StatusRequestEntityTooLarge) // Otherwise default to logging the error and sending a 500 Internal // Server Error response. default: log.Print(err.Error()) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) } return } // Check that the request body only contained a single JSON object. if dec.More() { msg := "Request body must only contain a single JSON object" http.Error(w, msg, http.StatusBadRequest) return } fmt.Fprintf(w, "Person: %+v", p) } func main() { mux := http.NewServeMux() mux.HandleFunc("/person/create", personCreate) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } The clear downside here is that this code is a lot more verbose, and IMO, a little bit ugly. Things aren't helped by the fact that there are quite a few open issues with json/encoding which are on hold pending a wider review of the package. But from a security and client perspective it's a lot better : ) The handler is now stricter about the content it will accept; we're reducing the amount of server resources used unnecessarily; and the client gets clear and consistent error messages that provide a decent amount of information without over-sharing. As a side note, you might have noticed that the json/encoding package contains some other error types (like json.UnmarshalFieldError) which aren't checked in the code above — but these have been deprecated and not used by Go 1.13. Making a Helper Function If you've got a few handlers that need to to process JSON request bodies, you probably don't want to repeat this code in all of them. A solution which I've found works well is to create a decodeJSONBody helper function, and have this return a custom malformedRequest error type which wraps the errors and relevant status codes. For example: // File: helpers.go package main import ( "encoding/json" "errors" "fmt" "io" "net/http" "strings" "github.com/golang/gddo/httputil/header" ) type malformedRequest struct { status int msg string } func (mr *malformedRequest) Error() string { return mr.msg } func decodeJSONBody(w http.ResponseWriter, r *http.Request, dst interface{}) error { if r.Header.Get("Content-Type") != "" { value, _ := header.ParseValueAndParams(r.Header, "Content-Type") if value != "application/json" { msg := "Content-Type header is not application/json" return &malformedRequest{status: http.StatusUnsupportedMediaType, msg: msg} } } r.Body = http.MaxBytesReader(w, r.Body, 1048576) dec := json.NewDecoder(r.Body) dec.DisallowUnknownFields() err := dec.Decode(&dst) if err != nil { var syntaxError *json.SyntaxError var unmarshalTypeError *json.UnmarshalTypeError switch { case errors.As(err, &syntaxError): msg := fmt.Sprintf("Request body contains badly-formed JSON (at position %d)", syntaxError.Offset) return &malformedRequest{status: http.StatusBadRequest, msg: msg} case errors.Is(err, io.ErrUnexpectedEOF): msg := fmt.Sprintf("Request body contains badly-formed JSON") return &malformedRequest{status: http.StatusBadRequest, msg: msg} case errors.As(err, &unmarshalTypeError): msg := fmt.Sprintf("Request body contains an invalid value for the %q field (at position %d)", unmarshalTypeError.Field, unmarshalTypeError.Offset) return &malformedRequest{status: http.StatusBadRequest, msg: msg} case strings.HasPrefix(err.Error(), "json: unknown field "): fieldName := strings.TrimPrefix(err.Error(), "json: unknown field ") msg := fmt.Sprintf("Request body contains unknown field %s", fieldName) return &malformedRequest{status: http.StatusBadRequest, msg: msg} case errors.Is(err, io.EOF): msg := "Request body must not be empty" return &malformedRequest{status: http.StatusBadRequest, msg: msg} case err.Error() == "http: request body too large": msg := "Request body must not be larger than 1MB" return &malformedRequest{status: http.StatusRequestEntityTooLarge, msg: msg} default: return err } } if dec.More() { msg := "Request body must only contain a single JSON object" return &malformedRequest{status: http.StatusBadRequest, msg: msg} } return nil } Once that's written, the code in your handlers can be kept really nice and compact: // File: main.go package main import ( "errors" "fmt" "log" "net/http" ) type Person struct { Name string Age int } func personCreate(w http.ResponseWriter, r *http.Request) { var p Person err := decodeJSONBody(w, r, &p) if err != nil { var mr *malformedRequest if errors.As(err, &mr) { http.Error(w, mr.msg, mr.status) } else { log.Print(err.Error()) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) } return } fmt.Fprintf(w, "Person: %+v", p) } func main() { mux := http.NewServeMux() mux.HandleFunc("/person/create", personCreate) log.Print("Starting server on :4000...") err := http.ListenAndServe(":4000", mux) log.Fatal(err) }

• Using PostgreSQL JSONB with Go
  May 14, 2019

  PostgreSQL provides two JSON-related data types that you can use — JSON and JSONB. The principal differences are: JSON stores an exact copy of the JSON input. JSONB stores a binary representation of the JSON input. This makes it slower to insert but faster to query. It may change the key order, and will remove whitespace and delete duplicate keys. JSONB also supports the ? (existence) and @> (containment) operators, whereas JSON doesn't. The PostgreSQL documentation recommends that you should generally use JSONB, unless you have a specific reason not too (like needing to preserve key order). Here's a cribsheet for the essential commands: -- Create a table with a JSONB column. CREATE TABLE items ( id SERIAL PRIMARY KEY, attrs JSONB ); -- You can insert any well-formed json input into the column. Note that only -- lowercase `true` and `false` spellings are accepted. INSERT INTO items (attrs) VALUES ('{ "name": "Pasta", "ingredients": ["Flour", "Eggs", "Salt", "Water"], "organic": true, "dimensions": { "weight": 500.00 } }'); -- Create an index on all key/value pairs in the JSONB column. CREATE INDEX idx_items_attrs ON items USING gin (attrs); -- Create an index on a specific key/value pair in the JSONB column. CREATE INDEX idx_items_attrs_organic ON items USING gin ((attrs->'organic')); -- The -> operator is used to get the value for a key. The returned value has -- the type JSONB. SELECT attrs->'dimensions' FROM items; SELECT attrs->'dimensions'->'weight' FROM items; -- Or you can use ->> to do the same thing, but this returns a TEXT value -- instead. SELECT attrs->>'dimensions' FROM items; -- You can use the returned values as normal, although you may need to type -- cast them first. SELECT * FROM items WHERE attrs->>'name' ILIKE 'p%'; SELECT * FROM items WHERE (attrs->'dimensions'->>'weight')::numeric 'dimensions' ? 'weight'; -- The ? operator can also be used to check for the existence of a specific -- text value in json arrays. SELECT * FROM items WHERE attrs->'ingredients' ? 'Salt'; -- Use @> to check if the JSONB column contains some specific json. This can -- be useful to filter for a specific key/value pair like so: SELECT * FROM items WHERE attrs @> '{"organic": true}'::jsonb; SELECT * FROM items WHERE attrs @> '{"dimensions": {"weight": 10}}'::jsonb; -- Note that @> looks for *containment*, not for an exact match. The -- followingquery will return records which have both "Flour" and "Water" -- as ingredients, rather than *only* "Flour" and "Water" as the ingredients. SELECT * FROM items WHERE attrs @> '{"ingredients": ["Flour", "Water"]}'::jsonb; For a full description of all available operators please see the official JSON Functions and Operators documentation. Using with Go If you're not familiar with the general patterns for working with SQL databases in Go, you might want to read my introduction to the database/sql package before continuing. Known JSON fields When the fields in a JSON/JSONB column are known in advance, you can map the contents of the JSON/JSONB column to and from a struct. To do this, you'll need make sure the struct implements: The driver.Valuer interface, such that it marshals the object into a JSON byte slice that can be understood by the database. The sql.Scanner interface, such that it unmarshals a JSON byte slice from the database into the struct fields. Here's a demonstration: package main import ( "database/sql" "database/sql/driver" "encoding/json" "errors" "log" _ "github.com/lib/pq" ) type Item struct { ID int Attrs Attrs } // The Attrs struct represents the data in the JSON/JSONB column. We can use // struct tags to control how each field is encoded. type Attrs struct { Name string `json:"name,omitempty"` Ingredients []string `json:"ingredients,omitempty"` Organic bool `json:"organic,omitempty"` Dimensions struct { Weight float64 `json:"weight,omitempty"` } `json:"dimensions,omitempty"` } // Make the Attrs struct implement the driver.Valuer interface. This method // simply returns the JSON-encoded representation of the struct. func (a Attrs) Value() (driver.Value, error) { return json.Marshal(a) } // Make the Attrs struct implement the sql.Scanner interface. This method // simply decodes a JSON-encoded value into the struct fields. func (a *Attrs) Scan(value interface{}) error { b, ok := value.([]byte) if !ok { return errors.New("type assertion to []byte failed") } return json.Unmarshal(b, &a) } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Initialize a new Attrs struct and add some values. attrs := new(Attrs) attrs.Name = "Pesto" attrs.Ingredients = []string{"Basil", "Garlic", "Parmesan", "Pine nuts", "Olive oil"} attrs.Organic = false attrs.Dimensions.Weight = 100.00 // The database driver will call the Value() method and and marshall the // attrs struct to JSON before the INSERT. _, err = db.Exec("INSERT INTO items (attrs) VALUES($1)", attrs) if err != nil { log.Fatal(err) } // Similarly, we can also fetch data from the database, and the driver // will call the Scan() method to unmarshal the data to an Attr struct. item := new(Item) err = db.QueryRow("SELECT id, attrs FROM items ORDER BY id DESC LIMIT 1").Scan(&item.ID, &item.Attrs) if err != nil { log.Fatal(err) } // You can then use the struct fields as normal... weightKg := item.Attrs.Dimensions.Weight / 1000 log.Printf("Item: %d, Name: %s, Weight: %.2fkg", item.ID, item.Attrs.Name, weightKg) } Unknown JSON fields The above pattern works great if you know in advance what keys and values your JSON/JSONB data will contain. And it has the major advantage of being type safe. For the times that you don't know this in advance (for example, the data contains user-generated keys and values) you can map the contents of the JSON/JSONB column to and from a map[string]interface{} instead. The big downside of this is that you will need to type assert any values that you retrieve from the database in order to use them. Here's the same example, but re-written to use a map[string]interface{}: package main import ( "database/sql" "database/sql/driver" "encoding/json" "errors" "log" _ "github.com/lib/pq" ) type Item struct { ID int Attrs Attrs } type Attrs map[string]interface{} func (a Attrs) Value() (driver.Value, error) { return json.Marshal(a) } func (a *Attrs) Scan(value interface{}) error { b, ok := value.([]byte) if !ok { return errors.New("type assertion to []byte failed") } return json.Unmarshal(b, &a) } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } item := new(Item) item.Attrs = Attrs{ "name": "Passata", "ingredients": []string{"Tomatoes", "Onion", "Olive oil", "Garlic"}, "organic": true, "dimensions": map[string]interface{}{ "weight": 250.00, }, } _, err = db.Exec("INSERT INTO items (attrs) VALUES($1)", item.Attrs) if err != nil { log.Fatal(err) } item = new(Item) err = db.QueryRow("SELECT id, attrs FROM items ORDER BY id DESC LIMIT 1").Scan(&item.ID, &item.Attrs) if err != nil { log.Fatal(err) } name, ok := item.Attrs["name"].(string) if !ok { log.Fatal("unexpected type for name") } dimensions, ok := item.Attrs["dimensions"].(map[string]interface{}) if !ok { log.Fatal("unexpected type for dimensions") } weight, ok := dimensions["weight"].(float64) if !ok { log.Fatal("unexpected type for weight") } weightKg := weight / 1000 log.Printf("%s: %.2fkg", name, weightKg) }

• An overview of Go's tooling
  Apr 15, 2019

  Occasionally I get asked “why do you like using Go?” And one of the things I often mention is the thoughtful tooling that exists alongside the language as part of the go command. There are some tools that I use everyday — like go fmt and go build — and others like go tool pprof that I only use to help solve a specific issue. But in all cases I appreciate the fact that they make managing and maintaining my projects easier. In this post I hope to provide a little background and context about the tools I find most useful, and importantly, explain how they can fit into the workflow of a typical project. I hope it'll give you a good start if you're new to Go. Or if you've been working with Go for a while, and that stuff's not applicable to you, hopefully you'll still discover a command or flag that you didn't know existed before : ) The information in this post is written for Go 1.12 and assumes that you're working on a project which has modules enabled. Installing Tooling Viewing Environment Information Development Running Code Fetching Dependencies Refactoring Code Viewing Go Documentation Testing Running Tests Profiling Test Coverage Stress Testing Testing all Dependencies Pre-Commit Checks Formatting Code Performing Static Analysis Linting Code Tidying and Verifying your Dependencies Build and Deployment Building an Executable Cross-Compilation Using Compiler and Linker Flags Diagnosing Problems and Making Optimizations Running and Comparing Benchmarks Profiling and Tracing Checking for Race Conditions Managing Dependencies Upgrading to a New Go Release Reporting Bugs Installing Tooling In this post I'll mainly be focusing on tools that are a part of the go command. But there are a few I'll be mentioning which aren't part of the standard Go 1.12 release. To install these while using Go 1.12 you'll first need to make sure that you're outside of a module-enabled directory (I usually just change into /tmp). Then you can use the GO111MODULE=on go get command to install the tool. For example: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/cmd/stress This will download the relevant package and dependencies, build the executable and add it to your GOBIN directory. If you haven't explicitly set a GOBIN directory, then the executable will be added to your GOPATH/bin folder. Either way, you should make sure that the appropriate directory is on your system path. Note: This process is a bit clunky and will hopefully improve in future versions of Go. Issue 30515 is tracking the discussion about this. Viewing Environment Information You can use the go env tool to display information about your current Go operating environment. This can be particularly useful if you're working on an unfamiliar machine. $ go env GOARCH="amd64" GOBIN="" GOCACHE="/home/alex/.cache/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/home/alex/go" GOPROXY="" GORACE="" GOROOT="/usr/local/go" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build245740092=/tmp/go-build -gno-record-gcc-switches" If there are specific values that you're interested in, you can pass them as arguments to go env. For example: $ go env GOPATH GOOS GOARCH /home/alex/go linux amd64 To show documentation for all go env variables and values you can run: $ go help environment Development Running Code During development the go run tool is a convenient way to try out your code. It's essentially a shortcut that compiles your code, creates an executable binary in your /tmp directory, and then runs this binary in one step. $ go run . # Run the package in the current directory $ go run ./cmd/foo # Run the package in the ./cmd/foo directory Note: As of Go 1.11 you can pass the path of a package to go run, like we have above. This means that you no longer have to use workarounds like go run *.go wildcard expansion to run multiple files. I like this improvement a lot! Fetching Dependencies Assuming that you've got modules enabled, when you use go run (or go test or go build for that matter) any external dependencies will automatically (and recursively) be downloaded to fulfill the import statements in your code. By default the latest tagged release of the dependency will be downloaded, or if no tagged releases are available, then the dependency at the latest commit. If you know in advance that you need a specific version of a dependency (instead of the one that Go would fetch by default) you can use go get with the relevant version number or commit hash. For example: $ go get github.com/foo/bar@v1.2.3 $ go get github.com/foo/bar@8e1b8d3 If the dependency being fetched has a go.mod file, then its dependencies won't be listed in your go.mod file. In contrast, if the dependency you're downloading doesn't have a go.mod file, then it's dependencies will be listed in your go.mod file with an // indirect comment next to them. So that means your go.mod file doesn't necessarily show all the dependencies for your project in one place. Instead, you can view them all using the go list tool like so: $ go list -m all Sometimes you might wonder why is that a dependency? You can answer this with the go mod why command, which will show you the shortest path from a package in your main module to a given dependency. For example: $ go mod why -m golang.org/x/sys # golang.org/x/sys github.com/alexedwards/argon2id golang.org/x/crypto/argon2 golang.org/x/sys/cpu Note: The go mod why command will return an answer for most, but not all, dependencies. Issue 27900 is tracking this. If you're interested in analyzing or visualizing the dependencies for your application, then you might want to also check out the go mod graph tool. There's a great tutorial and example code for generating visualizations here. Lastly, downloaded dependencies are stored in the module cache located at GOPATH/pkg/mod. If you ever need to clear the module cache you can use the go clean tool. But be aware: this will remove the downloaded dependencies for all projects on your machine. $ go clean -modcache Refactoring Code Chances are you're probably familiar with using the gofmt tool to automatically format your code. But it also supports rewrite rules that you can use to help refactor your code. I'll demonstrate. Let's say that you have the following code and you want to change the foo variable to Foo so it is exported. var foo int func bar() { foo = 1 fmt.Println("foo") } To do this you can use gofmt with the -r flag to implement a rewrite rule, the -d flag to display a diff of the changes, and the -w flag to make the changes in place, like so: $ gofmt -d -w -r 'foo -> Foo' . -var foo int +var Foo int func bar() { - foo = 1 + Foo = 1 fmt.Println("foo") } Notice how this is smarter than a find-and-replace? The foo variable has been changed, but the "foo" string in the fmt.Println() statement has been left unchanged. Another thing to note is that the gofmt command works recursively, so the above command will run on all *.go files in your current directory and subdirectories. If you want to use this functionality, I recommend running rewrite rules without the -w flag first, and checking the diff first to make sure that the changes to the code are what you expect. Let's take a look at a slightly more complicated example. Say you want to update your code to use the new Go 1.12 strings.ReplaceAll() function instead of strings.Replace(). To make this change you can run: $ gofmt -w -r 'strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)' . In rewrite rules, single lowercase characters act as wildcards matching arbitrary expressions, and those expressions will be substituted-in in the replacement. Viewing Go Documentation You can view documentation for the standard library packages via your terminal using the go doc tool. I often use this during development to quickly check something — like the name or signature of a specific function. I find it faster than navigating the web-based documentation and it's always available offline too. $ go doc strings # View simplified documentation for the strings package $ go doc -all strings # View full documentation for the strings package $ go doc strings.Replace # View documentation for the strings.Replace function $ go doc sql.DB # View documentation for the database/sql.DB type $ go doc sql.DB.Query # View documentation for the database/sql.DB.Query method You can also include the -src flag to display the relevant Go source code. For example: $ go doc -src strings.Replace # View the source code for the strings.Replace function Testing Running Tests You can use the go test tool to run tests in your project like so: $ go test . # Run all tests in the current directory $ go test ./... # Run all tests in the current directory and sub-directories $ go test ./foo/bar # Run all tests in the ./foo/bar directory Typically I run my tests with Go's race detector enabled, which can help pick up some of the data races that might occur in real-life usage. Like so: $ go test -race ./... It's important to note that enabling the race detector will increase the overall running time of your tests. So if you're running tests very frequently part of a TDD workflow, you might prefer to save using this for a pre-commit test run only. Since 1.10, Go caches test results at the package-level. If a package hasn't changed between test runs — and you're using the same, cachable, flags for go test — then the cached test result will be displayed with a "(cached)" next to it. This is hugely helpful in speeding up the test runtime for large codebases. If you want force your tests to run in full (and avoid the cache) you can use the -count=1 flag, or clear all cached test results by using the go clean tool. $ go test -count=1 ./... # Bypass the test cache when running tests $ go clean -testcache # Delete all cached test results Note: Cached test results are stored alongside cached build results in your GOCACHE directory. Check go env GOCACHE if you're not sure where this is on your machine. You can limit go test to running specific tests (and sub-tests) by using the -run flag. This accepts a regular expression, and only tests which have names that match the regular expression will be run. I like to combine this with the -v flag to enable verbose mode, so the names of running tests and sub-tests are displayed. It's a useful way to make sure that I haven't screwed up the regexp and that the tests I expect are actually being run! $ go test -v -run=^TestFooBar$ . # Run the test with the exact name TestFooBar $ go test -v -run=^TestFoo . # Run tests whose names start with TestFoo $ go test -v -run=^TestFooBar$/^Baz$ . # Run the Baz subtest of the TestFooBar test only A couple more flags that it's good to be aware of are -short (which you can use to skip long-running tests) and -failfast (which will stop running further tests after the first failure). Note that -failfast will prevent test results from being cached. $ go test -short ./... # Skip long running tests $ go test -failfast ./... # Don't run further tests after a failure. Profiling Test Coverage You can enable coverage analysis when running tests by using the -cover flag. This will display the percentage of code covered by the tests in the output for each package, similar to this: $ go test -cover ./... ok github.com/alexedwards/argon2id 0.467s coverage: 78.6% of statements You can also generate a coverage profile using the -coverprofile flag and view it in your web browser by using the go tool cover -html command like so: $ go test -coverprofile=/tmp/profile.out ./... $ go tool cover -html=/tmp/profile.out This will gives you a navigable listing of all the test files, with code covered by the tests displayed in green, and uncovered code in red. If you want you can go a step further and set the -covermode=count flag to make the coverage profile record the exact number of times that each statement is executed during the tests. $ go test -covermode=count -coverprofile=/tmp/profile.out ./... $ go tool cover -html=/tmp/profile.out When viewed in the browser, statements which are executed more frequently are shown in a more saturated shade of green, similar to this: Note: If you’re using the t.Parallel() command in any of your tests, then you should use the flag -covermode=atomic instead of -covermode=count instead to ensure an accurate count. Lastly, if you don't have a web browser available to view a coverage profile, you can see a breakdown of test coverage by function/method in your terminal with the command: $ go tool cover -func=/tmp/profile.out github.com/alexedwards/argon2id/argon2id.go:77: CreateHash 87.5% github.com/alexedwards/argon2id/argon2id.go:96: ComparePasswordAndHash 85.7% ... Stress Testing You can use the go test -count command to run a test multiple times in succession, which can be useful if you want to check for sporadic or intermittent failures. For example: $ go test -run=^TestFooBar$ -count=500 . In this example, the TestFooBar test will be repeated 500 times in a row. But it's important to note that the test will be repeated in serial — even if it contains a t.Parallel() instruction. So if your test is doing something relatively slow, like making a round trip to a database, hard disk or the internet, running a large number of tests can take quite a long time. In that case you might want to use the stress tool to repeat the same test multiple times in parallel instead. You can install it like so: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/cmd/stress To use the stress tool, you'll first need to compile a test binary for the specific package you want to test. You can do using the go test -c command. For example, to create a test binary for the package in your current directory: $ go test -c -o=/tmp/foo.test . In this example, the test binary will be outputted to /tmp/foo.test. You can then use the stress tool to execute a specific test in the test binary like so: $ stress -p=4 /tmp/foo.test -test.run=^TestFooBar$ 60 runs so far, 0 failures 120 runs so far, 0 failures ... Note: In the example above I've used the -p flag to restrict the number of parallel processes used by stress to 4. Without this flag, the tool will default to using a number of processes equal to runtime.NumCPU(). Testing all Dependencies Before you build an executable for release or deployment, or distribute your code publicly, you may want to run the go test all command: $ go test all This will run tests on all packages in your module and all dependencies — include testing test dependencies and the necessary standard library packages — and it can help validate that the exact versions of the dependencies being used are compatible with each other. This can take quite a long time to run, but the results cache well so any subsequent tests should be faster in the future. If you want, you could also use go test -short all to skip any long-running tests. Pre-Commit Checks Formatting Code Go provides two tools to automatically format your code according to the Go conventions: gofmt and go fmt. Using these helps keep your code consistent across your files and projects, and — if you use them before committing code — helps reduce noise when examining a diff between file versions. I like to use the gofmt tool with the following flags: $ gofmt -w -s -d foo.go # Format the foo.go file $ gofmt -w -s -d . # Recursively format all files in the current directory and sub-directories In these commands, the -w flag instructs the tool to rewrite files in place, the -s instructs the tool to apply simplifications to the code where possible, and the -d flag instructs the tool to output diffs of the changes (because I'm curious to see what is changed). If you want to only display the names of changed files, instead of diffs, you can swap this for the -l flag instead. Note: The gofmt command works recursively. If you pass it a directory like . or ./cmd/foo it'll format all .go files under the directory. The other formatting tool — go fmt — tool is a wrapper which essentially calls gofmt -l -w on a specified file or directory. You can use it like this: $ go fmt ./... Performing Static Analysis The go vet tool carries out static analysis of your code and warns you of things which might be wrong with your code but wouldn't be picked up by the compiler. Issues like unreachable code, unnecessary assignments and badly-formed build tags. You can use it like so: $ go vet foo.go # Vet the foo.go file $ go vet . # Vet all files in the current directory $ go vet ./... # Vet all files in the current directory and sub-directories $ go vet ./foo/bar # Vet all files in the ./foo/bar directory Behind the scenes, go vet runs a bunch of different analyzers which are listed here and you can disable specific ones on a case-by-case basis. For example to disable the composite analyzer you can use: $ go vet -composites=false ./... There are a couple of experimental analyzers in golang.org/x/tools which you might want to try: nilness (which checks for redundant or impossible nil comparisons) and shadow (which check for possible unintended shadowing of variables). If you want to use these, you'll need to install and run them separately. For example, to install nilness you would run: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/go/analysis/passes/nilness/cmd/nilness And you can then use it like so: $ go vet -vettool=$(which nilness) ./... Note: when the -vettool flag is used it will only run the specified analyzer — all the other go vet analyzers won't be run. As a side note, since Go 1.10 the go test tool automatically executes a small, high-confidence, subset of the go vet checks before running any tests. You can turn this behavior off when running tests like so: $ go test -vet=off ./... Linting Code You can use the golint tool to identify style mistakes in your code. Unlike go vet, this isn't concerned with correctness of the code, but helps you to align your code with the style conventions in Effective Go and the Go CodeReviewComments. It's not part of the standard library, so you'll need to install it like so: $ cd /tmp $ GO111MODULE=on go get golang.org/x/lint/golint You can then run it as follows: $ golint foo.go # Lint the foo.go file $ golint . # Lint all files in the current directory $ golint ./... # Lint all files in the current directory and sub-directories $ golint ./foo/bar # Lint all files in the ./foo/bar directory Tidying and Verifying your Dependencies Before you commit any changes to your code I recommend running the following two commands to tidy and verify your dependencies: $ go mod tidy $ go mod verify The go mod tidy command will prune any unused dependencies from your go.mod and go.sum files, and update the files to include dependencies for all possible build tags/OS/architecture combinations (note: go run, go test, go build etc are ‘lazy' and will only fetch packages needed for the current build tags/OS/architecture). Running this before each commit will make it easier to determine which of your code changes were responsible for adding or removing which dependencies when looking at the version control history. I also recommend using the go mod verify command to check that the dependencies on your computer haven't accidentally (or purposely) been changed since they were downloaded and that they match the cryptographic hashes in your go.sum file. Running this helps ensure that the dependencies being used are the exact ones that you expect, and any build for that commit will be reproducible at a later point. Build and Deployment Building an Executable To compile a main package and create an executable binary you can use the go build tool. Typically I use it in conjunction with the -o flag, which let's you explicitly set the output directory and name of the binary like so: $ go build -o=/tmp/foo . # Compile the package in the current directory $ go build -o=/tmp/foo ./cmd/foo # Compile the package in the ./cmd/foo directory In these examples, go build will compile the specified package (and any dependent packages), then invoke the linker to generate an executable binary, and output this to /tmp/foo. It's important to note that, as of Go 1.10, the go build tool caches build output in the build cache. This cached output will be reused again in future builds where appropriate, which can significantly speed up the overall build time. This new caching behavior means that the old maxim of “prefer go install to go build to improve caching” no longer applies. If you're not sure where your build cache is, you can check by running the go env GOCACHE command: $ go env GOCACHE /home/alex/.cache/go-build Using the build cache comes with one important caveat — it does not detect changes to C libraries imported with cgo. So if your code imports a C library via cgo and you've made changes to it since the last build, you'll need to use the -a flag which forces all packages to be rebuilt. Alternatively, you could use go clean to purge the cache: $ go build -a -o=/tmp/foo . # Force all packages to be rebuilt $ go clean -cache # Remove everything from the build cache Note: Running go clean -cache will delete cached test results too. If you're interested in what go build is doing behind the scenes, you might like to use the following commands: $ go list -deps . | sort -u # List all packages that are used to build the executable $ go build -a -x -o=/tmp/foo . # Rebuild everything and show the commands that are run Finally, if you run go build on a non-main package, it will be compiled in a temporary location and again, the result will be stored in the build cache. No executable is produced. Cross-Compilation This is one of my favorite features of Go. By default go build will output a binary suitable for use on your current operating system and architecture. But it also supports cross-compilation, so you can generate a binary suitable for use on a different machine. This is particularly useful if you're developing on one operating system and deploying on another. You can specify the operating system and architecture that you want to create the binary for by setting the GOOS and GOARCH environment variables respectively. For example: $ GOOS=linux GOARCH=amd64 go build -o=/tmp/linux_amd64/foo . $ GOOS=windows GOARCH=amd64 go build -o=/tmp/windows_amd64/foo.exe . To see a list of all supported OS/architecture combinations you can run go tool dist list: $ go tool dist list aix/ppc64 android/386 android/amd64 android/arm android/arm64 darwin/386 darwin/amd64 ... Hint: You can use Go's cross-compilation to create WebAssembly binaries. For a bit more in-depth information about cross compilation I recommend reading this excellent post. Using Compiler and Linker Flags When building your executable you can use the -gcflags flag to change the behavior of the compiler and see more information about what it's doing. You can see a complete list of available compiler flags by running: $ go tool compile -help One flag that you might find interesting is -m, which triggers the printing of information about optimization decisions made during compilation. You can use it like this: $ go build -gcflags="-m -m" -o=/tmp/foo . # Print information about optimization decisions In the above example I used the -m flag twice to indicate that I want to print decision information two-levels deep. You can get simpler output by using just one. Also, as of Go 1.10, compiler flags only apply to the specific packages passed to go build — which in the example above is the package in the current directory (represented by .). If you want to print optimization decisions for all packages including dependencies can use this command instead: $ go build -gcflags="all=-m" -o=/tmp/foo . As of Go 1.11, you should find it easier to debug optimized binaries than before. However, you can still use the flags -N to disable optimizations and -l to disable inlining if you need to. For example: $ go build -gcflags="all=-N -l" -o=/tmp/foo . # Disable optimizations and inlining You can see a list of available linker flags by running: $ go tool link -help Probably the most well-known of these is the -X flag, which allows you to "burn in" a (string) value to a specific variable in your application. This is commonly used to add a version number or commit hash. For example: $ go build -ldflags="-X main.version=1.2.3" -o=/tmp/foo . For more information about the -X flag and some sample code see this StackOverflow question and this post and this post. You may also be interested in using the -s and -w flags to strip debugging information from the binary. This typically shaves about 25% off the final size. For example: $ go build -ldflags="-s -w" -o=/tmp/foo . # Strip debug information from the binary Note: If binary size is something that you need to optimize for you might want to use upx to compress it. See this post for more information. Diagnosing Problems and Making Optimizations Running and Comparing Benchmarks A nice feature of Go is that it makes it easy to benchmark your code. If you're not familiar with the general process for writing benchmarks there are good guides here and here. To run benchmarks you'll need to use the go test tool, with the -bench flag set to a regular expression that matches the benchmarks you want to execute. For example: $ go test -bench=. ./... # Run all benchmarks and tests $ go test -run=^$ -bench=. ./... # Run all benchmarks (and no tests) $ go test -run=^$ -bench=^BenchmarkFoo$ ./... # Run only the BenchmarkFoo benchmark (and no tests) I almost always run benchmarks using the -benchmem flag, which forces memory allocation statistics to be included in the output. $ go test -bench=. -benchmem ./... By default, each benchmark test will be run for a minimum of 1 second, once only. You can change this with the -benchtime and -count flags: $ go test -bench=. -benchtime=5s ./... # Run each benchmark test for at least 5 seconds $ go test -bench=. -benchtime=500x /.... # Run each benchmark test for exactly 500 iterations $ go test -bench=. -count=3 ./... # Repeat each benchmark test 3 times over If the code that you're benchmarking uses concurrency, you can use the -cpu flag to see the performance impact of changing your GOMAXPROCS value (essentially, the number of OS threads that can execute your Go code simultaneously). For example, to run benchmarks with GOMAXPROCS set to 1, 4 and 8: $ go test -bench=. -cpu=1,4,8 ./... To compare changes between benchmarks you might want to use the benchcmp tool. This isn't part of the standard go command, so you'll need to install it like so: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/cmd/benchcmp You can then use it like this: $ go test -run=^$ -bench=. -benchmem ./... > /tmp/old.txt # make changes $ go test -run=^$ -bench=. -benchmem ./... > /tmp/new.txt $ benchcmp /tmp/old.txt /tmp/new.txt benchmark old ns/op new ns/op delta BenchmarkExample-8 21234 5510 -74.05% benchmark old allocs new allocs delta BenchmarkExample-8 17 11 -35.29% benchmark old bytes new bytes delta BenchmarkExample-8 8240 3808 -53.79% Profiling and Tracing Go makes it possible to create diagnostic profiles for CPU use, memory use, goroutine blocking and mutex contention. You can use these to dig a bit deeper and see exactly how your application is using (or waiting on) resources. There are three ways to generate profiles: If you have a web application you can import the net/http/pprof package. This will register some handlers with the http.DefaultServeMux which you can then use to generate and download profiles for your running application. This post provides a good explanation and some sample code. For other types of applications, you can profile your running application using the pprof.StartCPUProfile() and pprof.WriteHeapProfile() functions. See the runtime/pprof documentation for sample code. Or you can generate profiles while running benchmarks or tests by using the various -***profile flags like so: $ go test -run=^$ -bench=^BenchmarkFoo$ -cpuprofile=/tmp/cpuprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -memprofile=/tmp/memprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -blockprofile=/tmp/blockprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -mutexprofile=/tmp/mutexprofile.out . Note: Using the -***profile flags when running benchmarks or tests will result in a test binary being outputted to your current directory. If you want to output this to an alternative location you should use the -o flag like so: $ go test -run=^$ -bench=^BenchmarkFoo$ -o=/tmp/foo.test -cpuprofile=/tmp/cpuprofile.out . Whichever way you choose to create a profile, when profiling is enabled your Go program will stop about 100 times per second and take a snapshot at that moment in time. These samples are collected together to form a profile that you can analyze using the pprof tool. My favourite way to inspect a profile is to use the go tool pprof -http command to open it in a web browser. For example: $ go tool pprof -http=:5000 /tmp/cpuprofile.out This will default to displaying a graph showing the execution tree for the sampled aspects of your application, which makes it possible to quickly get a feel for any resource usage 'hotspots'. In the graph above, we can see that the hotspots in terms of CPU usage are two system calls originating from ioutil.ReadFile(). You can also navigate to other views of the profile including top usage by function and source code. If the amount of information is overwhelming, you might want to use the --nodefraction flag to ignore nodes that account for less than a certain percentage of samples. For example to ignore nodes that use appear in less than 10% of samples you can run pprof like so: $ go tool pprof --nodefraction=0.1 -http=:5000 /tmp/cpuprofile.out This makes the graph a lot less 'noisy' and if you zoom in on this screenshot, it's now much clearer to see and understand where the CPU usage hotspots are. Profiling and optimizing resource usage is big, nuanced, topic and I've barely scratched the surface here. If you're interested in knowing more then I encourage you to read the following blog posts: Profiling and optimizing Go web applications Debugging performance issues in Go programs Daily code optimization using benchmarks and profiling Profiling Go programs with pprof Another tool that you can use to help diagnose issues is the runtime execution tracer. This gives you a view of how Go is creating and scheduling goroutines to run, when the garbage collector is running, and information about blocking syscall/network/sync operations. Again, you can generate trace from your tests or benchmarks, or use net/http/pprof to create and download a trace for your web application. You can then use go tool trace to view the output in your web browser like so: $ go test -run=^$ -bench=^BenchmarkFoo$ -trace=/tmp/trace.out . $ go tool trace /tmp/trace.out Important: This is currently only viewable in Chrome/Chromium. For more information about Go's execution tracer and how to interpret the output please see Rhys Hiltner's dotGo 2016 talk and this excellent blog post. Checking for Race Conditions I talked earlier about enabling Go's race detector during tests by using go test -race. But you can also enable it for running programs when building a executable, like so: $ go build -race -o=/tmp/foo . It's critical to note that race-detector-enabled binaries will use more CPU and memory than normal, so you shouldn't use the -race flag when building binaries for production under normal circumstances. But you may want to deploy a race-detector-enabled binary on one server within a pool of many. Or use it to help track down a suspected race-condition by using a load-test tool to throw traffic concurrently at a race-detector-enabled binary. By default, if any races are detected while the binary is running a log will be written to stderr. You can change this by using the GORACE environment variable if necessary. For example, to run the binary located at /tmp/foo and output any race logs to /tmp/race. you can use: $ GORACE="log_path=/tmp/race" /tmp/foo Managing Dependencies You can use the go list tool to check whether a specific dependency has a newer version available like so: $ go list -m -u github.com/alecthomas/chroma github.com/alecthomas/chroma v0.6.2 [v0.6.3] This will output the dependency name and version that you're currently using, followed by the latest version in square brackets [], if a newer one exists. You can also use go list to check for updates to all dependencies (and sub-dependencies) like so: $ go list -m -u all You can upgrade (or downgrade) a dependency to the latest version, specific tagged-release or commit hash with the go get command like so: $ go get github.com/foo/bar@latest $ go get github.com/foo/bar@v1.2.3 $ go get github.com/foo/bar@7e0369f If the dependency you're updating has a go.mod file, then based on the information in this go.mod file, updates to any sub-dependencies will also be downloaded if necessary. If you use the go get -u flag, the contents of the go.mod file will be ignored and all sub-dependencies will be upgraded to their latest minor/patch version… even if the go.mod specifies a different version. After upgrading or downgrading any dependencies it's a good idea to tidy your modfiles. And you might also want to run the tests for all packages to help check for incompatibilities. Like so: $ go mod tidy $ go test all Occasionally you might want to use a local version of a dependency (for example, you need to use a local fork until a patch is merged upstream). To do this, you can use the go mod edit command to replace a dependency in your go.mod file with a local version. For example: $ go mod edit -replace=github.com/alexedwards/argon2id=/home/alex/code/argon2id This will add a replace rule to your go.mod file like so, and any future invocations of go run, go build etc will use the local version. File: go.mod module alexedwards.net/example go 1.12 require github.com/alexedwards/argon2id v0.0.0-20190109181859-24206601af6c replace github.com/alexedwards/argon2id => /home/alex/Projects/playground/argon2id Once it's no longer necessary, you can remove the replace rule with the command: $ go mod edit -dropreplace=github.com/alexedwards/argon2id You can use the same general technique to import packages that exist only on your own file system. This can be useful if you're working on multiple modules in development at the same time, one of which depends on the other. Note: If you don't want to use the go mod edit command, you can edit your go.mod file manually to make these changes. Either way will work. Upgrading to a New Go Release The go fix tool was originally released back in 2011 (when regular changes were still being made to Go's API) to help users automatically update their old code to be compatible with the latest version of Go. Since then, Go's compatibility promise means if you're upgrading from one Go 1.x version to a newer 1.x version everything should Just Work and using go fix should generally be unnecessary. However, there are a handful of very specific issues that it does deal with. You can see a summary of them by running go tool fix -help. If you decide that you want or need to run go fix after upgrading, you should you run the following command, then inspect a diff of the changes before you commit them. $ go fix ./... Reporting Bugs If you're confident that you've found an unreported issue with Go's standard library, tooling or documentation, you can use the go bug command to create a new Github issue. $ go bug This will open a browser window containing an issue pre-filled with your system information and reporting template.

• How to hash and verify passwords with Argon2 in Go
  Dec 10, 2018

  Thanks to Andreas Auernhammer, author of the golang.org/x/crypto/argon2 package, for checking over this post before publication. If you're planning to store user passwords it's good practice (essential really) to hash them using a computationally expensive key-derivation function (KDF) like Bcrypt, Scrypt or Argon2. Hashing and verifying passwords in Go with Bcrypt and Scrypt is already easy to do thanks to the golang.org/x/crypto/bcrypt package and Matt Silverlock's elithrar/simple-scrypt package. I recommend them both. If you want to use Argon2 — which is widely considered to be the best in class KDF for hashing passwords — then you've got a couple of choices. The tvdburgt/go-argon2 package provides Go bindings to the libargon2 C library, or you can implement a pure Go solution by wrapping the golang.org/x/crypto/argon2 package with helpers for hashing and verifying passwords. In the rest of this post I'm going to explain exactly how to use this pure Go approach. A Brief Introduction to Argon2 But first, a little bit of background. It's important to explain that the Argon2 algorithm has 3 variants which work slightly differently: Argon2d, Argon2i and Argon2id. In general, for password hashing you should use the Argon2id variant. This is essentially a hybrid of the Argon2d and Argon2i algorithms and uses a combination of data-independent memory access (for resistance against side-channel timing attacks) and data-depending memory access (for resistance against GPU cracking attacks). The Argon2 algorithm accepts a number of configurable parameters: Memory — The amount of memory used by the algorithm (in kibibytes). Iterations — The number of iterations (or passes) over the memory. Parallelism — The number of threads (or lanes) used by the algorithm. Salt length — Length of the random salt. 16 bytes is recommended for password hashing. Key length — Length of the generated key (or password hash). 16 bytes or more is recommended. The memory and iterations parameters control the computational cost of hashing the password. The higher these figures are, the greater the cost of generating the hash. It also follows that the greater the cost will be for any attacker trying to guess the password. But there's a balance that you need to strike. As you increase the cost, the time taken to generate the hash also increases. If you're generating the hash in response to a user action (like signing up or logging in to a website) then you probably want to keep the runtime to less than 500ms to avoid a negative user experience. If the Argon2 algorithm is running on a machine with multiple cores, then one way to decrease the runtime without reducing the cost is to increase the parallelism parameter. This controls the number of threads that the work is spread across. There's an important thing to note here though: changing the value of the parallelism parameter changes the output of the algorithm. So — for example — running Argon2 with a parallelism parameter of 2 will result in a different password hash to running it with a parallelism parameter of 4. Choosing Parameters Picking the right parameters for Argon2 depends heavily on the machine that the algorithm is running on, and you'll probably need to do some experimentation in order to set them appropriately. The recommended process for choosing the parameters can be paraphrased as follows: Set the parallelism and memory parameters to the largest amount you are willing to afford, bearing in mind that you probably don't want to max these out completely unless your machine is dedicated to password hashing. Increase the number of iterations until you reach your maximum runtime limit (for example, 500ms). If you're already exceeding the your maximum runtime limit with the number of iterations = 1, then you should reduce the memory parameter. Hashing Passwords Now that those explanations are out of the way let's jump into writing the code to hash a password with Argon2. First, you'll need to go get the golang.org/x/crypto/argon2 package which implements the Argon2 algorithm: $ go get golang.org/x/crypto/argon2 And you can use it to hash a specific password like so: File: main.go package main import ( "crypto/rand" "fmt" "log" "golang.org/x/crypto/argon2" ) type params struct { memory uint32 iterations uint32 parallelism uint8 saltLength uint32 keyLength uint32 } func main() { // Establish the parameters to use for Argon2. p := &params{ memory: 64 * 1024, iterations: 3, parallelism: 2, saltLength: 16, keyLength: 32, } // Pass the plaintext password and parameters to our generateFromPassword // helper function. hash, err := generateFromPassword("password123", p) if err != nil { log.Fatal(err) } fmt.Println(hash) } func generateFromPassword(password string, p *params) (hash []byte, err error) { // Generate a cryptographically secure random salt. salt, err := generateRandomBytes(p.saltLength) if err != nil { return nil, err } // Pass the plaintext password, salt and parameters to the argon2.IDKey // function. This will generate a hash of the password using the Argon2id // variant. hash = argon2.IDKey([]byte(password), salt, p.iterations, p.memory, p.parallelism, p.keyLength) return hash, nil } func generateRandomBytes(n uint32) ([]byte, error) { b := make([]byte, n) _, err := rand.Read(b) if err != nil { return nil, err } return b, nil } A quick note on terminology and naming. Formally, Argon2 is a key-derivation function and it produces a key derived from the provided password and salt. This derived key is our 'hashed password'. The other important thing to point out here is the generateRandomBytes() function. In this we're using Go's crypto/rand package to generate a cryptographically secure random salt, rather than using a fixed salt or a pseudo-random salt. If you run the program at this point it should print a slice containing the bytes of the hashed password, similar to this: $ go run main.go [9 18 35 54 101 221 120 189 57 241 229 248 140 1 102 58 93 211 115 49 131 162 24 50 167 142 227 198 85 186 200 248] Each time you run the program you'll see that it results in a completely different output for the same password, thanks to the addition of our random salt. Storing Passwords So, creating a hashed password with some specific parameters is straightforward enough. But in most cases you'll want to store the salt and specific parameters that you used alongside the hashed password, so that it can be reproducibly verified at a later point. The standard way to do this is to create an encoded representation of the hashed password which looks like this: $argon2id$v=19$m=65536,t=3,p=2$c29tZXNhbHQ$RdescudvJCsgt3ub+b+dWRWJTmaaJObG Let's break down what this represents: $argon2id — the variant of Argon2 being used. $v=19 — the version of Argon2 being used. $m=65536,t=3,p=2 — the memory (m), iterations (t) and parallelism (p) parameters being used. $c29tZXNhbHQ — the base64-encoded salt, using standard base64-encoding and no padding. $c29tZXNhbHQ$RdescudvJCsgt3ub+b+dWRWJTmaaJObG — the base64-encoded hashed password (derived key), using standard base64-encoding and no padding. Let's update the generateHash() function so that it returns a string in this format: File: main.go package main import ( "crypto/rand" "encoding/base64" "fmt" "log" "golang.org/x/crypto/argon2" ) ... func generateFromPassword(password string, p *params) (encodedHash string, err error) { salt, err := generateRandomBytes(p.saltLength) if err != nil { return "", err } hash := argon2.IDKey([]byte(password), salt, p.iterations, p.memory, p.parallelism, p.keyLength) // Base64 encode the salt and hashed password. b64Salt := base64.RawStdEncoding.EncodeToString(salt) b64Hash := base64.RawStdEncoding.EncodeToString(hash) // Return a string using the standard encoded hash representation. encodedHash = fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s", argon2.Version, p.memory, p.iterations, p.parallelism, b64Salt, b64Hash) return encodedHash, nil } ... And if you run the code again now, the output should look similar to this: $ go run run.go $argon2id$v=19$m=65536,t=3,p=2$Woo1mErn1s7AHf96ewQ8Uw$D4TzIwGO4XD2buk96qAP+Ed2baMo/KbTRMqXX00wtsU Verifying Passwords The final aspect to cover is how to verify passwords. In most cases, you'll take the encoded password hash that we've just produced and store it in a database of some kind. Then at a later point, you'll want to check whether a plaintext password provided by a user matches the one represented by the encoded password hash. In essence, the steps to do this check are: Extract the salt and parameters from the encoded password hash stored in the database. Derive the hash of the plaintext password using the exact same Argon2 variant, version, salt and parameters. Check whether this new hash is the same as the original one. You can implement this like so: File: main.go package main import ( "crypto/rand" "crypto/subtle" "encoding/base64" "errors" "fmt" "log" "strings" "golang.org/x/crypto/argon2" ) var ( ErrInvalidHash = errors.New("the encoded hash is not in the correct format") ErrIncompatibleVersion = errors.New("incompatible version of argon2") ) type params struct { memory uint32 iterations uint32 parallelism uint8 saltLength uint32 keyLength uint32 } func main() { p := &params{ memory: 64 * 1024, iterations: 3, parallelism: 2, saltLength: 16, keyLength: 32, } encodedHash, err := generateFromPassword("password123", p) if err != nil { log.Fatal(err) } match, err := comparePasswordAndHash("password123", encodedHash) if err != nil { log.Fatal(err) } fmt.Printf("Match: %v\n", match) } ... func comparePasswordAndHash(password, encodedHash string) (match bool, err error) { // Extract the parameters, salt and derived key from the encoded password // hash. p, salt, hash, err := decodeHash(encodedHash) if err != nil { return false, err } // Derive the key from the other password using the same parameters. otherHash := argon2.IDKey([]byte(password), salt, p.iterations, p.memory, p.parallelism, p.keyLength) // Check that the contents of the hashed passwords are identical. Note // that we are using the subtle.ConstantTimeCompare() function for this // to help prevent timing attacks. if subtle.ConstantTimeCompare(hash, otherHash) == 1 { return true, nil } return false, nil } func decodeHash(encodedHash string) (p *params, salt, hash []byte, err error) { vals := strings.Split(encodedHash, "$") if len(vals) != 6 { return nil, nil, nil, ErrInvalidHash } var version int _, err = fmt.Sscanf(vals[2], "v=%d", &version) if err != nil { return nil, nil, nil, err } if version != argon2.Version { return nil, nil, nil, ErrIncompatibleVersion } p = &params{} _, err = fmt.Sscanf(vals[3], "m=%d,t=%d,p=%d", &p.memory, &p.iterations, &p.parallelism) if err != nil { return nil, nil, nil, err } salt, err = base64.RawStdEncoding.DecodeString(vals[4]) if err != nil { return nil, nil, nil, err } p.saltLength = uint32(len(salt)) hash, err = base64.RawStdEncoding.DecodeString(vals[5]) if err != nil { return nil, nil, nil, err } p.keyLength = uint32(len(hash)) return p, salt, hash, nil } If you run this code now, you should get a positive match when comparing the plaintext and hashed password and see output like this: $ go run main.go Match: true If you change the plaintext password used in one of the function calls, like so: File: main.go package main ... func main() { p := &params{ memory: 64 * 1024, iterations: 3, parallelism: 2, saltLength: 16, keyLength: 32, } encodedHash, err := generateFromPassword("password123", p) if err != nil { log.Fatal(err) } // Use a different password... match, err := comparePasswordAndHash("pa$$word", encodedHash) if err != nil { log.Fatal(err) } fmt.Printf("Match: %v\n", match) } ... Then running the code should result in a negative match: $ go run main.go Match: false The complete sample code for this post is available in this gist.

Dave Cheney

• A few bytes here, a few there, pretty soon you’re talking real memory
  Jan 05, 2021

  Today’s post comes from a recent Go pop quiz. Consider this benchmark fragment. A convenience wrapper around sort.Sort(sort.StringSlice(s)), sort.Strings sorts the input in place, so it isn’t expected to allocate (or at least that’s what 43% of the tweeps who responded thought). However it turns out that, at least in recent versions of Go, each […]

• The story of the one line fix
  Dec 15, 2020

  Picture yourself, an engineer working at the hottest distributed microservices de jour, assigned to fix a bug. You jump into an unfamiliar codebase and quickly locate the line where the problem occurred. The fix is simple, just return early or substitute a default value in the case that one cannot be determined from your input. […]

• How to dump the GOSSAFUNC graph for a method
  Jun 19, 2020

  The Go compiler’s SSA backend contains a facility to produce HTML debugging output of the compilation phases. This post covers how to print the SSA output for function and methods. Let’s start with a sample program which contains a function, a value method, and a pointer method: Control of the SSA debugging output is via […]

• Diamond interface composition in Go 1.14
  May 24, 2020

  Per the overlapping interfaces proposal, Go 1.14 now permits embedding of interfaces with overlapping method sets. This is a brief post explain what this change means: Let’s start with the definition of the three key interfaces from the io package; io.Reader, io.Writer, and io.Closer: Just as embedding a type inside a struct allows the embedded type’s […]

• Fatih’s question
  May 16, 2020

  A few days ago Fatih posted this question on twitter. I’m going to attempt to give my answer, however to do that I need to apply some simplifications as my previous attempts to answer it involved a lot of phrases like a pointer to a pointer, and other unhelpful waffling. Hopefully my simplified answer can […]

• Ensmallening Go binaries by prohibiting comparisons
  May 09, 2020

  Conventional wisdom dictates that the larger the number of types declared in a Go program, the larger the resulting binary. Intuitively this makes sense, after all, what’s the point in defining a bunch of types if you’re not going to write code that operates on them. However, part of the job of a linker is […]

• Mid-stack inlining in Go
  May 02, 2020

  In the previous post I discussed how leaf inlining allows the Go compiler to reduce the overhead of function calls and extend optimisation opportunities across function boundaries. In this post I’ll discuss the limits of inlining and leaf vs mid-stack inlining. The limits of inlining Inlining a function into its caller removes the call’s overhead […]

• Inlining optimisations in Go
  Apr 25, 2020

  This is a post about how the Go compiler implements inlining and how this optimisation affects your Go code. n.b. This article focuses on gc, the de facto Go compiler from golang.org. The concepts discussed apply broadly to other Go compilers like gccgo and tinygo but may differ in implementation and efficacy. What is inlining? […]

• go test -v streaming output
  Mar 10, 2020

  The testing package is one of my favourite packages in the Go standard library, not just because of its low noise approach to unit testing, but, over the lifetime of Go, it has received a steady stream of quality of life improvements driven by real world usage. The most recent example of this is, in […]

• Are large slices more expensive than smaller ones?
  Mar 01, 2020

  Programmers have a tendency to be superstitious. Particularly, when a programmer hears that copies are expensive, they start to see them everywhere, especially when they learn that, in Go, every assignment is a copy. Consider this code; x is three orders of magnitude larger than y, is the assignment of x to a more expensive […]

• The Zen of Go
  Feb 23, 2020

  This article was derived from my GopherCon Israel 2020 presentation. It’s also quite long. If you’d prefer a shorter version, head over to the-zen-of-go.netlify.com. A recording of the presentation is available on YouTube. How should I write good code? Something that I’ve been thinking about a lot recently, when reflecting on the body of my […]

• Dynamically scoped variables in Go
  Dec 08, 2019

  This is a thought experiment in API design. It starts with the classic Go unit testing idiom: func TestOpenFile(t *testing.T) { f, err := os.Open("notfound") if err != nil { t.Fatal(err) } // ... } What’s the problem with this code? The assertion. if err != nil { ... } is repetitive and in the […]

• Internets of Interest #15: The Queen of the Skies
  Dec 05, 2019

  If, like me, you’re a commercial aviation otaku, this walkthrough of an enthusiast built 747 cockpit simulator should be highly relevant to your interests.

• Complementary engineering indicators
  Dec 04, 2019

  Last year I had the opportunity to watch Cat Swetel’s presentation The Development Metrics You Should Use (but Don’t). The information that could be gleaned from just tracking the start and finish date of work items was eye opening. If you’re using an issue tracker this information is probably already (perhaps with some light data […]

• Internets of interest #14: UNIX v0
  Nov 17, 2019

  Read more over at the Living Computer Museum’s restoration page.

• Internets of interest: Warner Losh on the first ten years of UNIX
  Nov 16, 2019

  UNIX turns 50 this year which means 7th edition Research UNIX is that 40.

• Use internal packages to reduce your public API surface
  Oct 06, 2019

  In the beginning, before the go tool, before Go 1.0, the Go distribution stored the standard library in a subdirectory called pkg/ and the commands which built upon it in cmd/. This wasn’t so much a deliberate taxonomy but a by product of the original make based build system. In September 2014, the Go distribution […]