Reading

A reading list from the RSS feeds I follow (happy to get good recommendations)

See my feeds.yaml.

Bob Nystrom

• Setting Up an SDL3 Mac App in XCode 16
  Jul 13, 2025

  This is mainly a note for my future self, but making it a blog post in case it helps anyone else. This weekend, I decided to try making a little SDL3 app on my new M4 MacBook Pro. It took me a few hours to figure out how to get XCode to play nice with the SDL3 framework given MacOS’s paranoia around unsigned code and malware. Here’s the summary for anyone whose Google searching takes them here. The instructions are fairly similar to the SDL3 README for MacOS: Start up XCode. Choose “File > New > Project…”. Select “macOS”, then under “Application” choose “App”. This step is important. Choosing “Command Line Tool” will not work. Click “Next”. Give your app a name. For “Interface”, choose “XIB”. For “Language”, choose “Objective-C”. Click “Next”. Choose a place to save your project. Now you have a vanilla “Hello World” Mac app. The next step is integrating SDL: Download a release of SDL. As I’m writing this, the latest is 3.2.16. Download the corresponding .dmg file. Open it in Finder. Open SDL3.xcframework/macos-arm64_x86_64 and copy SDL3.framework from there to somewhere in your XCode project’s source tree. (I made a lib folder and put it there in mine.) In XCode, in the file explorer on the left, click the top project icon to open the settings for it. In the main panel on the right, click the icon for your app target under “Targets” to open the target settings. Under “Frameworks, Libraries, and Embedded Content”, click the “+” to add a new framework. In the popup that appears, click “Add Other… > Add Files…”. Navigate to wherever you copied “SDL3.framework” and choose it. Then let’s use SDL and verify everything is working. In main.m, comment out the entire main() function. Make a new main.c file and copy the contents of hello.c from the SDL docs into it. Compile and run. If that all works, you should see a delightful pixelated “Hello World!” fill your screen. Press any key to quit. Let’s make sure the full iteration loop works. (Read below to see why I feel the need to verify this.): In main.c inside SDL_AppIterate(), change the message string. Hit Command-R to run. You should see your updated message on screen. There you go, you now have a working shell of a Mac app with using a local install of SDL3. You can probably delete the main.m file entirely and maybe some of the other generated files that XCode put in the new project, but I haven’t figured that out yet. Now here’s all the wrong stuff I tried first before I got that working… Attempt 1: Download a release and use the xcframework#attempt-1-download-a-release-and-use-the-xcframework In XCode, I created a new “CLI Command” C app. That got me a “Hello, world!” up and running without any problems. Then I copied the contents of this SDL hello example into my source file. As expected, that failed to compile because it couldn’t find SDL3/SDL.h. So next is getting SDL linked in. The first thing I did was download the 3.2.16 stable release installer. I downloaded the DMG file and opened that. Then following the instructions, I dragged the entire SDL3.xcframework directory into the XCode project. That correctly added a reference to the Framework and got XCode to a point where it could resolve the include and compile the program. Great. But when I ran it, it failed at startup. The OS tried to dynamically load the SDL3 framework, but then balked because it thinks the framework isn’t correctly signed and safe to use. The OS gave me a popup error with “SDL3.framework Not Opened” because “macOS cannot verify that this app is free from malware”. Crap. Attempt 2: Embed the framework#attempt-2-embed-the-framework Poking around online, some people say this error can be fixed by embedding the SDL framework in my app and having XCode sign it when it embeds it. In XCode, in the target settings under “Build Phases”, there is a panel for “Embed Frameworks”. I dragged SDL3.xcframework under there and checked “Code Sign On Copy”. Alas, when I tried to compile, now I got a build error. It complained that two tasks in the build were trying to produce the same file. I’ve lost the exact error message but it looked like one of the tasks was a simple file copy and the other was a “ProcessXCFramework” invocation. I’m guessing the latter is the task that signs and embeds the framework. I tried basically every combination of build settings to try to fix this: removing the framework from “Link Binary with Libraries”, checking “Copy only when installing”, and a bunch of other things I don’t remember. Nothing worked. Either I removed so much that the compiler couldn’t find the framework at all, or it got the duplicate build file error. Attempt 3: Don’t use the entire “SDL3.xcframework”#attempt-3-dont-use-the-entire-sdl3xcframework I started watching this video to see if it could help. He doesn’t use the entire SDL3.xcframework bundle. Instead, he opens that, goes into macos-arm64_x86_64 and copies the SDL3.framework framework out of there. I removed SDL3.xcframework from my XCode project and dragged SDL3.framework into it instead. That gave me something that compiled, but when I tried to run, it failed at load time with: dyld[15605]: Library not loaded: @rpath/SDL3.framework/Versions/A/SDL3 Referenced from: /Users/... Reason: tried: '/Users/.../Build/Products/Debug/SDL3.framework/ Versions/A/SDL3' (no such file) In the target settings, I again added the framework (but this time SDL3.framework not SDL3.xcframework to “Embed frameworks”). For reasons that are entirely unclear to me, that worked. I hit Command-R and… Hallelujah. All done, right? Not so fast. I made a tiny tweak to the code and tried to run it again. The compile immediately failed with: 'SDL3/SDL.h' file not found What? I literally just compiled and ran it. Just to make sure I wasn’t crazy, I cleaned the build directory and compiled again. That worked! “Hello World!” popped up on screen again. Build again… same error. Clean build and then build. Works! So I can build once but then something in the build directly causes any subsequent compile to fail. The full error is: Did not find header 'SDL.h' in framework 'SDL3' (loaded from '/Users/.../Index.noindex/Build/Products/Debug') So what I think is happening is that in the first clean build, it locates the framework inside my app’s main directory. That framework does have the headers so the compiler can find what it needs. Then after that first build, the compiler starts using the framework that it copied into the build directory. But when I look in there, Debug is completely empty. Attempt 4: Project-relative DerivedData#attempt-4-project-relative-deriveddata I hate when tools stuff built data in random system directories unrelated to the project I’m working on because those files end up sitting around forever. So the next thing I did was go under “Project Settings” and changed the “Derived Data” to be “Project-relative Location”. I cleaned and built successfully. But, again, if I made a trivial code change and tried to build again, it failed. This time, though, there was actually something in DerivedData/my_app/Build/Products/Debug/. I can see an SDL3.framework directory under there. It contains Versions/A/SDL3 (and some other stuff). So the library is there as expected. But there is no Headers directory under Versions/A/, nor is there a Headers symlink in SDL3.framework pointing to it. In theory, that’s fine. In the compiled app, all that’s needed is the library itself. But for reasons I don’t understand, XCode is trying to use this semi-copied version of the framework for subsequent compiles instead of the original one in my project’s source tree. Attempt 5: Explicit header search path#attempt-5-explicit-header-search-path OK, if XCode can’t find the headers because it’s looking in the wrong place, maybe I can just tell it where to look. Under “Search Paths”, I added lib/SDL3.framework/Headers to “Header Search Paths”. (I put the framework under lib in my project tree, hence the lib/ part.) Didn’t help. Tried making it recursive. Nope. Maybe just lib/SDL3.framework? No. I’m guessing the problem here is that the include paths look like SDL3/SDL.h, not just SDL.h, so I need to get XCode to understand that the headers are inside a framework to resolve the SDL3/ part. Attempt 6: Framework search path#attempt-6-framework-search-path OK, so maybe it’s not a header search path. Instead, I tried adding lib/ to “Framework Search Paths”. No help. Making it recursive didn’t help. Neither did lib/SDL3.framework. I do a bunch of Googling and eventually stumble onto this old StackOverflow post. He specifically mentions only getting the error after the first build, so this is my exact issue. And the author apparently found a solution! …which they described only as “It works know after certain changes!”. So helpful. I eventually found this Apple Developer forum thread, which also sounds like my problem. Attempt 7: An App instead of a CLI Command#attempt-7-an-app-instead-of-a-cli-command The forum thread suggests that the issue may have something to do with creating a “CLI Command” instead of an “App”. I’ve always done the former in XCode because that seemed like the simplest way to get a vanilla C/C++ app up and running. In older versions of XCode/macOS, it worked fine. But maybe it doesn’t play nice with frameworks in later versions? I trash the entire XCode project and create a new one. This time I choose “App” instead of “CLI Command”. I set the language to “Objective-C” (because that’s the “closest” to C, I guess—I’m winging it here). I set “Interface” to “XIB” based on this README. I run it and get an empty window, the “Hello World” of GUI apps. OK. I copy SDL3.framework out of SDL3.xcframework/macos-arm64_x86-x64 into lib/ in my new app’s source tree. Then in XCode, in the “General” tab of the main target, under “Frameworks, Libraries, and Embedded Content”, I click the “+”. From there I choose “Add Other…”, “Add Files…”, and add lib/SDL3.framework. I admit to feeling a moment of anticipation. Will this actually work? I make a new .c file and paste in the hello.c SDL3 example code. I comment out main() in the main.m that XCode generated. Compile and… it runs! I’ve got an SDL3 “Hello World!” on screen again. Now the real test… can I make a change and build again? Yes! It works! Finally! As far as I can tell, I am up and running now. I didn’t have to add SDL3 to my system framework path. (I didn’t want to do that because ultimately, I want an app that users can just run without having to install SDL themselves.) So it looks like the failure mode was creating a “CLI Command” instead of an “App”. How on Earth would I be expected to know that?

• Consider Knitting
  May 30, 2025

  Let’s say that, like me, you are a person who stares at a computer and writes code for a living. As a straight male who grew up in a time where knitting was very strongly female coded, it for the most part never occurred to me that knitting was a thing I could do and might enjoy. Regardless of your demographic categories and background, it’s possible that you have also not really considered knitting. In retrospect, this is pretty dumb. I did learn to cross-stitch when I was a kid and spent many hours doing that. I was mostly raised by my mom, so I learned a lot of crafty stuff. I also tried to learn how to crochet from my Grandmother a few times. But some combination of her being a mean old lady and me being left-handed made that not really work out. I made a friend in my twenties who was and is an avid knitter. I certainly admired her skills, but didn’t think to ask her to teach me. This article exists to get you to do so. Specifically, I’ll try to convince you, one software person to another, why it might be a good fit for your life and brain. This is a pitch for knitting, but—for better or worse—an extremely nerdily argued one. Frog and Toad, from this delightful pattern. I made these as Christmas presents for my daughters. Before I start, note that when I say “knitting”, you can read that as any of the various fiber arts, including crochet, weaving, macramé, cross-stitch, etc. I talk about knitting here because that’s the one closest to my heart and I strive to speak from the heart. You can make stuff out of string however you want. We are all fiber friends. The sense of touch#the-sense-of-touch I love the aesthetics of programming. Sitting in a cool quiet room, techno thumping in my headphones, coffee mug next to me, while a neatly arranged field of glowing monospace glyphs stream across my screen. But there’s one sense unmentioned in that sentence: touch. The sense we devote more neurons to than any other is curiously underutilized while pumping out code. It’s no surprise that some programmers fetishize keyboards. It’s just about the only part of programming that has any physical sensation at all. I got into knitting a few years after the pandemic. While I have a variety of hobbies, most still involve staring at a screen and maybe pushing some buttons and turning knobs. When I wasn’t doing those, I was staring at a screen for work, or staring at a screen for not-work. I don’t know if I have a good way to explain how much my body craved tactile experience by the end of that. It’s like my fingers ached. A deep hunger, but not for taste. I’d wander around the house, driving my wife insane, unable to sit down and get comfortable. My body was just screaming at me to do something. My youngest daughter had just picked up knitting, and she taught me. At first it was frustrating and annoying. But once I had the basics down, it was like a a deep sigh felt in my hands. Knitting is so touch-centered. Skilled knitters can knit without looking at their hands at all—touch alone is sufficient. Starting to work on a slanting stripes scarf for myself. I wish you could reach into this photo and feel how unbelievably soft and squishy this scarf is. Wearing it feels like giving myself a warm gentle hug. If I ever become a billionaire, I’m going to build an indoor swimming pool and fill it with Malabrigo Rios yarn. There are so many different kinds of yarn to work with and they all feel different. Cotton is tough and firm, like twine. Wool is soft and springy, forgiving as you pull stitches open to work them. Superwash wool is smooth and glides off the needles. Non-superwash wool has this very slight stickiness to it that makes the resulting fabric feel solid and whole. Thin fingering weight yarn wraps tightly around your finger like a reminder knot. Working it is like performing delicate surgery. Squishy chunky wool spreads your fingers wide and works so quickly it’s like fabric is spooling out of your hands on its own accord. The wool for this hat is a mixture of alpaca and sheep's wool. It's soft, fuzzy, and just the tiniest bit scratchy. Even needles each have their own personality. Stiff grippy bamboo with its dull clack. Less worrisome to use because stitches don’t slide off as easily, but harder work to push against the friction. Polished stainless steel where the stitches fly off the needles—good when you are done with those stitches but not so much otherwise. An inauspicious start for our friend Toad. I knitted Frog and Toad using lacquered wood size 2 double-pointed needles. It’s hard to get a sense of scale from this photo, but the needles are only six inches long. Like very fancy cocktail toothpicks. Once you’ve worked a few thousand stitches into your muscle memory, you can watch your fingers form stitches almost of their own accord, hypnotically. Right needle opens the stitch and slides in. Left finger wraps the yarn around the needle. Right finger grabs yarn. Right hand pulls the needle back out, a new stitch formed and transferred to the other needle. Over and over, like a meditation in the body. Knitting feels good. It is an intimate, constant reminder that we are a tool-using species with thousands of years of evolution giving us incredible dexterity and the emotional wiring to make us want to use it. An open world game with optimized skill curve#an-open-world-game-with-optimized-skill-curve Of course, you could get much of that same tactile joy by driving to your nearest yarn store and wandering around the aisles jamming your fingers into every ball and skein of yarn they have. (An activity I certainly also do and highly recommend.) Knitting isn’t just about having a hedonistic tactile experience. It is a skilled art with an unbelievably deep lore. I used to be a game programmer, and I tend to look at a lot of activities through the lens of game design. Games are interesting because they are user experience distilled to its essence. When you are, say, using a banking app to transfer some money into savings, there is a user experience in play. But there’s also a utility, a real effect you are trying to have in the material world. You may use your banking app and be satisfied that it helps you save money even if the UX is trash. But a game, almost by definition, doesn’t do anything “out there”. Aside from leaderboards and stuff, the point of a game is to be a low-stakes sandbox where you can play without, you know, accidentally deleting money out of your bank account. Because of that, it’s a pure user experience. All experience and nothing else. Game designers have to make a user experience so good that you don’t care that you aren’t actually accomplishing anything at all with it. That makes the tools they use to think about gameplay experiences useful lenses to evaluate just about any kind of human endeavor. Two aspects (at least) are relevant to knitting. Linearity#linearity A game is “linear” if there’s only one path from beginning to end, one way to play. Don’t particularly enjoy the desert level? Tough shit, you gotta get through it to get to the end. “Nonlinear” games let players choose among multiple paths to reach the end. “Open world” or “sandbox” games blow the gameplay wide open and let players go where they want when they want. The game may not even have an “end”. Knitting is an open world game. There are all sorts of objects you can make out of yarn, and all sorts of styles and techniques to make them. Don’t like socks? Fine, you don’t ever have to go through a “sock making phase” to graduate into what you really want to make. Find stranded colorwork too fiddly? There’s intarsia or just buy a ball of self-striping yarn and let the yarn change colors for you. Is a sweater too big of a commitment? You can make hats forever. I've made a lot of hats. They're quick projects, only take a single skein of yarn, and almost always fit the wearer. The yarn for these hats was a gift from my father and stepmother. I returned the favor by turning some of the yarn into these two hats and giving them back. This was my first time doing stranded colorwork. That means you are holding two yarns at the same time and switching between them frequently. Holding both yarns and the project and the two needles feels overwhelming, but once you get the hang of it, it’s very satisfying being able to switch between the colors at each stitch. Like making analog pixel art. Once you get past the very basics of getting loops on the needles and making stitches, knitting very rarely forces you to slog through something you don’t want to do in order to reach some other goal. Except weaving in ends, I guess, which is kind of a chore. But, honestly, it’s not that bad. Skill curve#skill-curve A skill curve is sort of like a learning curve. It’s an imaginary graph of how much effort it takes to reach greater and more rewarding levels of skill. Some skill curves are steep at the beginning and then flatten out once you’re over the hump. When you first start playing guitar and don’t have the hand strength, callouses, or dexterity to form chords, it’s really hard. But after a few days you can get the basics down. Then it’s pretty easy to learn more and more chords after that. Other skill curves start shallow and get steeper. You can learn chess in a day and have fun playing it, but as you get more serious about it, each incremental increase in skill requires a greater commitment to studying the theory of the game. Knitting has a marvellously smooth, user-controllable skill curve. There’s a small hump at the beginning. It does take a little while to figure out how to hold the needles, control the tension of the yarn with your fingers, and get your hands to work in concert to make stitches. It feels like you’re making a shadow puppet of a sewing machine. Your initial experience will be frustrating. But you can push through that in an hour or so. In a day, you can learn a basic way to cast stitches onto the needles, make knit and purl stitches, and then cast off to finish the work. With just those, you can make scarves and dishcloths. You are a knitter. Working on a swimsuit for Toad. The legs are stockinette stitch, just simple knit stitches over and over, switching colors every few rows. Then the world is your oyster. There are thousands of patterns out there, each listing the techniques required. You can pick ones well inside your comfort zone and grow your skills slowly. Or you can challenge yourself to learn a bunch of techniques at once. There are dozens of tiny little independent tricks to learn, each it’s own little merit badge serotonin hit: long-tail cast on, increases and decreases, cables, etc. So many fun different ways to form stitches. Each is one bite-sized lesson and no matter how many or few you want to chew on at a time, there is a project out there that will satisfy your appetite. I learned how to do a picot bind-off to make these tiny frog fingers. It’s not just the steepness of the curve, but also its height. Some skill curves top out early. I suspect the world’s greatest kazoo player is not profoundly better at kazoo than I am. Others seem to have no limit, like the world’s best violinists or Go players. Friend, the knitting lore goes deep. People have been developing this artform for literally over a thousand years. Knitters have sat there, brain semi-idle, while their fingers worked yarn for millions of hours. They had plenty of time to invent all sorts of crazy ways to tangle yarn up. You could knit full-time for the rest of your life and never run out of new things to learn. Knitting will never stop rewarding you. This pattern taught me short rows, wrap-and-turns, and cable cast-ons. Stuffed animals are typically made using crochet which more easily handles complex organic shapes, but there are some patterns for knitted ones too, like the famous Emotional Support Chicken giving you the side-eye here. Structured but not a game#structured-but-not-a-game To be clear, I’m talking about knitting and videogames as a metaphor. I don’t think of knitting as a game to win. Like a lot of programmers, I am prone to pointing my dumb optimizing brain at random activities and trying to min-max the shit out of it. Ask my wife how many ways to make coffee I have tried to find the optimal effort/reward ratio. When I knit, I do try to knit efficiently. If it takes 10,000 stitches to finish a single scarf, it behooves one to put some thought into the process. At the same time, knitting isn’t just a pointless exercise to scratch my gamification itch. It is a real artform, and I am making real objects. Thousands of stitches, spooled out from time into space. For me, knitting strikes a good balance between structure and unstructure. I like making music and when it goes well, it’s very rewarding. But sometimes I sit down and every melody that comes out sounds like a knock-off NES game (and not in that cool chiptune way). Or I’ll spend an entire evening working on the drum mixing and at the end I can’t tell if I made things better or worse. It’s too unstructured. When I sit down to knit, I might make a mistake that needs unwinding and fixing. But for the most part, I can be confident that an hour spent knitting will get me closer to a beautiful finished object. It provides a reliable serotonin hit of “I’m making progress”. But it’s not so rigidly structured that it triggers my optimizing brain into sucking the joy out of it by turning it into math homework. A time and a space#a-time-and-a-space A particularly nice property of knitting is that it is able to provide that reliable gratification while accommodating all of the other complexities in my life. It takes very little time to make progress knitting. Unlike, say, painting, there’s almost no set up at the beginning of a session or clean up at the end. When I pick up the kids from school, I can get ten minutes of knitting in while I wait for the bell. Working on a blanket square while I wait for the kids. Nor does it take up much space. A zip-loc bag with a ball of yarn and two needles is basically all you need, which is always in my backpack when I get on an airplane now. Conveniently, the TSA specifically allows knitting needles on flights. Note that this only applies to domestic flights in the US. Other countries have their own rules, though I haven’t had any trouble bringing knitting to Denmark or Costa Rica. The fact that the TSA is explicitly fine with you bringing a satchel full of foot-long sharpened metal spikes onto a plane as long as its accompanied by some string really says something about how meaningless these security rules are. Knitting expands and contracts around not just physical space, but headspace too. Had a long day and want something mindless to help you unwind? Slap together a garter stitch scarf and just do the same stitch over and over again. Stressed out by work or grieving a loss and need something consuming to take your mind off it? Start a lacework or cable knit project and the chart and counting won’t leave room to think about anything else. Working on a little dishcloth while sipping hot cocoa. Whatever logistical or mental capacities you have, there is a knitting project that will tuck neatly into it. And then at the end#and-then-at-the-end So far, I’ve been talking about knitting as an activity. A personal hobby to kill time for your own joy. Kill time it does, but knitting isn’t just about whiling away the hours. It’s not playing solitaire or binging a TV show for the fifth time. As yarn spools through your fingers and the hands twirl around the clock, an actual physical, beautiful object emerges at the end. Well, your first couple of objects may not be so beautiful. But even the lumpiest knitted scarf is imbued with something increasingly elusive these days: care. The first real thing I knitted was a scarf for my mother-in-law. In retrospect, I can’t say it’s a great scarf. Kinda cheap acrylic yarn. Not really her color. 4x4 rib was about all I could handle complexity-wise at the time, and it means the scarf tends to bunch up on itself. But when she opened the package on Christmas and saw it, her eyes teared up. Mine are tearing up now writing this. The scarf. Because regardless of how good the object itself is, it is an inarguable testament to the fact that I chose to spend dozens of quiet hours making stitch after stitch, all the while thinking about her and how much she means to me. A fraction of my life’s wick that I burned for her and no one else. In a world where so many seem to want to get more and more out of less and less, to automate and AI-ify everything until an infinite content firehose is blasting into every orifice of every consumer, hand knitting to me is the antidote. An acknowledgement that all we really have is time and thus there is no gift more precious than spending it on someone. Also, once you finish a project, you get to buy more yarn. Because, if I’m honest, a little consumption feels kinda nice too. OK, I’m sold#ok-im-sold Anyway, this is what knitting means to me. Which, now that I read all this, is a lot more than I realized. If that didn’t pique your interest, fine. It’s not for everyone. I do hope you find something out there to spend your time on that provides as much joy as knitting does to me. You deserve that. If this did make you want to give knitting a try, you’re probably wondering what next. Fortunately, there are, like, a million “learn how to knit” tutorials out there. One of the actually marvellous things about living in the world today is good access to lots of videos, and knitting is an activity that’s really hard to convey in book form. You kind of need to watch someone’s hands. Learning from someone in person is best, but if you don’t have that, YouTube is a pretty good substitute. Keep in mind that everyone’s hands are different! There are many ways (a handful, heh heh) to hold the needles and form stitches because our anatomy and the texture of our skin varies so much. Watch a few videos and don’t worry if what works for them doesn’t work for you. Eventually, you’ll find one that does. I knit “Continental style”. That means I hold the working yarn with my left hand and make stitches with my right. In English style knitting, the right hand both holds the yarn and makes stitches. That works well if all of your coordination is in that hand. But as a lefty, I find Continental takes better advantage of the dexterity in both of my hands. Even though I’m left-handed, I knit the same way a right-handed person does. I don’t mirror patterns or make stitches from left to right. If you are sinistral as well, I recommend still learning right-handed. It doesn’t make that much of a difference in terms of dexterity, and it’s much easier if you don’t have to try to mentally mirror every pattern, instruction, and video you watch. Expect to be challenged and frustrated at first. There’s a lot to going on all at once: controlling the tension of the working yarn, keeping the stitches from the previous row on the left needle, keeping the new stitches on the right needle, forming new stitches. This may be one of the first times you’ve used this many of your fingers doing different things all at once. I promise that if you’re patient with yourself and give it a few tries, you will get over the hump. Once you can knit a swatch of garter stitch, everything else will come naturally over time. So go your local craft store, buy a cheap pair of size 7 needles, a ball of worsted (i.e. medium) weight acrylic or wool yarn in a color you think is pretty, and give it a try. The worst that can happen is you’ll waste a few bucks. If you’re lucky, you might end up making your mother-in-law cry (in a good way).

• Access Control Syntax
  May 26, 2025

  I’m still tinkering on a scripting language for my hobby fantasy console project. I’m ashamed to admit this, but up to this point, the language had absolutely no notion of modules. Literally every source file is dumped into one big global namespace and compiled together. I always planned to have some sort of module system. I just hadn’t figured it out yet because I had other, harder language design problems to solve. I assumed that the module system mostly didn’t interact with other language features, so I could kick it down the road for now. That was true until it wasn’t. I’ve been beating my head against the wall around generics for… oh God I just checked the Git history and it’s three years now. I still don’t have that pinned down. Parametric types are hard. Anyway, one of the approaches I’m exploring does get tangled up in modules and scoping so now I have to figure modules out. This post is about one little syntax design question I ran into: how do you distinguish public and private declarations? A basic module system#a-basic-module-system Since my language is a scripting language, my ambitions for the module system are pretty minimal. Think more like Python or Dart than Java or C#. Every file has its own top-level scope that isn’t shared with others. If you want to access top-level declarations from another file, you import that file. That makes its top-level declarations available in the importing file. Of course, a module might have some declarations that are only for its own internal use and should not be made available when you import it. A module should be able to encapsulate parts of its implementation. Thus, I need a way for users to indicate which declarations are private and which are public. What other languages do#what-other-languages-do Every language out there has some kind of module system and an ability to control access (though it did take JavaScript about 20 years to get there, bless its heart). In some sense, it’s a solved problem. But they don’t all solve it the same way, especially if you dig into some of the more obscure corners of the language world. Let’s go on a tour… Modifier keywords#modifier-keywords The approach you probably already have in mind is modifiers before declarations. In Java, C#, PHP and others, that’s public, private. Maybe also more specific ones like protected, and internal. It’s clear, explicit, and gets the job done. It lets you support a large number of flavors of access control if you need. It’s also extremely common, so easy for users coming to a new language to pick up. The flip side is that it’s quite verbose. Java is reviled for being too boilerplate-heavy and wordy, and I believe that having private and public scattered throughout every single file is a major contributor to that. Picking the wrong defaults doesn’t help. I used to program in C# professionally for several years and in that time, I can’t recall ever wanting a member to have internal access, which is the default. So I had to write public or private on basically everything. I think the same is true in Java land. Rust improves the situation by having a shorter modifier, pub, and picking what is arguably the right default for a language designed for Serious Programming: private. Modifier sections#modifier-sections C++ has a strange but perhaps underappreciated spin on modifiers. It has access modifier keywords, but they apply to all subsequent declarations. This lets you write a keyword once and apply it to a whole bunch of things. That really does cut down the verbosity. The price to pay is that it also makes the language curiously context-sensitive. If you’re ever defined a preprocessor macro that inserts code in a class and had to be careful where in the class you called the macro, you’ve run into this. Access control sections mean that you can’t look at a single declaration and know what its access is. You have to know what section contains the declaration. There’s also the funny historical thing where members in struct default to public and members in classes default to private. C++ is weird. Like a house built on the back of some Eldritch being whose architecture and plumbing reveals the unholy foundation it is mated with. For presumably less Eldritch reasons, I believe Ada takes a similar approach. Sigils in the names#sigils-in-the-names Modifiers are clear but verbose. If you want something more syntactically economical, why not encode the access control directly in the name of the declaration itself? That’s what Python, Go, and Dart do. Python’s system is a mixture of informal and language supported. A leading underscore in a name doesn’t prohibit it from being used outside of the module but it sends a signal to the user that they shouldn’t use it. Sort of “velvet rope” security. If a class member starts with two leading underscores, then it really is private. The language will name mangle it to make it inaccessible. In Go, if a declaration’s name starts with a capital letter, it’s public. Otherwise, it’s private. Because Go allows any Unicode letter in identifiers, the definition of uppercase is not trivial. Wikipedia tells me there are 1,858 uppercase letters and 2,258 lowercase letters that can be used in identifiers in Go. Fun! Dart only allows ASCII in identifiers and follows Python: If an identifier starts with _, it’s private. Otherwise it’s public. Unlike Python, this is strictly enforced by the language. Private names are completely inaccessible outside of the library where the declaration appears. I don’t have enough Python experience to have an opinion there aside from finding __ pretty long. I work full-time on Dart and have only tinkered in Go so I’m highly biased, but Go’s approach has always felt strange to me. I’m sure it’s mostly habit, but my monkey brain really wants case to be used to distinguish kinds of declarations: leading capitals for types and lowercase for functions. Seeing types with lowercase names or functions with capital names gives me the same squishy feeling in my gut as watching The Polar Express. Dumb subjective biases aside, though, it seems to work fine in practice. I’m much more used to Dart’s style. I don’t love it, but, eh, it gets the job done. These approaches both have the advantage of being very terse. No modifiers, no additional reserved words. There are two strikes against them: They’re obscure. If you’re new to Java and you see the word private, you probably don’t know what it does, but you can guess it has something to do with “privacy”. With Go, if you see that some names are capitalized and some aren’t, that tells you absolutely nothing about what’s going on. Historical baggage? Weird personal preference? Maybe the author is German and prefers capitalizing nouns? Likewise, if you’re skimming some Dart code, why do some of the names start with _? Do you pronounce that when you say the name out loud? You just have to be told what’s going on because the language’s own syntax doesn’t guide you towards an explanation. They show up at the use sites. Access control is a property of a declaration. So, in principle, any syntax for specifying it should only appear at that declaration. When you go to use a name, either you can access it (in which case no syntax is necessary) or you can’t (in which case there’s nothing you could say). But with Go and Dart, every place you mention the name also carries the access control. In theory, this could be a major problem if you want to change the access control of a declaration. You have to fix the identifier at every single use site! In practice, this doesn’t actually cause much pain. If a name is going from private to public, then every existing use site is already confined to one file, so you can rename those without affecting the rest of the program. If an identifier is going from public to private, again all uses must already be confined to that file or you can’t make it private without actually breaking things. I used to say that if I had a time machine, I would go back and change Dart to not use _ for privacy. But I have mellowed on that opinion over the years and now I’m not sure. I don’t like how it shows up at every use site, but it is terse, which is nice. Export manifests#export-manifests Modifers put access control at the declaration. Sigils put it at the declaration and the use sites. The third major approach puts it at neither. Someone will tell me this generalization is wrong but I think most functional languages including the ML family and Lisps (including Scheme) have some kind of separate syntax for listing exports. Within a module, you declare all the functions and types you want without worrying about modularity. Then at the top of the file, the language gives you some dedicated syntax to list the things that should be made visible outside of the module. For example, in Scheme, the syntax for defining a module looks like: (library library-name (export list-of-names-to-export...) declarations-in-library...) SML and its spawn have a notion of a signature which is declared separately from the module. You could also argue that header files in C and C++ are this pattern. A cool thing about export manifests is that they keep all the access control logic in one place. When you’re writing declarations, you don’t have to worry about access control then. You just write declarations. As a user of a library, you can look at just the export section to see what the module lets you do without digging in to the implementation at all. It does a good job of firmly separating interface from implementation. On the other hand, it’s quite verbose. You end up saying the name of every exported declaration twice. Maybe even the full type signatures too. Since the export manifest is separate from the declarations, they have to be manually kept in sync. Rename an exported function and you have to remember to rename it in the manifest too. I ragged on JavaScript for not having its act together, but I gotta say when they did add modules, they came up with a clever blended approach. You can use export as a modifier right on a declaration: export function area(width, height) { return width * height; } But you can also use the export keyword to export a set of declarations that are already declared elsewhere: export { area }; // Code... function area(width, height) { return width * height; } Given that modules were added later to the language, I’m guessing this makes it easier to add modularity to a large existing JS library. Sigils at the declaration#sigils-at-the-declaration In poking around the annals of language history, I found one last approach in Niklaus Wirth’s magnum opus language Oberon. In Oberon, top-level declarations are private by default and only accessible to the containing module. If you want to make a declaration public, you mark the declaration with an asterisk: MODULE Hello; IMPORT Out; PROCEDURE SayHello*; BEGIN Out.String("Hello World!"); Out.Ln; END SayHello; END Hello. Here, the * after PROCEDURE SayHello means that SayHello is public. Note that this marker is only at the declaration. It’s not part of the name. At callsites, the name is just HelloWorld. This has the same inscrutability that Python, Go, and Dart have. If you’re a new Oberon programmer and you see *, you have no idea what it means. Fortunately, there are no new Oberon programmers, so this is not a problem in practice. This approach has the advantage of only appearing at the declaration. You don’t have to repeat the sigil every time you use the thing. Any syntax at all?#any-syntax-at-all I’m sure there are interesting tweaks and refinements in various languages, but I think that mostly covers the approaches in use out there. (If you’re aware of any others, do tell.) Which approach is the right one for my little scripting language? A deeply related question is the choice of defaults. Java has public for public stuff and private for private stuff, but it also has a syntax for package private declarations: nothing at all. If you don’t specify anything, C++ makes your struct members public and your class members private. It’s hard to beat “zero syntax at all” when it comes to brevity, so the choice of what access control a declaration gets by default is an important one. When I put on my rigorous software engineer hat—probably some kind of construction site hard hat—the obvious right default is private. No one can fit an entire huge program in their head, so code needs to be broken into smaller isolated pieces you can reason about locally. Defaulting to private encourages users to make smaller independent modules. If I was making a language for big mission-critical infrastructure software, I would definitely do that. I think Rust made the right choice by defaulting to private and requiring you to opt in to public with pub. But I am making, like, the opposite of that. A hobby project that may never see the light of day for people to make their own hobby games that, realistically, also may never see the light of day. If anything, I want to encourage (hypothetical) users of my (currently mostly vaporware) language to get things done. Access control can get in the way of that. Class modifiers in Dart#class-modifiers-in-dart A few years ago, we added a slew of class modifiers to Dart. These are not directly tied to public and private, but in the process of designing those, I spent a lot of time talking to users about how they use the language and what defaults they prefer. Dart defaults to public. You have to opt in to private by prefixing a name with an underscore. It turns out that most users actually do prefer public being the default. And after looking at mountains of Dart code, what I see is that most Dart code is relatively small applications. Those apps are built on top of a set of frameworks and libraries, but compared to the volume of application code out there, those libraries make up a fairly small fraction of the entire Dart ecosystem. These apps are the leaves of the dependency tree. Nothing depends on them or imports them. The authors of most of those are just trying to slap some UI together and ship a thing. And for most of those, defaulting to public keeps the language out of their way. Syntax for privacy#syntax-for-privacy That all leads me to feel that my scripting language should default to public and have a way to opt in to private. How do to that? No export manifests#no-export-manifests I know I don’t want export manifests. They are much too verbose. Also, there is some extra language complexity and error reporting that gets sucked in. Since the manifest is separate from the declaration, the compiler has to handle cases where you try to export a name that doesn’t exist. My language also has overloading, which means when exporting a function, you would need to a way to specify which of the overloads to export and which to leave private. Export manifests are just way more machinery than fits for a scripting language. This approach is out. No modifier sections#no-modifier-sections The C++ approach is nice in that it’s clear while still being pretty terse. You only need to write a modifier and can apply it to a bunch of declarations in the same scope. But at least in C++ (not sure about Ada), that only works within a type declaration. My language isn’t really object oriented. It’s more procedural. Sort of “structs plus functions” like C. Instead of classes, you have records. But a record declaration tends to be pretty small with only a handful of fields in it. There’s not enough stuff inside a single record to benefit from reusing a modifier for a bunch of nested declarations. I could allow private at the top level and have it apply to everything after, but that feels like it errs too far in the other direction. It would be really confusing to have a thousand line file and not realize that half of it is private because some modifier way offscreen above flipped the access control. Modifier sections are out. No sigils in identifiers#no-sigils-in-identifiers Sigils in identifiers do have a sort of terse, scripty feel. No extra keywords. I can’t do Go’s approach because my language already makes identifier leading case significant. It supports destructuring pattern matching. Like some functional languages, it uses case to distinguish when a pattern is binding a new local variable (lowercase) versus matching against a type or constant (uppercase). Now I’m not sure that corner of the language will stick, but it’s there right now. I could do Python/Dart’s approach and use a leading underscore. But I’d really prefer to not have to mention the sigil at every single use site. After having written probably a million lines of Dart code, I know it’s not intolerable. But it feels like an annoying tax. And for a language I’m designing for my own joy, I’d like to eliminate as many annoyances as possible. Maybe modifiers#maybe-modifiers That leaves two approaches and is where I’m currently at. Approach one is the typical solution to use an access control modifier at the declaration. Unfortunately, defaulting to public makes this harder. In Rust, pub is a really nice little keyword to flip to public when the default is private, but there is no obvious converse. private would be the longest keyword in my language. pri doesn’t read like anything. priv is… strange. I suppose that pvt is the well-established abbreviation for “private” but that feels a tad militaristic. I can’t find any other synonyms for “private” that admit reasonable abbreviations either. So I’m open to taking this path, but I’m simply failing to come up with a good keyword for it. (A silly goblin part of my brain suggested I use shh to mean “private”, like the declaration is being whispered and can’t be heard outside of its module.) Maybe sigils at the declaration#maybe-sigils-at-the-declaration This leaves Oberon’s weird approach. There are several advantages here: It is maximally terse. Zero syntax at the use site, and only a single character at the declaration site. No additional reserved words which might get in the way of user identifiers. By defaulting to public, the opacity of the syntax is less of an issue. Yes, the sigil won’t immediately convey what it does, but if you’re just starting to use the language, you don’t need to even know it exists at first. You can just make everything public. So there is some appeal. I don’t think I can literally take Oberon’s syntax with * following the name. My language lets you define operators and it would be confusing to have * right after other punctuation: def +*(left Vec, right Vec) Vec(left.x + right.x, left.y + right.y) end This is declaring a private + function. It would probably look equally weird to have * after the name in a record declaration: rec Vec* val left Int val right Int end That led me to thinking the sigil should be by the declaration keyword. In my language, every declaration form does start with a leading keyword (unlike, say, C where function declarations don’t have one), so it’s feasible. Something like: rec* Vec val left Int val right Int end def* sayHi() print("Hello!") end I don’t hate it. But using * feels weird for “private”. If anything, it seems to emphasize the declaration (which is what it does in Oberon where it means “public”). Maybe alternate keywords#maybe-alternate-keywords That led to my last idea which I can’t decide if I like or not. Given Python and Dart, it seems like underscore vaguely conveys “private” to some people. So maybe use that? It looks like: rec_ Vec val left Int val right Int end def_ sayHi() print("Hello!") end It looks odd, but I don’t hate it. Or at least I don’t hate it any more than any novel programming language syntax usually triggers revulsion. However, there’s sort of a problem. Underscore is a valid identifier character. So the lexer is not going to scan rec_ as a rec keyword token followed by a _ token. Instead of will treat rec_ as a single identifier. But then… I could just let it do that. I’d then define a separate set of reserved words for all of the private declaration keywords: def_, rec_, var_, and val_. So now instead of a modifier or sigil to control access, you explicitly choose one of two declaration keywords. One makes it public and one makes it private. This technically means adding a handful more reserved words, but they aren’t ones that are particularly useful for users anyway. I keep trying to talk myself out of this approach because it’s so unusual but so far it seems to be lodged in my head better than any of the alternatives I’ve considered. What do you think?

• Does Go Have Subtyping?
  Oct 19, 2023

  I’ve been noodling on a static type system for my current hobby language. To try to keep the language as simple as possible, I’m trying to see if I can live without subtyping. Since most of my programming experience is in object-oriented languages, I’ve been learning more about languages that lack—or at least claim to lack—subyping, to see how they work. The most intriguing one to me is Go because the authors say it doesn’t have subtyping, but when you look at interfaces, it does seem to have something really close to subtyping. Is it subtyping just under another name, or is there really something different going on? This post is the answer to that question as best as I can tell. The short answer is that no, Go doesn’t have subtyping. But also, yes, it sort of does. What is subtyping?#what-is-subtyping If you’re reading my blog, you probably already know what subtyping is, but let’s make sure we’re all starting from the same place. Subtyping defines a relation between two types. Given two types A and B, it might be the case that B is a subtype of A, or it might not be. Since subtyping is a relation between a pair of types, it only comes into play in places in code where two types are involved. The main place is assignment. You have an expression of type A and you assign the result to a variable with type B. Is that assignment allowed? Programming language folks usually generalize “assignment” to mean any place where a variable is given some value. That includes assignment expressions, but also covers initialized variable declarations and function calls where argument values passed to the function are bound to their corresponding parameters. There are a couple of other places where subtyping comes into play, usually around type inference, but assignment is the main one: You have a context that requires some type B and a value of some type A. What are the types A and B where that code is valid? That question is the heart of what a type checker does. The main user interface of a static system is compile errors, and the most common compile error is “I expected a value of this type but you gave me a value of this other type”. Why have subtyping?#why-have-subtyping You have a context that expects type B and you give it a value of type A. In languages without subtyping, that’s only OK if A and B are the exact same type. In Pascal, if you declare a variable with type integer, the only thing you can initialize it with is a value of type integer. Subtyping exists largely to loosen that restriction—to allow multiple different types to flow into some context. Why might a language want to permit that? The reason is polymorphism: Subtyping lets you write a piece of code and reuse that same code with a range of different (but related) types. In languages without subtyping, you can often find yourself copy/pasting the same function to work with multiple different input types. (Generics can help, but that’s another form of polymorphism that we’ll ignore for this post.) In, say, Java, if you define a method that takes an Iterable, then you can pass a List to it, a Stack, etc. You get to amortize the usefulness of that method across all types that implement the Iterable interface. Subtyping is a force multiplier for your code. (Of course, that benefit isn’t without significant costs in terms of language complexity, which is why I’m hoping to avoid it.) Does Go have subtyping?#does-go-have-subtyping If you search the (extremely well-written!) Go language spec for “subtype”, you get zero results. So the answer is a clear “no” at the textual level. However, we needer a deeper hermeneutics. Java does have subtyping. Now, if you were to make a new language named “Blava” that was a literal copy/paste of the Java language specification with every use of “subtype” replaced with “blubtype”, would you say that Blava has subtyping? It behaves indistinguishably from a language with subtyping, so I’d be inclined to say yes. The Go spec doesn’t mention “subtype”, but it does have a notion of “assignability”. When you have a context that expects some type and you give it a value of some other type, assignability determines which set of other types are allowed. Concretely, the rules are: A non-interface type T is assignable to an interface type I if T implements I. An interface type A is assignable to interface type B if A’s methods are a superset of B’s. You know, that sounds an awful lot like subtyping. Is “assignable to” just Rob Pike’s idiosyncratic way of saying “subtype of”? Does Go have subtyping in everything except name? Are we just playing semantics? (I mean, we’re designing a programming language, so obviously everything we do is playing semantics. But I mean are we playing semantics with the language spec itself?) To fully answer that, we’ll need to look at all of the kinds of types in a program. Composite types and variance#composite-types-and-variance If the only types in Go’s type system were primitives like numbers, structs, and interfaces then I think you’d have a good argument that Go does have subtyping, just spelled differently. But once you start looking at slice types and function types, the story changes. (And array and channel types too, but slices and functions are enough to make the point.) The thing that these latter kinds of types have in common is that they contain other types. A slice type has an inner type for the slice elements. A function type has a list of parameter types and a list of return types. You ready for some more computer science jargon? We’ve been talking about relations on pairs of types like “is subtype” and “is assignable”. But now we have types that contain other types. That raises the question of whether a relation on the inner types of two composite types says anything about the relation between the two outer types. For example, let’s say we have two slice types []E1 and []E2. They have element types E1 and E2, respectively. If E1 is assignable to E2 does that mean that []E1 is assignable to []E2? Does the assignability “propagate” from the inner types to the outer types? Computer scientists call this property (meta-property?) variance. They phrase the question like “how does assignability of slice types vary with respect to their element types?”. There are a few possible answers to a question like this. Variance of slice types#variance-of-slice-types For slice types in Go specifically, there are a handful of assignability rules, but the only one that applies to slice types is: V and T are identical. In other words, for two slice types to be assignable, they have to be the exact same type. That in turn means they must have the exact same element types. Even if two element types are assignable, slices of those two types are not. Judging by an endless series of confused people asking questions on StackOverflow, that behavior is unintuitive to programmers, both in Go and in other languages. Let’s say you have this Go program: type Dog struct { name string } type Barker interface { Bark() } func (d Dog) Bark() { fmt.Println("Woof!") } Here we have a Dog concrete type, which is assignable to the interface Barker. So this is fine: func speak(barker Barker) { barker.Bark() } func main() { speak(Dog{"Sparky"}) } Given that, you might expect this to work too: func speakAll(barkers []Barker) { for _, barker := range barkers { barker.Bark() } } func main() { dogs := []Dog{Dog{"Sparky"}, Dog{"Fido"}} speakAll(dogs) } But, no. The type system giveth and the type system taketh away: example.go:29:11: cannot use dogs (variable of type []Dog) as []Barker value in argument to speakAll If the type system didn’t yell at you, this program would be fine at runtime. All it does is call Bark() on every element in the array, and both Sparky and Fido do implement that method. So what’s the deal? In this case, the program is coincidentally fine because speakAll() is only reading from the slice. But what if we wrote: type Tree struct { species string } func (t Tree) Bark() { fmt.Println("Rough (but not ruff)!") } func appendTree(barkers []Barker) []Barker { return append(barkers, Tree{"Elm"}) } There’s nothing wrong with this appendTree() function. It adds a Tree to the given slice. Since Tree is assignable to Barker, that’s fine. But if you were to call this and pass in a []Dog, you’d end up with an array of dogs that had a tree stuck in it! That would violate the soundness of the language. This is why Go only treats two slice types as assignable if they have the exact same element types. In PL parlance, slice types are invariant with respect to their element types. And, for a mutable data structure like slices, that rule makes sense. (A reasonable person might wonder then why Java and C# don’t have this rule and instead say that array types are assignable if their element types are. And then, because as you can see, it isn’t safe to do so, they have to add runtime checks if you try to stuff an element of the wrong type into the array.) So, OK, it makes sense for slice (and array) types to be invariant. What about function types? Variance of function types#variance-of-function-types To keep things simple, first we’ll consider just functions that don’t take any parameters and have a single return type. Given two function types like that, when are they assignable? Again, the only rule in the Go language spec that matches function types is V and T are identical. So two function types are only assignable if they have the exact same return types. Even if the return types are themselves assignable, if they are different types, the functions aren’t assignable. Do we need to be that strict to preserve soundness? Actually, no! Here’s an example: func returnDog() Dog { return Dog{"Rex"} } func useCallback(callback func() Barker) { barker := callback() barker.Bark() } func main() { useCallback(returnDog) } So we have a function, returnDog that returns a value of type Dog. We pass a reference to that function to useCallback() whic expects a function that returns a Barker. The Dog type does implement Barker. If this program were run, it would be perfectly safe. And, in fact, there’s nothing you could put inside useCallback() that would make passing returnDog to it violate the soundness of the type system. It’s sound and semantically kosher in principle… but Go disallows it: ./prog.go:49:14: cannot use returnDog (value of type func() Dog) as func() Barker value in argument to useCallback Every other language I know that has subtyping and function types allows this. A function type A is a subtype of another function type B if the return type of A is a subtype of the return type of B. So the subtyping relation of the return types propagates out to determine the subtype relation of the function types. We call this covariance and say that function types are covariant in their return types. The “co-” prefix means that the subtyping relation between the inner types goes in the “same direction” as the subtyping relation it implies about the outer types. That direction matters because relations like subtyping and assignability aren’t symmetric. The Dog type is assignable to Barker, but Barker is not assignable to Dog. The underlying value might be a Tree! Are there cases where the variance of an inner type doesn’t go in the same direction as the outer types? Indeed there are, and they’re right there next to us. Instead of return types, let’s look at parameter types. Now let’s say we only care about functions that accept a single parameter and return nothing. Here’s an example: func acceptBarker(barker Barker) { barker.Bark() } func useCallback(callback func(Dog)) { callback(Dog{"Laika"}) } func main() { useCallback(acceptBarker) } Note that the parameter types are flipped compared to the return type example. Here, the callback type in useCallback() takes a more precise type of Dog. The function we pass to it, acceptBarker has a parameter whose type is Barker. You may feel a slight disorientation here. The code feels weird and sort of backwards. Wait a minute and the dizziness will pass. Dramamine might help. While this definitely isn’t as intuitive as return types being covariant, if you think about it carefully, you’ll see that the above program is completely sound. In other languages with subtyping, function type A is a subtype of function type B if the parameter types of B are subtypes of the parameter types of A. Note how A and B are reversed in the second half of that sentence. The variance of parameter types is reversed. In technical terms, we say that function types are contravariant in their parameter types. The prefix “contra-” means “against”. (You might wonder what happens when you have a function type with a parameter whose type is itself a function type with some parameter type. How does that flow out? When there’s two levels of nesting it flips around to going in the same direction as the outermost type. The way I think about it is that contravariance is a 180° flip in the direction of the relation. If you nest contravariant types, you flip it twice and get back to the original direction.) Contravariant parameter types are sound, but again Go doesn’t allow them. Two function types are only assignable if their parameter types are exactly the same. Invariance in Go#invariance-in-go In every language I know with subtyping, function types are covariant in their return types and contravariant in their parameter types. But in Go, function types are invariant. Go is not a language known for getting in the programmer’s way when they want to do something, so why are function types more restrictive than would be necessary for soundness? It’s not just function types either. All composite types are invariant in Go: arrays, slices, channels, maps, functions. So ground types—types that don’t contain any other type—have some subtype-like notion of assignability. But once you wrap a type in another, any notion of assignability goes away. Why did the designers of Go do that? If you’re going to bother having interfaces and assignability, why not go all the way and have assignability for functions and other composite types where its sound? If all the designers cared about was semantic correctness and having a beautiful elegant specification written in LaTeX, then they probably would have supported variance, at least for functions. (The other types all should be invariant since they are mutable. When types can flow both in and out, any other variance isn’t sound.) But Go was designed from day one to be a high-performance systems language. It’s the exact opposite of an ivory tower language designed for proofs and publications. The goal of the language is to let real users ship real applications. And, importantly, be able to ship fast applications and reason about the performance of their code. Representing values#representing-values Up to this point, we’ve only been concerned with how types flow through the type checker at compile time. But—assuming there are no compile errors—the compiler eventually excretes some machine code which gets executed at runtime. When that happens, all of the types the type checker poked at have cracked out of their chrysalides and emerged as beautiful runtime value butterflies flitting around in memory. The choice of how values of different types are represented in memory has a massive effect on performance. So how do the rules around assignability and subtyping interact with those representation choices? In many object-oriented languages (Java, C#, Python, etc.) values of object types are represented by pointers to a heap-allocated structure. That structure has some header information for garbage collection and runtime type tracking, maybe some kind of pointer to a vtable for virtual method dispatch, then (finally!) the memory used to store the instance’s fields. There are differences between language implementations, of course, but objects are generally both: Slow to create since they are allocated on the heap. Fairly large with some additional bookkeeping information stored for every single object. Indirect, where a variable or field whose type is an object holds only a pointer to that object, which is always on the heap. Accessing state on an object always requires a pointer indirection which can be slow due to poor locality and cache misses. Struct types#struct-types Those are unnacceptable costs for a systems language like Go. When you want runtime polymorphism, of course, you have to pay for it somehow. But if you’re just storing data in memory, Go doesn’t want to make you pay for something you aren’t using. To that end values of struct types in Go store just the bytes needed for the struct’s own fields. If a field of a struct is itself some struct type, the inner struct’s fields are splatted directly into the surrounding struct’s contiguous memory. If you have a local variable of a struct type, the fields are stored right on the stack (unless you take a pointer to the struct which escapes the function). This reduces memory overhead for structs and (probably more importantly for performance) reduces pointer indirections. In a typical Java program, the heap ends up being a huge spiderweb of tiny objects all pointing to each other and the poor CPU an exhausted spider traipsing all over that web trying to find the actual bits of data it wants to eat. In a typical Go program, more state is stored directly on the stack, and the heap is “chunkier” with fewer, larger blobs of memory. The CPU does fewer hops around the heap and chews on bigger data insects every time it does. That makes memory access more cache friendly and also lightens the load on the garbage collector since there are fewer individual allocations to traverse. (Java does something similar for primitive types, as does C# for struct types.) Interface types#interface-types So structs are fast, great. But Go does feature runtime polymorphism in the form of interfaces. How does interface method dispatch work if a value is stored directly inline with no extra data to track its runtime type or method implementations? The answer is that interfaces have a completely different runtime representation. A variable of interface type takes up two words: A pointer to the type information used for runtime dispatch of the interface methods (in other words, basically a vtable). A pointer to the actual data used by the concrete type implementing the interface. (In cases where the data is just a single word, I think it’s stored inline.) The cute industry term for a representation like this is fat pointer: instead of a single word with a single pointer, it’s a pair of them, one for data and one for some kind of metadata or bookkeeping information. One of the really cool things about Go is that you only use this representation—you only pay for the increased memory and indirection cost of this representation—when you ask for it and when you need it. In places where you need some virtual dispatch, you use an interface type and accept the overhead of a fat pointer and indirection. But in places where you just want to store a single concrete type, you use its underlying type and the memory is stored directly inline. C# supports a similar distinction with classes and structs. But that’s mostly a “declaration time” choice. Once you’ve decided something is a class, every variable of that class’s type will store it as a reference to a heap-allocated object. Conversely, if you’ve declared something as a struct, it will always be stored inline on the stack or in the containing object (unless you go out of your way to box it). In Go, the distinction between stored inline versus stored indirectly is made at each use site. That leads to some additional complexity for the user: they always have to think “should I use an interface, pointer, or struct type here?”, but it gives them more fine-grained control over how they spend memory and pointer indirection costs. Implicit conversions#implicit-conversions We’re close to understanding why Go lets you assign a struct to an interface but not a slice of those same structs to a slice of that same interface. If structs and interfaces have entirely different memory representations, how does assignability work at all? When you do: type Dog struct{ name string } type Barker interface { Bark() } func main() { var barker Barker = Dog{"Rex"} } Shouldn’t that just mangle memory when it tries to treat the memory representation of a struct as if it were an interface? The answer is of course no. When compiling your code, Go knows the type of every variable and every expression. At every assignment, variable declaration, or parameter binding, it reports an error if the value isn’t assignable to the destination. When the value is assignable, the compiler also knows whether or not those types are the same. If they’re exactly identical, then the assignment can be compiled to a single register move or memory copy. When they are different but still assignable types, the compiler silently inserts code to convert the value type’s memory representation to the destination type’s representation. When you assign a value of a struct type to an interface type, the compiler inserts code to build a fat pointer, wire up its method table pointer to the right interface implementation, move the struct’s data onto the heap, etc. Likewise, if you assign one interface type to another, the compiler inserts code to copy the data pointers over but look up the correct method table for the destination interface given the type information of the value’s interface type. This right here is the reason that all composite types are invariant in Go. When assigning a single value to a related but different type, the compiler can easily insert fixed-cost code to convert the value’s runtime representation to the destination type’s. But to convert a slice of some struct type to a slice of an interface type would require an O(n) traversal of the entire slice to convert each element. Function types are even harder. In order to support covariant return types and contravariant parameter types, the compiler would need to insert conversion code somewhere, but there’s no right place to put it. Putting it inside the function itself doesn’t work because it might be called with a variety of different parameter types and we don’t know what to convert it from. Putting it at the callsite before the parameters are passed likewise wouldn’t work because we don’t know what types every callback might require. There is potentially something clever you could do by supporting multiple entrypoints to functions for each pair of source and destination types, but with multiple parameters you quickly run the risk of exponential code size explosions. This is why languages that do support subtyping and variance almost always have a uniform memory representation for all objects that participate in the subtype hierarchy. Does Go have subtyping?#does-go-have-subtyping If you make it this far, congratulations. This ended up being a much deeper dive than I expected. I learned a lot exploring this corner of the language space and I hope you learned something too. Getting back to the original question, I think we could accurately describe Go’s subtyping story in two equivalent ways: Yes, Go has subtyping, but it has no support for variance and all composite types are invariant. This is, I think, how someone who is focused only on the abstract semantics of the language would describe it. If you were writing papers about type systems and needed to model Go’s you might adopt this perspective. If you didn’t care about how Go could be efficiently implemented because you were treating it purely as an abstraction, then this is a good way to look at it and compare it to other languages. The main problem with the looking at the language this way is that it obscures why every composite type is invariant. No, Go doesn’t have subtyping, but it does have implicit conversions between some pairs of types. This is how the designers of Go describe the language. It’s the way you’d want to look at the language if you were tasked with sitting down and writing a production quality implementation of it. It describes what the language actually does mechanically at compile time and runtime. The challenge I found with this perspective is that it made it harder for me to relate Go’s design choices to other more explicitly object-oriented languages. You can look at this entire long article as my process of trying to figure out the first interpretation in terms of this one. I started digging into this not because I’m an active Go user and want to know what’s going on under the hood. My job and hobby is designing programming languages, so I want to know how other languages work to see what good ideas are out there to be harvested. So the question always on my mind at this point is, “Why did they design it this way and does that choice make sense in other languages?” And for this specific design choice, I think it’s pretty cool. You can imagine a language wanting three things: Non-uniform representation: Values in memory take up only as much space as they need and avoid pointer indirection when possible to maximize runtime efficiency. Polymorphism: The ability to reuse code to work with a range of values of different types. Variance: Sort of the “lifted” form of polymorphism: The ability to reuse code to work with composite types that contain a range of inner types. Those are all nice to have features, but it’s really hard to get all three at once. Most object-oriented languages sacrifice the first one to get the other two. That gives you flexibility and expressiveness but at a pervasive runtime cost spread throughout the entire program. Some statically-typed languages with simpler type systems like C, Pascal, and SML (ignoring modules, which are a whole other thing) give up polymorphism and variance which can give you more efficient representations at the cost of less code reuse. Languages like C++ and Rust more or less give you all three at the expense of the compiler monomorphizing and generating specialized versions of a function for the different types it gets passed, which makes compilation much slower and can have some runtime costs from all of the extra code sitting around in memory. Go is aiming for a sweet spot where they give you fast compiles, efficient runtime execution, and as much flexibility as they can get away with. It sacrifices variance but keeps polymorphism at the individual value level. That married with implicit conversions enables non-uniform representation. Of the three, variance is probably the least valuable for users, so I think that’s a pretty smart trade-off.

• Representing Heterogeneous Data
  Aug 04, 2023

  As I mentioned in the last post, I’m working on taking my little videogame scripting language and turning it into a statically typed one. As much as possible, I’m trying to make the language simple and familiar. But sometimes those goals are in opposition and the most familiar solution to a problem is kind of a mess. So, I’m also exploring novel approaches and delving deeper into programming language history to scavenge forgotten ideas. The heterogeneous data problem#the-heterogeneous-data-problem One problem every language has to solve is giving users a way to represent heterogeneous data. By that, I mean: Data that might or might not be present. Imagine you have a record for storing a street address: rec Address var number Int var street String var apartmentNumber Int var city String var zipCode Int var state String end But some addresses don’t have apartment numbers. How do you store the apartment number when an address has one but also support its absence? Data that might be in one of several different forms. You’re making a game where a hero can wield weapons. Melee weapons like swords have a single number for how much damage they do. Ranged weapons like crossbows have a pair of numbers for the minimum and maximum range they can reach. How do different kinds of weapons have different fields? These are two sides of the exact same coin. You can treat optional data as data that can be in one of two forms: present with an associated value or absent with no value attached. Functional languages with an option or maybe type do exactly that: The language directly supports data that can have one of multiple forms, and they model absent data using that. Conversely, you could model data being in one of several different forms by having separate fields for all possible forms it could be in. At any point in time, only one of the fields has a value and the others are all absent. If you’ve ever found yourself building a struct or class and writing a comment that says “If this field is blah then this other field will be null.” then you’ve taken this path (and probably felt a little gross doing it). What other languages do#what-other-languages-do I don’t know if broad language tours are your thing, but so much of my job working on Dart involves researching how other languages solve a problem that I can’t help myself anymore. There are a handful of solutions to the problem. I’ll just throw out the ones I know: Null. The most famously maligned approach is to allow any reference variable to potentially refer to “null”, “nil”, or nothing. This means every reference type can directly also represent an absent value. Of course, the problem is that many data fields aren’t heterogeneous and should always be present. If you make every single reference nullable, you’ve lost the ability to distinguish ones that can be absent from ones that really shouldn’t be. This is why many newer statically typed languages either don’t support null at all (Rust and most other statically typed functional languages) or support non-nullable types (Dart, Kotlin, and TypeScript). Variant types. A “variant” type is a type that can hold a value of one of multiple different types. You can think of nullable references as a limited variant type that can hold either a value of one specific type or the special null value but that’s it. Some languages have looser variants that let you store values of any type in the same variable. Untagged unions. C lets you define a data structure whose fields all share overlapping memory. If you have a few different pieces of data that are disjoint—you should only ever have one of them and not the others—then this avoids the memory overhead of storing them all separately. However, in C, the language itself doesn’t keep track of which piece of data you have in the union. It will freely let you write one field and then read out another and it will happily just reinterpret the bits in memory as that other type. Efficient, yes. Safe? No. Sum types. Functional languages going all the way back to ML have a feature also sometimes confusingly called “unions” that is fairly different. Again, you have an object that can store one of a few different kinds of data. But the language also stores a tag in there so that it knows at runtime which piece of data you have. (This is why they’re also called “tagged unions” or “discriminated unions”.) The language uses pattern matching to cleverly prevent you from accessing the data as the wrong type. Subtyping. The object-oriented dual to sum types is subtyping: either inheritance or interface implementation. In an object-oriented language, we could model our weapon example like: interface class Weapon {} class MeleeWeapon implements Weapon { int damage; MeleeWeapon(this.damage); } class RangedWeapon implements Weapon { int minRange; int maxRange; RangedWeapon(this.minRange, this.maxRange); } Code that wants to work with weapons generally uses the Weapon supertype. The two subtypes for melee and ranged weapons each store the fields they need. If you want to go all the way to an object-oriented style, these fields would be private and then you’d have abstract methods in Weapon that are overridden in the subclasses to use them. It’s a complex, heavyweight approach, but a powerful and flexible one. There may be a couple of other weirder language features you can use to model varied data, but I think these are the big ones. Languages tend to pick and choose from this list: Dynamically typed languages essentially treat all variables as variant types. And all the ones I know also go ahead and allow null too. If you’re not going to have any static checking anyway, you may as well be maximally permissive, I guess. Statically typed functional languages lean really hard on sum types. Object-oriented languages obviously primarily use subtyping, though most also have nullable reference types. C makes pointer types nullable and supports untagged unions. It doesn’t have (checked) variants or subtyping, but it can approximate both by allowing pointers to be cast to different types. C++ takes everything C has and also explicitly supports subtyping. Newer, bigger multi-paradigm languages like C# and Swift tend to take just about all of the approaches. Whither for my little language?#whither-for-my-little-language OK, so what’s the right approach for my aspirationally simple and elegant statically typed game scripting language? I quite like object-oriented programming in general, but subtyping adds a lot of complexity to a static type system, so my current plan is to not have subtyping in the language at all. That rules out that approach. My goal is for the language to be fairly high level and expressive. It’s supposed to be a language that makes making games fun, not necessarily a high-performance machine for engineering giant AAA games. I want you to have a good time tinkering on pixelly 2D games, not write the next Unreal Engine in it. To that end, the language is garbage-collected. That means memory safety, which rules out untagged unions. Back when this language was dynamically typed, it had nil, so that’s an obvious approach. But I’ve spent, like, way too much of my time rooting out nullable references from Dart and the last thing I want to do with my hobby project is to go back to square one. That basically just leaves sum types and variant types. Given that my language is statically typed and not object-oriented, sum types are the obvious approach. Everyone who uses sum types loves them, myself included. Algebraic datatypes are just cool. And, in fact, I went ahead and implemented a protype of sum types and pattern matching and destructuring in my language. It worked. It was… just OK. To explain why requires a little context An imperative, procedural language#an-imperative-procedural-language My language is unabashedly imperative. I like imperative programming, especially for scripting little games. Games are giant balls of mutable state. I’ve watched my kids and many others learn to program, and imperatively modifying stuff seems to be a natural way to think about defining a process. When you read a recipe for cake, you don’t see steps like: “Produce a new bowl of batter which is the previous bowl of batter and 2 cups of sugar.” It just says “Add 2 cups of sugar to the bowl.” Now, I know all of the problems with mutation of state and imperative code when programming in the large. I get it. But this is supposed to be a fun little language for fun little games and, to me, imperative programming fits that to a tee. The basic vibe I have for the language is similar to Pascal, C, or BASIC: In other words, a classic procedural language. Structures and functions. It looks like this: rec MeleeWeapon var damage Int end def attack(weapon MeleeWeapon, monster Monster, distance Int) if distance > 1 then print("You are out of range.") return end var damage = rollDice(weapon.damage) if monster.health damage then print("You kill the monster!") monster.health = 0 else print("You wound the monster.") monster.health = monster.health - damage end end What’s cool about simple procedural code is that even though I have no idea what language you know and you certainly have never programmed in this one, I’m still pretty confident that you understand this code. With sum types#with-sum-types Let’s see how it looks with something like sum types: rec Weapon case MeleeWeapon var damage Int case RangedWeapon var minRange Int var maxRange Int end def attack(weapon Weapon, monster Monster, distance Int) var isInRange = match weapon case MeleeWeapon(damage) then distance == 1 case RangedWeapon(min, max) then distance >= min and distance max end if !isInRange then print("You are out of range.") return end var damage = match weapon case MeleeWeapon(damage) then rollDice(damage) case RangedWeapon(min, max) then max - min end if monster.health damage then print("You kill the monster!") monster.health = 0 else print("You wound the monster.") monster.health = monster.health - damage end end The sort of weird rec syntax is defining a sum type, Weapon, with type constructors MeleeWeapon and RangedWeapon. I’m still noodling on the syntax. Now, the code here works. And it’s safe. The compiler and the structure of the pattern matching code itself prevent you from accessing the wrong fields from a weapon of a different kind. That’s cool. But it’s so much weirder than the previous code. In a procedural language, the idiomatic way to access fields on records is simply record.field. That syntax is in almost every programming language all the way back to Algol. But once you hop over to sum types, you lose that syntax entirely and have to instead sort of “invert” the code and use pattern matching and destructuring. I do love pattern matching and destructuring—I just spent the past year of my life adding it to Dart. But for this language, I’m pushing really hard on simplicity. If possible, I don’t want two different ways to access state on a value, depending on whether the field is case-specific or not. More to the point, there’s no graceful way to handle mutable sum type fields using pattern matching. SML eschews mutability in general and then works around it by allowing you to define explicit mutable ref types. But that’s definitely not how my language rolls. Variant records#variant-records There is one other approach to heterogeneous data that I found that I didn’t put in the list up there because, as far as I can tell, it’s basically a dead end in the evolutionary history of programming languages. Some versions of Pascal have a thing called “variant records”. A record in Pascal is your basic “collection of fields” struct type. A variant record says that some of those fields are only accessible when the record is one of a few different enumerated states. In C, it’s common to wrap an untagged union in a struct along with a tag enum indicating which branch of the union is active: typedef enum { WEAPON_MELEE, WEAPON_RANGED } WeaponType; typedef struct { WeaponType type; union { struct { int damage; } melee; struct { int minRange; int maxRange; } ranged; } as; } Weapon; Using it looks something like: Weapon weapon; weapon.type = WEAPON_MELEE; weapon.as.melee.damage = 6; A variant record in Pascal (as I understand it from the half dozen ancient slideshows I’ve been able to find about it) essentially models that pattern directly. The cool thing about this feature is that the variant-specific fields are accessed using the same familiar field access syntax used everywhere else. That also means variant-specific fields can be mutable. Of course, the not cool thing about using that same field syntax is that there’s nothing preventing you from accessing the wrong variant field: Weapon weapon; weapon.type = WEAPON_MELEE; weapon.as.melee.damage = 6; printf("Min range %d\n", weapon.as.ranged.minRange); // Oops. There is a type tag, but the language doesn’t know and doesn’t check it. This is definitely true in C and I think true in Pascal. (It’s always hard to talk about Pascal definitively because there’s no “Pascal”, just a huge family of loosely-related Pascal-ish languages.) In a memory safe language like mine, I definitely don’t want users to be able to reinterpret memory. But that’s a solvable problem. Record cases#record-cases Which, finally, brings us to the feature I designed for my language. It’s very close to variant records in Pascal. The type declaration looks just like the sum type example: rec Weapon case MeleeWeapon var damage Int case RangedWeapon var minRange Int var maxRange Int end The difference is that you don’t need to rely on pattern matching to access the variant fields. They’re just fields: def attack(weapon Weapon, monster Monster, distance Int) if weapon is MeleeWeapon and distance > 1 or distance weapon.minRange or distance > weapon.maxRange then print("You are out of range.") return end var damage = if weapon is MeleeWeapon then rollDice(weapon.damage) else weapon.maxRange - weapon.minRange end if monster.health damage then print("You kill the monster!") monster.health = 0 else print("You wound the monster.") monster.health = monster.health - damage end end Of course, you lose the compile-time safety that pattern matching gives you where you can’t access fields of the wrong type. But we don’t need to go all the way to C’s level of unsafety. Instead, when you access a case-specific field on a record, if the record’s type tag is set to a different case, the access throws a runtime error. This preserves memory safety. This is a real trade-off. The feature I have here provides strictly less static safety than using sum types. There is a slight performance cost to checking the type tag when accessing case-specific fields. In return, you get simpler, more familiar syntax for working with case-specific fields, including mutable ones. Also, it allows a single record to have a mixture of shared and case-specific fields: rec Weapon var name String var bonus Int case MeleeWeapon var damage Int case RangedWeapon var minRange Int var maxRange Int end Here, name and bonus can be accessed on all Weapon instances, but the other fields are case specific. It sort of combines product and sum types into a single construct. I’ve found this to be really handy in practice. I haven’t decided if I’m totally sold on this feature yet. But in the (admittedly small) amount of example code I’ve written using it so far, it seems to feel pretty nice. For a small game scripting language, I think it may strike a decent balance between static safety and simplicity. Update: What about flow typing?#update-what-about-flow-typing When I first posted this, the most common reply was why not do some sort of flow typing? In code like: def attack(weapon Weapon, monster Monster, distance Int) if weapon is RangedWeapon and (distance weapon.minRange or distance > weapon.maxRange) or distance > 1 then print("You are out of range.") return end # ... end The compiler could do control flow analysis to determine that the .minRange and .maxRange calls are guarded by an is RangedWeapon and thus allow them. But if you don’t guard the code with that kind of check, you’d get an error: def attack(weapon Weapon, monster Monster, distance Int) if distance weapon.minRange or # Error! Can't access .minRange here. distance > weapon.maxRange then print("You are out of range.") return end # ... end This is definitely a thing you can do! TypeScript, Kotlin, Flow, Dart, and others all support it. The general technique is called “control flow analysis” and the specific feature is called “flow typing”, “smart casts”, or “type promotion” depending on which language. Is it a good fit for my language? I do like that it makes imperative code “just work” while being safe. But that “just” is doing a lot of heavy lifting. We do this analysis in Dart and it is fantastically complex. Proving that a certain piece of code can only be reached by going through some other piece of code first gets hard quickly in the presence of loops and closures. It seems like every release of Dart, we ship more extensions to flow analysis because users keep expecting it to be smarter and smarter. Also, it isn’t sound in many cases that users expect to work. Once the variable that you’re type testing can escape the current function, the compiler generally can’t prove that it won’t be mutated between when you test its type and when you use it as the more precise type later. Overall, my feeling is that it works out pretty well for Dart, but it’s a large sort of messy feature that feels a little too magical. A goal with my hobby language is that you should be able to have the whole language loaded into your head and rarely be surprised by what it does. Flow analysis in Dart still fairly often surprises me and I literally work on the language full-time. There’s also the question of what you promote the tested variable to. In my language as it currently stands, there is no subtyping. MeleeWeapon isn’t a subtype of Weapon, it’s a case constructor. The weapon is MeleeWeapon syntax looks like a type test, but it’s really more like an enum case check. So after that test, what type would weapon have? It would still have to be Weapon. I guess I could make this work by not promoting the type but by having the type checker track an extra “known case” property for each static type and then use that. That might work. But even with that, I worry that it would quickly become annoying. Let’s say you refactor the above code to: def attack(weapon Weapon, monster Monster, distance Int) if weapon is RangedWeapon and checkRange(weapon, distance) or distance > 1 then print("You are out of range.") return end # ... end def checkRange(weapon Weapon, distance Int) Bool distance weapon.minRange or distance > weapon.maxRange end That no longer works. Inside checkRange() the compiler has lost track that weapon is always a RangedWeapon. You could come up with a way to annotate that, but now we’re back to subtyping and all the complexity it involves. So, overall, yes, subtyping and flow analysis is a thing that could work here, but I’m trying to avoid it because I feel like it’s a bigger lump of complexity than I want to take on. I’d be more inclined to do sum types and destructuring, even though it feels a little weird in an imperative language, then do this kind of complex control flow analysis.

• Type Checking If Expressions
  Jan 03, 2023

  I have this hobby project I’ve been hacking on for several years. It’s a fantasy console, very much inspired by the delightful PICO-8. Like PICO-8, my console has its own built-in scripting language. Because I’m me, I of course took the opportunity to design an entirely new language. My goal for the project is a fun way to build small-ish 2D games. I want its scripting language to be expressive enough to be joyful, but small enough that you can learn the whole language and never need to consult a reference manual after that. My dream is a goofy little pixellated IDE where you can get lost in your own flow state and just make shit without having to periodically hit StackOverflow and then get distracted by the wonders/horrors of the Internet. I don’t know if I’ll ever pull this off or the language will ever see the light of day, but it’s a fun therapeutic thing for me to noodle on. A dynamically typed scripting language#a-dynamically-typed-scripting-language To make a language that fits in your head (or at least my head, whose working space seems to get smaller every year), I needed to jettison as many feature as I could. My experience across a range of hobby and not-so-hobby languages is that static types add roughly an order of magnitude of complexity, so types were one of the first things to go. Like most scripting languages, I made mine dynamically typed. Here’s an example: def onTick() var d = 0 if buttonHeld(2) then d = d - 1 end if buttonHeld(3) then d = d + 1 end if d != 0 then h = h + d else if h > 0 then h = h - 0.5 else if h 0 then h = h + 0.5 end if h -3.0 then h = -3.0 end if h > 3.0 then h = 3.0 end x = x + h if y 200 then v = v + 0.8 end y = y + v if y > 200 then y = 200 v = 0 end if buttonPressed(0) then if y == 200 then playSequence() v = -10.0 end end end Another simplification I made is to eliminate the distinction between statements and expressions. As in Ruby, Kotlin, and most functional languages, everything is an expression. The previous chained if could be written in a more explicitly expression-y style like: h = h + if d != 0 then d else if h > 0 then -0.5 else if h 0 then 0.5 end Unifying statements and expressions means the language doesn’t need a separate if statement and conditional expression. Also, I don’t know, I just like expression-oriented languages. An imperative language#an-imperative-language Even though the language is expression-oriented, it’s not explicitly functional. Functional languages are close to my heart, but this is a game scripting language. A game world is basically a big ball of incrementally updated mutable state. For the kind of programs and user experience I have in mind, I think an imperative, procedural style is easy to learn, and fun to program in. I want users thinking about their game, not, like, monads and persistent data structures. So while everything is an expression in my language, it doesn’t at all shy away from side effects and imperative control flow. Variables are assignable. Fields are settable. There are loops and breaks and early returns. All of those are as natural and idiomatic as they are in C++, JavaScript, C#, or any of the other languages that the majority of the world’s code is written in. Handmade Seattle#handmade-seattle Last fall, I attended the wonderful Handmade Seattle conference. I had a particularly inspiring conversation with Devine Lu Linvega about their tiny uxn VM. They had this idea to build the smallest possible system and programming language for their own use. Then they rebuilt their own personal tools—text editor, music stuff, etc.—using that. Now, UXN is really minimal. I get a certain satisfaction from programming in assembly, but it’s not the language I would want to use for my own joy. But it did make me rethink the scripting language for my fantasy console. I picked dynamic types because that made the language smaller and I figured it would be a good fit for my (entirely hypothetical at this point) users. But is it what I’d want to use to make little 2D videogames? The game I’ve spent the most time hacking on is my also-perennially-incomplete roguelike Hauberk. I’ve rewritten it several times, but every incarnation has been in a statically typed language: C++, C#, Java, and now Dart. My most pleasurable time spent working on Hauberk is when I’m refactoring and the type system guides me to what’s left to clean up. I just really like working with types. (It’s OK if you don’t. As our Burger Sovereign says, have it your way.) After talking to Devine, I realized that if I was making this fantasy console for me personally, its language would be typed. So over the past few weeks, I’ve been sketching out a statically typed variant of my console’s scripting language. I don’t know if it will really come together, but I thought maybe it would be fun to write about the exploration. Type checking if expressions#type-checking-if-expressions I slapped together a new prototype interpreter for my language. (The main implementation is a bytecode VM in C++, which is pretty fast but not exactly easy to hack on.) Then I dutifully started adding a type checking pass to it. One of the first challenges I hit is how to type check if expressions. As the title up there implies, that’s what this post is really about. Because it turns out that having if be an expression while also fully embracing an imperative style gets a little weird when it comes to type checking. I’ll walk through a bunch of examples and build up to the type checking rules I have settled on (so far, at least). We’ll start simple: var love = if isFriday then "in love" else "not in love" end We need a type for the if expression so that we can infer a type for the variable love. In this case, the type is obviously String since both the then and else branches evaluate to strings. So the basic rule we’ll start with is: An if expression’s type is the type of the branches. Different branch types#different-branch-types But what if they don’t have the same type? What about: var love = if isFriday then "in love" else 0 end Here, love could end up being initialized to either a String or an Int. Now what type do we choose? Crystal’s answer is String | Int. Union types are cool but definitely too complex for the language I’m trying to make. In Kotlin, which is also typed and expression-oriented, the answer is, apparently, {Comparable & java.io.Serializable}. Which I have to say does not seem super helpful. I assume that the compiler goes looking for a shared supertype of the two branch types, String and Int. Since String and Int both happen to implement Comparable (and I guess some serialization interface), you get that as the common supertype. In object-oriented languages with subtyping and where the type hierarchy forms a lattice, this common supertype is the least upper bound, and it’s a natural answer to the problem. It shows up in other languages when type-checking conditional ?: expressions and a few other places. It works, but, as we can see in the Kotlin example here, it doesn’t always produce intuitive or useful results. More to the point, one of the other features I jettisoned from my scripting language is subtyping, so LUB is off the table. Without subtyping, every type is disjoint: a value of one type is never a value of any other type too. That means that if the two branches of an if have different types, then there is no possible type I can infer that contains all of their values. The only other response is to make it a type error. That’s the next rule: If the branches have different types, it’s a compile error. Imperative ifs and unused values#imperative-ifs-and-unused-values That rule does work: It’s basically SML’s rule for if expressions. But I want my scripting language to feel familiar to users programming in an imperative style. Consider: var daysNotInLove = 0 if isFriday then print("in love") else daysNotInLove = 1 end Here, the two branches have different types. The then branch has type String because in my language, print() returns its argument. (That makes it handy for stuffing some debug printing in the middle of an expression.) The else branch has type Int because an assignment expression yields the assigned value. According to the previous rule, this is a type error because we don’t know what type of value the if expression evaluates to. But it doesn’t matter since the if’s value isn’t being used anyway. There’s no need for the compiler to yell at you, and code like this turns out to be very common in practice. To address this, the type checker takes some of the surrounding context into account. When an if expression appears in a location where its value won’t be used, then it’s no longer an error for the branches to have different types. How complex is tracking that context? Not too bad, actually. There are a handful of cases: In a block or function body where you have a sequence of expressions, the result is the value of the last expression. The values of all of the preceding expressions are discarded. So in an expression sequence, all but the last expression are in a “value not used” context. Like other expression-oriented languages, functions in my language implicitly return the value that the function body expression evaluates to: def three() Int print("About to return three...") 3 end def onInit() print(three()) # Prints "About to return three..." then "3". end But if a function has no return type (the same as void or unit in other languages), it doesn’t return a value. In that case, even the last expression in the body is a “value not used” context. Loop expressions don’t produce values, so their body is always a “value not used” context. (I’m toying with the idea of allowing break expressions to yield a value from the loop, but they don’t right now.) Whenever an if or match expression is in a “value not used” context, then we push that context into the branches too. Likewise with the right-hand side of and and or logic operators since those are control flow expressions to. That’s it. After I came up with this rule, I did some poking around and it seems like Kotlin does something similar. It frames it by saying that when you use an if “as an expression” then the two branches must have the same type. That’s roughly the distinction I’m making here too: when an if appears in a statement-like position where its value is discarded, then the branches can disagree. Missing else#missing-else This rule allows us to support an even more important flavor of if expressions that are common in imperative code: those without else clauses. In SML and some other functional languages, every if expression must have an else clause because the presumption is that you will be using the value produced by the expression and you need a value even when the condition is false. But in imperative code, it’s obviously common to have ifs whose main purpose is a side effect and where an else clause isn’t needed. In fact, when I analyzed a huge corpus of real-world Dart, I found that only about 20% of if statements had else branches. Now that we understand when an if expression is in a context where it’s value isn’t used, we can allow omitting else branches those. The next rule is: An if expression can omit the else branch when in a context where its value isn’t used. Exiting branches#exiting-branches We’re almost there. It’s starting to feel like we really are type-checking an imperative language, not ML in BASIC’s clothing. I coded this up and successfully wrote some little example programs. It was starting to feel like a real typed language! I could stop here, but there’s one last bit of type checking logic for if expressions. I haven’t decided if it’s worth keeping. Consider: def onInit() var love = if isFriday then "in love" else return end end When isFriday is true, then this initializes love with the string “in love”. When isFriday is false, then the return exits from the function entirely so love never gets initialized at all. So even though these branches don’t evaluate to the same type, love is always initialized with a String. This code should be fine. Or, at least, it should be sound according to the type system. Whether this is good style is definitely open for debate. I could probably not allow code like this. But my default stance is to be as permissive as possible without breaking soundness, and this is a corner where I can be. The trick is that expressions like break, return, and throw are special. While they are expressions grammatically, they don’t actually evaluate to values. If you do: var x = return That x never gets initialized. A return expression always jumps out of the surrounding code instead of producing a value. Languages that have expressions which can do control flow model this by giving these expressions a special type variously called “bottom”, ⊥ (“up tack”), Never, noreturn, etc. This type means “You’re never gonna get a value from me.” When checking the two branches of an if expression, if one branch has that special type (the compiler calls it “unreachable” right now), then we just use the type of the other branch for the if expression’s type. That allows the above example to work. In the sample code I’ve written so far, it rarely comes into play. It’s usually more idiomatic to hoist that control flow out of the if entirely. But we can type check it easily, so the language lets you do it. The rules altogether#the-rules-altogether That’s where I’m at right now. It took me a few iterations to get to a point where all the if expressions I expected to be able to write in my example programs actually type checked correctly but it seems pretty stable now. The rules are: When an if expression is in a context where its value is not used, then there is no restriction on what types the branches can have and we’re done. Otherwise, there must be an else branch and: If both branches have type “unreachable” then the if expression’s type is also “unreachable”. If one branch has type “unreachable” then the if expression’s type is the type of the other branch. Otherwise, the two branches must have the same type and the type of the if is that type.

• Stupid Dog
  Feb 13, 2022

  It’s my 30th birthday, and Megan and I are walking down Duval Street in Key West. There is a pet store here, which must be the most unlikely place for a pet store in all the world. Everyone here is a tourist. What kind of idiot buys a dog while on vacation? You are in the window surrounded by other puppies. A tottering orange ball of fuzz with little triangular ears, like a fox who lost its tail. Ink-black eyes and round button nose. My wife, pregnant and hormonal, is instantly in love with you. We go inside to play with the puppies, because why would you not want to play with cute puppies while on vacation? You curl up in my cupped hands, calm and sleepy. Megan gives me a lingering series of Significant Looks, which I do my best to ignore. I generally try to avoid making dumb decisions, and impulsively buying a puppy while on vacation eight hours away from home is clearly in that category. I make a silent vow to never let Megan know that I also fell in love with you the moment I saw you. It’s the next day and Megan and I are sitting at a Burger King across the street from the pet store. She is presenting her case for why we should buy this very cute half-Pomeranian half-Yorkie puppy even though: We are on vacation. We are expecting our first child in a couple of months and already have more than enough to keep us busy. We have absolutely no plans or intentions on getting a pet. Our house is tiny and barely fits the two of us and two cats, much less a baby and a dog. We are an eight-hour drive from home. You are, in fact, quite expensive. Do I even need to write the numbered list here? It’s obviously a terrible idea. Family members that we’re vacationing with look at us like we are idiots when we say we’re considering it. I think very very hard about my beloved first dog, Snickers, named because she was chocolate brown with swirls of white and caramel. She was a Sheltie, the smallest of her litter, a funny little runt of a puppy who entered my life when I was feeling small and cast out too. When I was seven, I was playing in a neighbor’s front yard while she was out with me. I got distracted and heard a screech of tires behind me. When I turned around, Snickers was on her back motionless in the street. I carried her limp body all the way back to my house. The shocked look on my mother’s face when she saw me arrive, tears streaming down my face, broken dog in my arms, is my most vivid childhood memory. I am the kind of person who analyzes 100 steps ahead. Getting a puppy means buying a ticket for the whole ride, including the end of it, in whatever form it takes. I am choosing to break my own heart. What if it’s not worth all that pain? Megan is driving us north from Miami. You are sleeping in my lap, impossibly warm, cold wet nose pressed against my arm. We talk about names. You almost end up “Dorito” because of your triangular orange ears, but it doesn’t stick. Instead, you become “Ginny”. Not for Ginny Weasely like everyone assumes but short for the liquor genever. When we buy you, I also buy some books on dog training because I am the kind of person who wants to be a very good dog owner that follows all the dog owning rules. Rule #1 is crate training starts immediately. A dog won’t respect you as the pack leader if they are free to sleep where you do. They’ll think they run the pack. It’s late at night by the time we get home, and all the pet stores have closed. Next to the bed, I try to improvise some crate-like thing for your first night involving cardboard boxes, a cooler, and anything else I can find around the house. You yowl in terror, or perhaps just disappointment at the poor accommodations. Eventually, I give up. I pick you up and lay you on my chest. You are asleep instantly, tiny heart fluttering against mine. You are taking your sweet time exploring every single corner of the jungle that is our backyard, looking for lizards. It is three o’clock in the morning. The green and brown anoles are sleeping under leaves where you never find them. I am shambling after you in pajama pants and old boots telepathically trying to get you to pee with every ounce of willpower I have. I am keenly aware of my dwindling opportunities to get a good night’s sleep before the baby arrives. Yet here we are. Over the past few weeks Megan and I have mostly learned to tell the difference between your yowl that means “I am a puppy and don’t like being alone in this crate” and the yowl that means “My bladder is too small to last the night and I need to pee right now.” Sorry about the mistakes we made while figuring that out. I pee in the yard too, hoping you’ll get the hint. I deeply regret ever seeing you in that pet store window. Half-awake, I have vivid fantasies of building a time machine, traveling back in time, and slapping myself in the middle of that Key West Burger King. After patrolling the perimeter of the yard in tiny puppy steps, twice, you finally pee. I tell you what a good girl you are and give you a treat. Because you are an adorable tiny puppy and I am a nerd with a DSLR, I take copious photos of you and upload them to Flickr. One day, I get an email from cuteoverload.com asking if they can share some of them. The next day, you are Internet famous. Millions of people look at your butthole. You are oblivious to your celebrity, as you are to most things in life. As you grow, your puppy fuzz is replaced by silk, still orange on top and white on the bottom like a fox. The whorls of fur behind your ears are the softest thing I have ever felt in my life. I run my fingers through them constantly, to your delight. I take you on walks every day, still trying to be some kind of adequate dog owner. The training manuals say that to teach a puppy walk properly on a leash you should stop walking whenever the dog pulls against it. Then wait patiently until the dog turns and makes eye contact with you. That way, they learn that you are in charge of walking and that pulling means the walk ends. It takes us half an hour to walk a single block. Despite repeating this process hundreds of times, you fail to make the mental connection that if you stop yanking on the leash then we can actually walk where you want to go. You never, ever stop pulling on the damn leash. I start a running joke of whispering insults in your ear that sound like terms of endearment. It’s not like you speak English, so when I say, “I hate your face,” all you hear is the caring tone I use to say it. “You’re so fucking stupid,” I coo into your ear. I rub your fuzzy belly to make sure you know how much I love your stupid face. Five weeks earlier than expected, Megan and I become parents. We haven’t had time to set up a nursery yet so we hastily buy a bassinet. The first night the baby is home from the NICU, we realize there isn’t enough room at the foot of the bed for both the bassinet and the crate. We let you sleep on the bed. You are immensely proud of earning a place on the bed while the new pink hairless dog is relegated to the funny crate. You are listless and frothing at the mouth. Megan and I are terrified. You had been in the yard stalking all of the various animals that live in the barely-tamed-jungle that is Orlando. This time, you found a toad and you’d managed to get your mouth on it before I could get to you. Did it poison you? Are you going to die? We rush you to the emergency vet. She takes your vitals while you sit there limp and apathetic. Your blood sugar is dangerously low. She gives you a treat. Then another. You immediately perk up. You’re fine, totally fine. I remember that a couple of days prior I put my foot down and decided you were ready to graduate from soft puppy food to kibble which is better for your teeth. You were uninterested in the new food. I confidently told Megan, “If she gets hungry enough, she’ll eat. It’s not like she’s going to starve herself.” Apparently, somehow during the thousands of years humans spent turning a wolf into the strange-colored, tiny, eternally cub-like thing that you are, we broke your survival instinct. Sorry about that. We learn to mix some wet food into your kibble. It is something like four in the morning and Megan and I are pushing a baby stroller and a luggage cart across the SeaTac airport with you and the cats crammed in pet carriers perched on top of our suitcases. We’ve just flown 3,000 miles to our new home. Our brilliant idea to take a red-eye flight so the baby would sleep through it failed disastrously. Megan and I spent the entire flight taking turns trying to get the baby to sleep on our laps. You seem to have mostly survived the flight intact. The cats were less thrilled by the experience. We get to the hotel, and I take you out just as the sun starts to come up. The parking lot of an Extended Stay in Redmond is my first view of our new home in the Pacific Northwest. It’s fifty degrees and rainy. I’m beyond tired. You take your sweet time before finding an acceptable spot to pee. We are on PetFinder looking for a second dog. You are just so… weird. Not food motivated. Impossible to train. High strung. Frankly, not very smart. You bark all the time, at everything. We tell ourselves that a second dog will give you company when we’re out. They will teach you how to dog like a normal dog. We find a part-Chihuahua part rat terrier that is, in defiance of everything I understand about genetics, practically your twin. We bring you to meet him at a dog park to see if you get along. You bark a lot. He tolerates you admirably well. For the rest of your lives, you will be his asshole boss. You’ll eat the best parts out of his food bowl and steal his treats. When he tries to sit next to us on the couch, you will wedge yourself in the middle and glare at us both. He will live in fear of your ire which is capricious and often. We name him Benny for “beignet” because he’s golden with a sprinkling of white on top. Maybe they were right about the crate training. We rent a house out in the sticks. There is a green belt behind the yard and we get deer sometimes. Occasionally, we hear an owl hoot. Inexplicably, you are absolutely terrified of the sound. Megan and I hoot at you to tease you and you bark back at us, shivering in fear. Later, we buy a house with a fenced in back yard. You stand proudly in the grass waiting for me to make eye contact with you. The moment I do, you throw your head back in delight and run full tilt to me. Over and over again from Megan to me and back. You wear yourself out and lay in the sun panting “keh keh keh”. When you snuggle with us on the couch later, you still smell like dirt and grass. I am in the basement awkwardly taking photos of myself. I’m speaking at a conference and they asked for a headshot. You wander in and I pick you up and start taking pictures of us together. They are my favorite photos of myself. When I finish my writing my book, I put one of them on the back cover. I get email from readers telling me how delighted they are with the cover. I set that photo as my profile image on all of my various accounts. When ride share drivers pick me up, they joyfully tell me in not-great English that they love my dog. You are still a celebrity. I am in bed staring at the dark ceiling, doing math in my head. Tonight, you are the same age in dog years that I am in real years. From this moment on, you are older than me. I picture you racing ahead of me into the future. I try not to think of where that path leads or when you’ll get there. You are sleeping stretched out, pressed against my thigh, trying to get every square millimeter of contact that you can. I run my fingers through the silky fur on your back, feel your breath rise and fall. We rent a cabin on the Pacific coast. You spend the first half of the drive shaking in excitement and anxiety, paws on the door looking out the window. The whole car smells like your horrid breath and Megan is coated in orange fur. Eventually you wear yourself out and pass out in her lap, head draped over her arm towards me. I rub the fur between your ears while I drive. After settling in, we go explore tide pools. You sniff everything. Every rock, bit of kelp, rotting fish, even the wind. It’s cold and wet, but you don’t mind. When we get back to the cabin, I wipe sand and seaweed from your paws. You fall asleep in front of the fireplace. The vet is explaining to us that you have hyperthyroidism and congestive heart disease. That’s why you pee all the time—often in the house—and why you seem so tense and wound up all the time. Well, that and because you are half Pomeranian. You get put on an expensive pill regimen which you will take for the rest of your life. Since you aren’t interested in food even on a good day, getting you to take them will be a constant battle. At least the peeing inside gets marginally better. The barking does not. I start writing a second book. I work on my laptop on the couch and you lay pressed against one leg the entire time. Once you’ve fallen asleep and let down your jealous guard over access to me, Benny claims the other side. We take another trip out to the Olympic Peninsula with friends. You spend much of it sleeping on the couch in front of the fire, like an overweight self-warming throw pillow. You are given affection from everyone, which you regard as your proper due. In return, you grace us with an endless series of farts that leave the cabin smelling like some sort of reptilian tire fire. It’s six o’clock in the morning. I am barely awake making coffee while you are outside for your morning consitutional. I hear a strange sort of yelp at the back door. I open it just in time to see you collapse. I scoop you up and run into the dining room, yelling out for Megan. I sit on the floor and cradle you, rocking back and forth. If these are your last moments, I want them to be peaceful. I hold your limp body in my arms. Your tongue hangs out, lifeless and blue. I am seven years old again. After an endless minute, you come back to us. Your back stiffens. Your legs poke out like a fainting goat and then relax. Your eyes start darting around and eventually you are able to pick your head up. Ten minutes later and you are back to normal, or at least as normal as you ever are. Honestly, a loss in cognitive function would be hard to detect. The vet begins to use the phrase “congestive heart failure” now. The pill regimen gets more complex. We throw out yet another rug too pee-stained to repair and decide to not have a rug in the dining room “for now”. You are mostly enjoying your retirement. You spend so much time laying on the back of the couch so that you can look out the back window and bark at anyone with the audacity to walk on the sidewalk that the cushion loses its shape entirely. We alternate between trying to give you as much affection as we can in the time we have left and cursing your name as we discover another part of the house ruined by your sneaky peeing. I discover right in the middle of a push-up, my nose an inch from the floor, that you and Benny have surreptitiously saturated every inch of the carpet in the office. Megan and I learn how to put in new flooring. We enter the phase where you take medications to cope with the side effects of other medications. You get over a dozen different pills a day, each on their own specific schedule. I print a chart and put it on the fridge to make sure I don’t miss any. It is exhausting, but mostly for us. You are still relatively content, lazing around the house. Stealing Benny’s food. Barking at people outside the window. Sometimes, at night, you crawl between my legs on top of the blankets, pinning me and forcing me to sleep on my back. I tolerate it as best I can. You’re warm and your ears are still velvet-soft. One morning, I wake up to discover you have leaked little brown droplets all over the blanket. You are sitting up in bed, panting, clearly uncomfortable. (I’m not exactly comfortable with the situation myself.) A couple of days later, you faint again. Perhaps a stroke. Dog.exe has stopped working. When you finish your reboot cycle, you aren’t the same. You don’t recognize Megan and snap at her viciously. You stand stock still in the middle of the dining room floor, eyes glazed. It’s clearly time. I feel an unexpected relief. For years, I have held a deep fear that I wouldn’t know when it was over. That I wasn’t adult enough to handle the responsibility of making the call. I’d procrastinate too long and leave you in misery or choose prematurely and cut your life short. Or I would just be wracked in anguish the whole time. Your last gift to me is a swift decline so that there’s no doubt. I learn there is an entire category of vet-adjacent businesses that specialize in this “transition”. Early on a Tuesday morning, I start calling them to see who can come out today, as soon as possible, right now. The women who answer the phone have well-rehearsed, soothing voices. I stumble over my words. I don’t know the right social protocol to ask, “Can you please come kill my dog?” You are sitting next to me on the couch, pressed warmly against my side like always. You’ve mostly calmed down but are sitting up and tense. A kind woman radiating Pacific Northwest Earth Mother vibes is sitting on the other side of you with a small array of syringes in front of her on the coffee table. We’ve tucked a blanket under you. The kids have already said goodbye to you, tears pouring down their faces. Surprisingly, that turns out to be the hardest part, realizing how much you are part of their stories too. The first shot makes you fall asleep. Literally, not in the “put to sleep” figurative sense. As the woman pushes the plunger, you loll slowly over onto my leg. You’re napping, head on my thigh, more comfortable than I’ve seen you in weeks. Like old times. I run my fingers through the fur on your back, up to where it gets shorter and velvety between your ears, around your side where it gets thinner and silkier, to your chest where it thickens into whorls. I leave my fingers there where I can feel your heart beating. I want to be here for it, to know that I was with you from beginning to end. When the second shot goes in, your music stops. With practiced timing, the woman has your body wrapped up in the blanket and out the door in minutes. The second the door closes, I feel the mask fall off my face and the tears start. It’s later that day. I’ve thrown out the pills, pill poppers, prescriptions, aftercare instructions, wrecked towels and blankets, every reminder of your failing health. I am out for a walk to get out of the house. It hits me a few blocks away. The answer to the question I carried since the day we got you. Will the life we have with you outweigh the pain at the end? Will it be worth it? It is. Every minute. It’s a couple of years later. Your ashes are in a little box in my nightstand. I remember the stress and exhaustion of the last few months of your life, but it feels vague, like I remember the stories I told people about it but not the actual experience. I remember all of the good moments with you vividly. The feel of your fur between my fingers, your body pressed against my side, your horrific breath panting in my face. Megan and I talk about your antics. I still miss you and your stupid face. Ginny 2008 – 2019

• 640 Pages in 15 Months
  Jul 29, 2021

  My book Crafting Interpreters on programming languages is done. OK, OK. I know I said it was done like fifteen months ago. But now it’s really done. And by that I mean, the print, e-book and PDF versions are done. You can buy it. You can hold it in your hands. And I do mean “hands” plural. Because this little “handbook” turned out way larger than I anticipated: This is a proof copy, so it looks a little different than the final design. Look at that thing. 640 pages, eight inches wide, ten inches tall. If you get tired of reading it, it can serve as a doorstop or protect you from small-arms fire. Remember back on Mr. Roger’s Neighborhood when he would take you to a factory and show you how pencils or umbrellas are made? I love that stuff, so I thought maybe you might like to see what I spent the past year on. You can read this as a peek behind the curtain, or maybe a long apology for why it took so long. Where I left off#where-i-left-off What I said in my last post was that Crafting Interpreters was finished, and by that I meant that I’d written all the code, dotted the last sentence, and scanned the last illustration. The content of the book was complete. But it wasn’t a book yet. Just a pile of Markdown and PNG files that my embarrassingly bad Python code would begrudgingly turn into a website. My goal has always been to make a real book, with pages and everything. Completing the content was just one (granted, the most important) step in that journey. Once I finished the last chapter and put it online, I took about a month-long break. I had been writing every single day for almost four years and I was fried. Also, in case you don’t remember, early 2020 was not exactly a Totally Fine Time in World History. A new build system#a-new-build-system After a few weeks, my desire to have this project completely done returned and I eased my way back into working on it. I fixed a whole pile of typos and other mistakes that readers filed bugs for (thanks!). Then, for no real good reason at all, I decided to rewrite the whole build system for the book in Dart. The build script I wrote for my first book was dead simple. Literally a single Python script that took a Markdown file for each book chapter and rendered it to HTML while weaving in the code snippets. The world’s dumbest static site generator. I started with that for Crafting Interpreters but then it grew and grew. My second book includes every single line of code for two complete interpreters, which it builds up incrementally across thirty chapters. I needed to not just build the HTML for the book’s website, but also make sure the code really did work. I gave the build system the ability to not only generate the site for the book, but also to slice and dice the code. Given a chapter, or even a single point within a chapter, it can output a program containing all of the code for the interpreters up to that point. Then I can take that code, compile it, and run it through my automated test suite to make sure the code I’m showing you does what it’s supposed to. Useful, but really straining the limits of how much code I want to maintain in a dynamically typed language like Python, at least, with my (low) level of Python expertise. Also, it was, frankly, really slow. So over a period of a couple of weeks, I rewrote the whole thing in Dart. I work at Google on the Dart language team as my day job, so picking Dart was not an unbiased choice. But it’s my build system for my book, and I know Dart and many of its core libraries and packages like the back of my hand. Hell, I personally wrote the initial version of the canonical markdown package. I found a pretty nice package for mustache templates, so I converted the book’s old Liquid templates to that. I didn’t find a good syntax highlighter. But it’s not like I looked very hard either. It seemed like just the kind of fun thing to implement from scratch, so I whipped one up loosely based on Pygments. The end result is a new build system that generates exactly the HTML and syntax-highlighted code that I want. Also, it is literally ten times faster than the old Python one. As you’ll see, it turned out to be handy that I had better control over the Markdown processing, but at the time I was basically just doing this for fun and to procrastinate the real work. I planned to implement some clever incremental rebuild logic in the dev server, but it builds so fast that I just rebuild everything on every refresh. Once I had the new build system outputting nice clean HTML and I’d deleted the old Python stuff, it was time to get started. Designing the book#designing-the-book Doing a large graphic design project like a book works a lot like web dev or game programming where it sort of has two levels. First you set up your “framework” or “engine”. On the web, this is your web framework and all of your CSS and HTML templates. In games, it’s your game engine. Then you pour content into that structure. With the right framework, adding content is easy. For graphic design using a program like Adobe InDesign, the way it works is you set up styles and masters. A master defines the margins and grid lines for a page. It’s the fences that corral all the text to keep those feral letters from running around eating all the whitespace. Styles are like CSS: they let you take a semantic kind of text or object and associate specific fonts, styles and colors for it. In theory, you get the masters and styles right and then typesetting is pretty easy and mechanical. Now, I did not make my life easy when it comes to book design. Book design is literally a two-dimensional spatial exercise and I made my job harder both horizontally and vertically. If you’ve read any of it on the web, you know my book has: Prose, of course. So many words. A lot of asides that need to be right next to certain pieces of text, code or illustrations that they refer to. Some of these can get pretty long. Plenty of code. Also, each code snippet has a little location blurb next to it telling you where the code goes in the resulting program. How wide?#how-wide Horizontally, the main text column needs to be wide enough to fit the longest line of code. I can use shorter lines and wrap the snippets more, but that makes them harder to read. It also makes them vertically taller, which causes its own problem. Then I need room next to that for the asides, since they often remark directly on specific sentences. I can make those narrower too, of course. But then they wrap and get taller. Some asides are fairly long and if they get too tall, they start colliding with each other or overlapping location snippets. Oh, and since I ended up writing a 200k+ word book, it’s going to have a high page count. That means a thick book. Thick books need wider inner margins so the text doesn’t disappear into the spine. All of this points towards a pretty wide page. Most CS textbooks—at least the ones on my bookshelf—are 7.5 inches wide. I tried hard to come up with a design that fit the code, asides, and healthy margins in that width while still giving a text size that didn’t require a magnifying glass. Eventually, I conceded defeat. Once I tried designing a set of metrics for an 8 inch wide page, everything fell into place. I could have enough breathing room around the text to make it enjoyable to read, a decent length for the code snippets, and plenty of room for the asides. (Using a narrower font for the asides helped too). Here are the final horizontal metrics. How tall?#how-tall That left the other dimension. If I were going with a real publisher doing a full offset print run, I could pick whatever page size I wanted. But since I’m self publishing, that would mean paying up front for thousands of copies to be printed and, I don’t know, storing all the boxes in my garage or something. Print on demand worked great for my last book, and I planned to do the same thing for this one. That meant sticking to the limited set of page sizes that KDP and IngramSpark support. The only reasonable one that is 8 inches wide is 8”×10”, so that’s what I picked. The end result is a book that feels big, but hopefully not awkwardly huge. I’m sorry. I promise to write a smaller book if I ever write another. The metrics for a complete chapter spread. Vertically, text is aligned to a classic 12pt baseline grid. The whole time I was picking these margins and metrics, I was also selecting fonts and building styles. You can’t do graphic design in a vacuum, so I picked a test chapter and just typeset it and tweaked the fonts over and over again. Eventually I got a set of fonts and styles I liked, some masters that seemed workable, and I was ready to go. The framework was done and now it was time to open the content hose. XML, like its 1999#xml-like-its-1999 Of course, problem #1 is that there is no content hose. I have to build one. InDesign doesn’t know what the hell Markdown or my weird ass build system is. I sure as hell didn’t want to, like, copy every chapter from my browser, paste it into InDesign and then manually apply all the styles. I have a high pain tolerance, but I’m not a masochist. What InDesign can do is import XML. Even better, you can set it up to automatically apply certain paragraph or character styles to certain tag names. InDesign’s XML support, alas, has not matured much since the time I described it as being implemented by a narcoleptic intern. For example, in HTML you can italicize a word in a header by taking an italics tag and nesting it inside a header tag. InDesign cannot comprehend such advanced data modeling. It needs a flat series of unnested tags and if you need italics in your header, you damn well better have a unique tag for it. But now I had an ace up my sleeve. Since I had microscopic control over my build system and its Markdown processing, I could write my own custom XML exporter that generated exactly the tags that would make InDesign not cry and avoid as many InDesign XML import bugs as possible. JavaScript, in my InDesign?#javascript-in-my-indesign Even so, XML import only gets you so far. Specifically, it gets you a “story” in InDesign terms: a single continuous narrative of text that fills the main text box and spans multiple pages: What the initial XML import looks like. The pink text is where an illustration goes and the "@" tells me where to anchor the aside. The main story is where the prose and code snippets go, but the asides and the location markers need to be pulled out float off to the side. With my last book, I yanked those out manually. I literally cut each aside from the main text column and pasted it into a new text box. It took forever, and that book was less than half as long as this one with way fewer code snippets. There are 1,133 code snippets in Crafting Interpreters. If I had to manually cut and paste the location markers for each of those, I would lose my mind. Worse, I would make a lot of mistakes, which would just create more work for myself. I’m an engineer so I am morally opposed to error-prone manual effort that can be automated. Did you know that InDesign can be scripted using JavaScript? Well, you do now, which apparently places you in a tiny minority because there are, like, no docs for it out there. You can find a couple of auto-generated references, a few sad cries for help from graphic designers clearly out of their element with no responses, and that’s it. JavaScripting InDesign is a special kind of pain. There is no debugger. There are no stack traces. There aren’t even debug prints. There is literally just alert(), and you can only call it once. And it halts your script. Fortunately, I actually learned JavaScript back when that’s all browsers gave you, so I can hack it. Just like developing for IE6. I managed to cobble together a horrific script that would find all of the asides and location markers, pull them out of the main text flow, and leave them off to the side: Asides and location markers are pulled out of the main flow and into their own text boxes. What I wasn’t able to get the script to do was position the boxes correctly. But InDesign has a thing called “anchors” where you can lock the position of an element relative to another. A couple of carefully crafted Object Styles would even set the horizontal metrics correctly and align the text to the right baseline. All I had to do was anchor each text box and it worked perfectly! Wait, did I say “perfectly”? I meant it worked right about half the time and the other half the time InDesign would inexplicably turn off the fucking borders of nearby code snippets. What it looks like when InDesign hates you. I lost hours of my life to this stupid bug. Eventually, I realized that some completely random subset of location tags would have to be manually positioned because anchoring invariably broke some borders. Editing, again#editing-again All of the above took me a good month or so and then it was time to stop fiddling around and get to work on the actual content. The first thing I did was another editing pass of the entire book, front to back. I had already done three drafts of each chapter as I was writing them, but I wanted to do one more now that it was done so that I could get a better feel for continuity. It turns out I repeated the same dumb jokes a lot. I fixed (most of) those. This took five months. There’s nothing fun to say about this, it was just a chore. Copy editing#copy-editing Next I hired an actual professional copy editor, Kari Somerton, to go through and do the same thing. She was great. Most of the editing world uses Microsoft Word and “Track Changes” to handle the editing process. Like most software engineers, I live and breathe plaintext and Git. That way I can see diffs of the changes, and go back through history. I didn’t want to abandon my workflow so I asked Kari to ramp up on Git and my weird completely bespoke build system. She handled it with aplomb and churned through the book in no time. She found hundreds and hundreds of mistakes. This despite me doing four drafts and readers filing hundreds of issues already. Professional copy editors are worth every penny. Typesetting the whole thing#typesetting-the-whole-thing Once the words were as good as they were gonna get, it was time to get them onto pages. The process went like this: Create a new InDesign file for the next chapter. Export it to XML. Import the XML into InDesign. Run my little JavaScript script to pull out the asides and location markers. Go through and anchor the side bar stuff. Fix up whitespace at the end of pages. The first five steps are a piece of cake. I’d wake up in the morning, brew a cup of coffee, shamble upstairs to the iMac, and get started. I could grind through those steps while half awake and get a chapter done in half an hour or so. It was peaceful. Almost meditative. And then step six. You see, that right there is the hard thing about typesetting a book. And it’s the really hard thing about typesetting this book. Because it turns out there are a bunch of constraints on how content can be fit vertically in a page. Obviously, we can’t slice an illustration in half and put the top half on one page and the bottom half on the next. The asides also really need to fit on one page too, or it gets confusing to keep track of what they refer to. Whenever possible, it’s nice for the code snippets to not get split across pages too. Some of those can be over a dozen lines long. (This is another reason why wider horizontal metrics helped. Because if I had made the code snippets narrower, they would end up taller, which would make them harder to fit on the page.) And you don’t want a header alone at the end of a page with no content after it. And it’s good to avoid widows and orphans… Take all of those rules and restrictions, and mix in the completely fixed height of a page and you got yourself a real constraint solving problem. Or, in my case, 640 of them, all interwoven with each other. Because, you see, InDesign is happy to solve all of this for you by just pushing content to later pages. Code snippet too long? Move it to the next page. No room for prose under the header? Move it all to the next page. What that gives you is a ton of dead white space at the bottom of pages. It looks terrible and wastes space, like this: I thoughtfully left room on the page for your own doodles. Speaking of illustrations…#speaking-of-illustrations At one level, the illustrations were easy. I specifically chose black and white pen and ink because it’s print friendly. When I first scanned the images as I wrote each chapter, I brought them in at glorious 1200 DPI. Here’s a crop: A little level adjustment in Photoshop produces: So detailed! Exporting these to high resolution bitmaps that print well was a snap, and they look great. (Well, as great as my handwriting looks, I guess.) Incorporating the illustrations into the page layout was another story. Halfway through typesetting, a lightbulb went off and I finally realized why most books say, “Refer to Figure 123 to see blah blah blah…” That gives the typesetter freedom to put Figure 123 wherever the hell it fits on any nearby page. In my dumb book, because I am an idiot, the prose just refers directly to the illustration. The illustration needs to be right there or the text doesn’t make sense. I didn’t think about that when each chapter was an infinitely scrolling web page, and by the time I realized, it was too late. With the hundreds of illustrations and thousand-plus code snippets, I had given myself thirty giant interrelated bin-packing exercises. The hard part of typesetting was figuring out how to adjust things to minimize that dead space. Sometimes I’d split a code snippet in two. Maybe add a little extra padding around one image to spread stuff across the page a little. Or crowd another one so that it just fits on the page. Sometimes I’d tweak an illustration to make it shorter to fit on a page or taller to eat up some whitespace. This was the real challenge of typesetting the book and why it took me two months to get through all the chapters. Front matter and back matter#front-matter-and-back-matter Did you know that there are professional indexers? People whose job it is to write indexes for books? They even write books about how to write indexes. (One would presume that these books have truly superb indexes.) I did not hire one of those eminently skilled professionals. Instead, I spent two weeks going through every damn chapter again doing my best to pretend that I know what I’m doing. InDesign’s support for indexes is actually pretty nice. You can basically just select some text and say, “Make an index entry for this.” Then it collects all of those and generates an index for the whole book. But actually adding all of those entries is a mind-numbing chore. The index is the main piece of back matter—the stuff at the end of a book after its main content. There is also front matter. You’ll never guess where that goes. I put together a title page, copyright page, dedication, and acknowledgements. Then I let InDesign generate a table of contents for me. This was a magical moment. At this point, I had a complete book: Here's what the inside looks like. Or, at least, I had the inside of one. Cover design#cover-design A lot of authors obsess over their cover and envision it the whole time they are writing. Despite what the aphorism says, people do judge books by their cover and a good one makes a big difference. At least, that’s the case over in fiction land. In computer science, judging by the other books I have laying around, the artistic merit of the cover appears to be somewhat less critical. I guess when the prof says you have to buy the book to pass the class, a clip art cover is sufficiently compelling. Since I am not a professor who can garner sales by fiat, I spent a lot of time on the cover design. I take photos, so I thought it could look nice to put something detailed on the cover to liven it up. I went through my thousands of photos trying to find something that fit. And, while I have some pretty pictures, none of them felt like they worked as covers. They felt arbitrary. Eventually I realized that the visual language of the book is those pen and ink illustrations. So I drew a bigger more detailed version of the mountain illustration I use as a metaphor for the compilation process. I also hand-lettered a new title: It’s a real typeface (Acumin Pro Extra Condensed) but I hand-traced a printout to give it some imperfect charm. I picked a color palette to try to give it a sort of a mimeographed 1950s scouting manual vibe. Proofreading the proof#proofreading-the-proof Now I really had a book. I uploaded the PDF exports to KDP and ordered a proof copy. A week later, a surprisingly heavy box arrived. This was the first moment I really understood just how big this book I wrote is. Up until this point, it was just data files. But seeing it fill up an Amazon box clarified the scale of the project in a way that the time I spent never quite did. So I had a book, but it still wasn’t done. Because the typesetting process involved a lot of manual labor. To err is human, so now I had to proofread—to literally go through the proof and read it looking for mistakes. I marked them all with sticky notes: I put an "x" on each sticky note to mark when I had applied the fix to the InDesign files. Here’s where it got stressful. If you’re a programmer, then source control and diff is deeply ingrained in your workflow. Whenever I make a change, I take for granted that I can then see a diff in the commit to verify that I changed only what I intended and nothing else. I did put the InDesign files in a Git repo, but they are giant opaque binary files. Also, InDesign has a habit of changing them even when it doesn’t seem like I’ve actually made any real changes. There’s nothing quite like syncing all the styles across the chapters, seeing every single file marked changed and wondering, “Did I just accidentally move every bullet list item 3 points to the left?” It felt like flying blind at exactly the stage in the process where I really wanted to see exactly what was happening. Did I engineer my way out of this problem? You’re damn right I did. I wrote a Dart script that would take a PDF of the book, extract every page, and then generate a single huge PNG file with every page tiled across it. It looks like this: If you zoom in, each page is about this big: If you're on a retina display, this is a 100% zoom. Every time I changed the InDesign files and committed them, I exported a PDF for that commit and generated a tile image. Then I wrote a little Photoshop action that would take two of those and draw a big red border around any pixels that differed. Here is what all of the proofreading changes look like: And zoomed in: There isn’t enough detail in the tile image to tell exactly what changed, but the red tells me which pages need a visual inspection. I know I should be proud of, like, writing an entire textbook on programming languages. But, honestly, I think I’m most proud of this dumb little script. It was such a relief to be able to programmatically verify that, yes, this PDF looks exactly like the last one except for the one expected change. Ebooks are also books#ebooks-are-also-books Once I incorporated all the proofreading fixes, the print edition was really for real totally done. But people also like reading ebooks and Kindle, so I needed to make those too. Again, writing my own build system helped. I tweaked it to be able to export the antiquated XHTML that EPUB requires as well as all the weird metadata and manifest stuff that goes into one. A few command-line invocations later and I had Kindle and EPUB e-books. I tested in a bunch of readers and tweaked the CCS to try to find a compromise between all of their renderers. Updating the site and launching#updating-the-site-and-launching At this point—and now we’re talking this past weekend as I write this sentence—I had a folder with all of the final files for every edition of the book. All that remained was to update the front page of the book’s website to point to where people can get them. I took some photos and put on my web designer hat for a while. I tried to make it tolerably responsive. Then I wrote this blog post. It’s weird to put that in the past tense since I just wrote it. But I guess it’s in the past now. Tomorrow, I’m going to upload the files to the various sites and stores. I’ll update the site to have links to all of those as they go live. Once the stores have processed the uploads and everything is available, I’ll publish this post and update the site. I’ll write a note for the mailing list and feel nervous emailing that many people. If you’re reading this, I already did all that and the book is really, really done. What next?#what-next Ever since I finished the last chapter, people have been asking me what I’m going to do next. I have people I consider close friends now who have never known me when I wasn’t writing this book. Many ask what I’m going to write next, or suggest a topic. I interpret this as a compliment—they like my writing and want more. But it also feels like asking a mom in labor whether she plans to have more kids. I’ve had my legs in the stirrups for six years, so I’m gonna just relax a bit after all the pushing is over before I even think about another book baby. What I really plan to do is… not plan to do anything. This writing project has been an enormous exercise in self-applied delayed gratification. On top of that, the pandemic brought its own basket of deferred dreams. I’ve gotten so skilled at denying myself and postponing, so good at tuning out what I feel like doing, that it’s hard to even hear those feelings any more. I don’t remember what my own joy sounds like. So I’m gonna check out for a while and go hunting for my pleasure centers. Maybe make some more music. Maybe go fishing. Probably spend more time with friends and family (safely). I might get back to working on my roguelike. Or maybe I won’t do any of those. Maybe I’ll just go out in the backyard and bask mindlessly in the sun like a lizard. The important part is I won’t decide until I feel like it. I’m sure eventually I’ll get the itch to work on something more sizeable again. (But, God-willing, I will never spend six years on a single project again in my life.) Until then, I hope you enjoy the book. Despite all the work, I had a lot of fun making it, and it was always a joy to hear from readers who liked it. It’s the only thing that kept me going this long and enabled me to get it done.

Alex Edwards

• How to manage configuration settings in Go web applications
  Jun 30, 2025

  When I'm building a web application in Go, I prefer to use command-line flags to pass configuration settings to the application at runtime. But sometimes, the client I'm working with wants to use environment variables to store configuration settings, or the nature of the project means that storing settings in a TOML, YAML or JSON file is a better fit. And of course that's OK — it makes sense to be flexible and vary how configuration is managed based on the specific needs of a project and/or client. So, in this tutorial, I want to share the patterns that I use for parsing configuration settings — whether they come from flags, environment variables or files — and explain how I pass the settings onwards to where they are needed in the rest of the web application code. I'll also end with a short discussion about the relative pros and cons of the different approaches. It's a fairly detailed post, so here are the shortcut links for quick reference: Using command-line flags Using environment variables Using configuration files Passing settings to where they are needed Discussion Example code To illustrate the patterns in the rest of this tutorial, let's pretend that we have a web application where we want to configure the following five settings: Setting Type Description port int The port number the web application listens on verboseLogging bool Enables detailed request and error logging requestTimeout time.Duration Maximum duration to wait for a request to complete basicAuthUsername string Username required for HTTP Basic Authentication basicAuthPassword string Password required for HTTP Basic Authentication Regardless of where the configuration settings are coming from (flags, environment variables or a file), I'm quite strict about keeping all the code related to configuration settings isolated in one place, and reading in the configuration setting values right at the start of the program, before doing almost anything else. Most of the time, I prefer to store all the configuration setting values in a single config struct, like so: type config struct { port int verboseLogging bool requestTimeout time.Duration basicAuth struct { username string password string } } I like this because it feels very clear — all the configuration settings are contained in a single struct, along with their appropriate Go type, and you can easily see at a glance what configuration settings the application expects and supports. Using command-line flags As I mentioned at the start of this tutorial, using command-line flags with the standard library flag package is my preferred approach to managing configuration settings. With this approach, you explicitly pass the configuration values as part of the command when running the program. For example: $ go run main.go -port=9999 -verbose-logging=true -request-timeout=10s -basic-auth-username=admin -basic-auth-password="secr3tPa55word" In your Go code, you define a specific command-line flag using syntax like this: flag.IntVar(&cfg.port, "port", 4000, "The port number the web application listens on")` In this example code, we define a command-line flag named port that accepts an integer value and stores it at the location pointed to by the &cfg.port pointer. It will have a default value of 4000 if no corresponding -port flag is provided when starting the application, and the final parameter is a description that will be displayed when a user runs the program with the -help flag. Importantly, after you've defined all the command-line flags for your application, you need to call the flag.Parse() function to actually read in the values from the command-line arguments. Let's put this together in a very simple application that reads the command-line flag values into a config struct, and then prints them out. File: main.go package main import ( "flag" "fmt" "time" ) // The config struct holds all configuration settings for the application. type config struct { port int verboseLogging bool requestTimeout time.Duration basicAuth struct { username string password string } } func main() { // Create a new config instance. var cfg config // Define the command-line flags. Notice that we define these so that the values // are read directly into the appropriate config struct field, and set sensible default // values for each of them. flag.IntVar(&cfg.port, "port", 4000, "The port number the web application listens on") flag.BoolVar(&cfg.verboseLogging, "verbose-logging", false, "Enables detailed request and error logging") flag.DurationVar(&cfg.requestTimeout, "request-timeout", 5*time.Second, "Maximum duration to wait for a request to complete") flag.StringVar(&cfg.basicAuth.username, "basic-auth-username", "", "Username required for HTTP Basic Authentication") flag.StringVar(&cfg.basicAuth.password, "basic-auth-password", "", "Password required for HTTP Basic Authentication") // Parse the flags with the flag.Parse function. This is important! flag.Parse() // Print all configuration settings. fmt.Printf("Port: %d\n", cfg.port) fmt.Printf("Verbose Logging: %t\n", cfg.verboseLogging) fmt.Printf("Request Timeout: %v\n", cfg.requestTimeout) fmt.Printf("Basic Auth Username: %s\n", cfg.basicAuth.username) fmt.Printf("Basic Auth Password: %s\n", cfg.basicAuth.password) } If you're following along, go ahead and run the application with your own values in the command-line flags. You should see the same values printed out by the application, like so: $ go run main.go -port=9999 -verbose-logging=true -request-timeout=30s -basic-auth-username=admin -basic-auth-password="secr3tPa55word" Port: 9999 Verbose Logging: true Request Timeout: 30s Basic Auth Username: admin Basic Auth Password: secr3tPa55word If you don't provide a value for a specific flag, the application will revert to using the default value you specified. For example, if you don't provide a -port flag it will default to the value of 4000, like so: $ go run main.go -basic-auth-username=admin -basic-auth-password="secr3tPa55word" Port: 4000 Verbose Logging: false Request Timeout: 5s Basic Auth Username: admin Basic Auth Password: secr3tPa55word Help text One of the great things about the standard library flag package is the support for automatic help text. If you run your application with the flag -help, it will list all the available flags for the application, along with their accompanying help text and default values if appropriate. Like so: $ go run main.go -help Usage of /tmp/go-build2103583960/b001/exe/main: -basic-auth-password string Password required for HTTP Basic Authentication -basic-auth-username string Username required for HTTP Basic Authentication -port int The port number the web application listens on (default 4000) -request-timeout duration Maximum duration to wait for a request to complete (default 5s) -verbose-logging Enables detailed request and error logging Boolean flags For boolean flags, if you want to pass a value of true you can simply include the flag name without assigning a value. The following two commands are equivalent: $ go run main.go -verbose-logging=true $ go run main.go -verbose-logging In contrast, you always need to use -flag=false if you want to set a boolean flag value to false. Dashes You can use one or two dashes in front of a flag name, both work identically. The standard library flag package does not support 'short' flags, and the number of dashes has no effect on the behavior or any special meaning. So it's just a matter of personal taste which you use. The following two commands are equivalent: $ go run main.go -verbose-logging -request-timeout=30s $ go run main.go --verbose-logging --request-timeout=30s Invalid flags If you try to pass an invalid value as a command-line flag, the application will automatically exit with an error message and the help text for reference. For example, if you try to pass a non-integer value in the -port flag, the parsing would fail and the output would look like this: $ go run main.go -port=foobar invalid value "foobar" for flag -port: parse error Usage of /tmp/go-build2103583960/b001/exe/main: -basic-auth-password string Password required for HTTP Basic Authentication -basic-auth-username string Username required for HTTP Basic Authentication -port int The port number the web application listens on (default 4000) -request-timeout duration Maximum duration to wait for a request to complete (default 5s) -verbose-logging Enables detailed request and error logging exit status 2 Similarly, if you try to use a flag that as not been defined, the application will automatically exit with an error message and the help text. For example: $ go run main.go -foobar=baz flag provided but not defined: -foobar ...etc Custom flag types The flag package provides functions for reading command-line flag values into the following Go types: bool, int, int64, uint, uint64, float64, string and time.Duration. If you want to parse a command-line flag value into another Go type (such as time.Time or []string), you have a few different options. The simplest approach is to use the flag.Func() function, which I've written about here. Or you can also make your own custom type that implements the flag.Value or encoding.TextUnmarshaler interfaces, and define the flag using either the flag.Var() or flag.TextVar() functions respectively. I've shared a gist demonstrating how to do this here. Alternatively, there are third-party packages (such as spf13/viper) that you can use, which automatically support parsing command-line flags into a wider range of Go types. Personally, I've never felt it necessary to use these, but YMMV. Flagsets Lastly, if you want you can create flagsets, which act like a 'container' for a distinct set of command-line flags. It's rare that I need to use flagsets in a web application, but I do often use them when building CLI applications with multiple subcommands. There's a good tutorial about how to use flagsets here. Using environment variables First, I'll start by saying that you can use environment variables in conjunction with command-line flags if you want. Simply set your environment variables as normal, and use them in the command when starting your application. Like so: $ export VERBOSE_LOGGING="true" $ export REQUEST_TIMEOUT="30s" $ go run main.go -verbose-logging=$VERBOSE_LOGGING -request-timeout=$REQUEST_TIMEOUT But if you don't want to do this, you can read the values from environment variables directly into your Go code using the os.Getenv() function. This will return the value of the environment variable as a string, or the empty string "" if the environment variable doesn't exist. You can also use the os.LookupEnv() function to check whether a specific environment variable exists or not. To help read values from environment variables, I like to create an internal/env package containing some helper functions that convert the environment variable string to the appropriate Go type, and optionally set a default value for if the environment variable doesn't exist (just like command-line flags). For example: File: internal/env/env.go package env import ( "fmt" "os" "strconv" "time" ) func GetInt(key string, defaultValue int) int { value, exists := os.LookupEnv(key) if !exists { return defaultValue } intValue, err := strconv.Atoi(value) if err != nil { panic(fmt.Errorf("environment variable %s=%q cannot be converted to an int", key, value)) } return intValue } func GetBool(key string, defaultValue bool) bool { value, exists := os.LookupEnv(key) if !exists { return defaultValue } boolValue, err := strconv.ParseBool(value) if err != nil { panic(fmt.Errorf("environment variable %s=%q cannot be converted to a bool", key, value)) } return boolValue } func GetDuration(key string, defaultValue time.Duration) time.Duration { value, exists := os.LookupEnv(key) if !exists { return defaultValue } durationValue, err := time.ParseDuration(value) if err != nil { panic(fmt.Errorf("environment variable %s=%q cannot be converted to a time.Duration", key, value)) } return durationValue } func GetString(key string, defaultValue string) string { value, exists := os.LookupEnv(key) if !exists { return defaultValue } return value } In some projects, I use a twist on these helper functions and panic if a specific environment variable isn't set, rather than returning a default value. For example: func MustGetInt(key string) int { value, exists := os.LookupEnv(key) if !exists { panic(fmt.Errorf("environment variable %s must be set", key)) } intValue, err := strconv.Atoi(value) if err != nil { panic(fmt.Errorf("environment variable %s=%q cannot be converted to an int", key, value)) } return intValue } Note: If you're looking at this code and thinking that it is bad practice to call panic() rather than returning an error, you'd be right. But in the context where these helpers are used, it seems a reasonable thing to do. If our application can't load the configuration settings that it needs to operate on startup, it can't reasonably continue, and depending on what has failed to load it may not even be safe or sensible to execute any further code. Terminating the application by panicking doesn't seem inappropriate in this scenario. I've written more about this in the post When is it OK to panic in Go?. Using those helper functions in your application then looks a bit like this: File: main.go package main import ( "fmt" "time" "your-project/internal/env" ) type config struct { port int verboseLogging bool requestTimeout time.Duration basicAuth struct { username string password string } } func main() { var cfg config cfg.port = env.GetInt("PORT", 4000) cfg.verboseLogging = env.GetBool("VERBOSE_LOGGING", false) cfg.requestTimeout = env.GetDuration("REQUEST_TIMEOUT", 5*time.Second) cfg.basicAuth.username = env.GetString("BASIC_AUTH_USERNAME", "") cfg.basicAuth.password = env.GetString("BASIC_AUTH_PASSWORD", "") fmt.Printf("Port: %d\n", cfg.port) fmt.Printf("Verbose Logging: %t\n", cfg.verboseLogging) fmt.Printf("Request Timeout: %v\n", cfg.requestTimeout) fmt.Printf("Basic Auth Username: %s\n", cfg.basicAuth.username) fmt.Printf("Basic Auth Password: %s\n", cfg.basicAuth.password) } If you'd like to try this out, go ahead and add the necessary environment variables to your /etc/environment or ~/.profile files, or export them in your shell, and try running the application again. You should see the configuration settings reflected in the output, or any default values for ones that you didn't set. $ export PORT="9999" $ export VERBOSE_LOGGING="false" $ export BASIC_AUTH_USERNAME="admin" $ export BASIC_AUTH_PASSWORD="secr3tPa55word" $ go run main.go Port: 9999 Verbose Logging: false Request Timeout: 5s Basic Auth Username: admin Basic Auth Password: secr3tPa55word Using .env files If you're working on multiple projects on the same development machine (and not using separate containers for each project), it can become awkward to manage environment variables and avoid clashes across the projects. Rather than setting environment variables in /etc/environment or ~/.profile, a fairly common workaround is to create an .env file in your project containing the environment variables, like so: File: .env export PORT=5000 export VERBOSE_LOGGING=true export REQUEST_TIMEOUT=10s export BASIC_AUTH_USERNAME=admin export BASIC_AUTH_PASSWORD=secr3tPa55word Then you can source the .env file to export the variables in the current terminal session and run your Go application: $ source .env $ go run main.go Port: 5000 Verbose Logging: true Request Timeout: 10s Basic Auth Username: admin Basic Auth Password: secr3tPa55word Alternatively, if you don't want to keep running the source command, you can use the joho/godotenv package to automatically load the values from the .env file into the environment when your application starts up. Using configuration files The third option that I sometimes use is configuration files, which store all the settings in a single file on-disk. I normally only use these in projects where there are a lot of configuration settings, and loading them all via command-line flags would be onerous and error-prone. Or also, if the configuration settings are complex, with a deeply nested 'structure' to them. There are a lot of different formats that you can use for configuration files, such as TOML or YAML — or even JSON. They all have different advantages and disadvantages, and you'll be hard-pressed to find one that everybody agrees is 'perfect'. But whatever format you choose, there is probably a Go package that you can use to automatically parse values from the file into a config struct for you. For example, let's say that you want to use TOML and have a configuration file that looks like this: File: config.toml # Server configuration port = 4000 verbose_logging = true request_timeout = "10s" # Basic authentication settings [basic_auth] username = "admin" password = "secr3tPa55word" You can use the BurntSushi/toml package to read the file and unpack the contents to a config struct like so: File: main.go package main import ( "fmt" "log" "time" "github.com/BurntSushi/toml" ) // Make sure the struct fields are exported, so that the BurntSushi/toml package // can write to them, and use struct tags to map the TOML key/value pairs to the // appropriate struct field. type config struct { Port int `toml:"port"` VerboseLogging bool `toml:"verbose_logging"` RequestTimeout time.Duration `toml:"request_timeout"` BasicAuth struct { Username string `toml:"username"` Password string `toml:"password"` } `toml:"basic_auth"` } func main() { var cfg config // Load configuration settings from the config.toml file. metadata, err := toml.DecodeFile("config.toml", &cfg) if err != nil { log.Fatalf("error loading configuration: %v", err) } // Check for any undecoded keys in the config.toml file. if len(metadata.Undecoded()) > 0 { log.Fatalf("unknown configuration keys: %v", metadata.Undecoded()) } fmt.Printf("Port: %d\n", cfg.Port) fmt.Printf("Verbose Logging: %t\n", cfg.VerboseLogging) fmt.Printf("Request Timeout: %v\n", cfg.RequestTimeout) fmt.Printf("Basic Auth Username: %s\n", cfg.BasicAuth.Username) fmt.Printf("Basic Auth Password: %s\n", cfg.BasicAuth.Password) } Notice that in this code we're making use of the metadata returned by the toml.DecodeFile() function to check if any settings were not decoded successfully — which should help to catch typos or invalid keys in the TOML file. Passing settings to where they are needed Getting the configuration settings into the config struct, wherever they come from, is the first half of the puzzle. The second part is getting those settings to where you need them in your Go code. There are many different ways to approach this, and no single 'right' way. For small or medium sized web applications, I often use a pattern of creating an application struct which contains all the dependencies that my HTTP handlers need, and I implement the handlers as methods on the application struct. To make the configuration settings available to the HTTP handlers, I simply include the config struct as a field in application. For example: File: main.go package main import ( "flag" "fmt" "log/slog" "net/http" "os" "time" ) type config struct { port int verboseLogging bool requestTimeout time.Duration basicAuth struct { username string password string } } // The application struct contains the dependencies for the handlers, including // the config struct type application struct { config config logger *slog.Logger } func main() { logger := slog.New(slog.NewTextHandler(os.Stdout, nil)) var cfg config flag.IntVar(&cfg.port, "port", 4000, "The port number the web application listens on") flag.BoolVar(&cfg.verboseLogging, "verbose-logging", false, "Enables detailed request and error logging") flag.DurationVar(&cfg.requestTimeout, "request-timeout", 5*time.Second, "Maximum duration to wait for a request to complete") flag.StringVar(&cfg.basicAuth.username, "basic-auth-username", "", "Username required for HTTP Basic Authentication") flag.StringVar(&cfg.basicAuth.password, "basic-auth-password", "", "Password required for HTTP Basic Authentication") flag.Parse() app := &application{ config: cfg, logger: logger, } mux := http.NewServeMux() mux.HandleFunc("/", app.home) // Use the port configuration setting logger.Info("starting server", "port", cfg.port) err := http.ListenAndServe(fmt.Sprintf(":%d", cfg.port), mux) if err != nil { logger.Error(err.Error()) os.Exit(1) } } func (app *application) home(w http.ResponseWriter, r *http.Request) { // Use the verboseLogging configuration setting if app.config.verboseLogging { app.logger.Info("handling request", "method", r.Method, "path", r.URL.Path) } fmt.Fprintf(w, "Hello!") } If you run this application with the -verbose-logging flag, and make a HTTP request to localhost:4000, you should see the details of the request in the log output, similar to below — demonstrating that the config setting is correctly available to the handler. $ go run main.go -verbose-logging time=2025-06-27T14:15:40.230+02:00 level=INFO msg="starting server" port=4000 time=2025-06-27T14:15:48.705+02:00 level=INFO msg="handling request" method=GET path=/ In larger applications where I want to define my handlers outside of package main, or pass the config struct to functions in other packages, I normally define an exported Config struct in an internal/config package, and pass this around as necessary. For example, let's say that you have a project structure like so: ├── go.mod ├── go.sum ├── main.go └── internal ├── config │ └── config.go └── handlers └── home.go Then the contents of those .go files would look something like this: File: internal/config/config.go package config import "time" type Config struct { Port int VerboseLogging bool RequestTimeout time.Duration BasicAuth struct { Username string Password string } } File: internal/handlers/home.go package handlers import ( "fmt" "log/slog" "net/http" "your-project/internal/config" ) func Home(cfg config.Config, logger *slog.Logger) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { if cfg.VerboseLogging { logger.Info("handling request", "method", r.Method, "path", r.URL.Path) } fmt.Fprintf(w, "Hello!") } } File: main.go package main import ( "flag" "fmt" "log/slog" "net/http" "os" "time" "your-project/internal/config" "your-project/internal/handlers" ) func main() { logger := slog.New(slog.NewTextHandler(os.Stdout, nil)) var cfg config.Config flag.IntVar(&cfg.Port, "port", 4000, "The port number the web application listens on") flag.BoolVar(&cfg.VerboseLogging, "verbose-logging", false, "Enables detailed request and error logging") flag.DurationVar(&cfg.RequestTimeout, "request-timeout", 5*time.Second, "Maximum duration to wait for a request to complete") flag.StringVar(&cfg.BasicAuth.Username, "basic-auth-username", "", "Username required for HTTP Basic Authentication") flag.StringVar(&cfg.BasicAuth.Password, "basic-auth-password", "", "Password required for HTTP Basic Authentication") flag.Parse() mux := http.NewServeMux() mux.HandleFunc("/", handlers.Home(cfg, logger)) // Use the port configuration setting logger.Info("starting server", "port", cfg.Port) err := http.ListenAndServe(fmt.Sprintf(":%d", cfg.Port), mux) if err != nil { logger.Error(err.Error()) os.Exit(1) } } Obviously I'm using command-line flags in these examples, but the same patterns work for environment variables or config files too — once the config struct is loaded with the data, it doesn't matter where it originally came from and the code patterns are the same. Discussion If you've been in the web development world for a long time and buy into the 12-factor app principles (which I generally do), you might think that the correct approach is "just use environment variables". But over the years I've come to the conclusion that they have some drawbacks: I've been bitten more times than I want by bugs that were ultimately a result of an unset or unexpected value in an environment variable — and I think that part of the problem here is that environment variables aren't readily and easily observable in the same way that the values in command-line flags or a configuration file are. If you're working on multiple projects on the same development machine (rather than working in separate containers for each project), you have to manage the lack of natural isolation between environment variables... you need to make sure that there aren't any naming clashes, and that (for example) application A isn't accidentally using the DB_PASSWORD setting intended for application B. I've also seen a lot of Go codebases where configuration settings are read in using os.Getenv() at the point in the code where they are needed. This makes discoverability difficult — it's hard to look at an application's code and easily see what the expected configuration settings are. You can mitigate these issues with some of the techniques that we've discussed in this tutorial. If you're strict about reading all the settings into a single config struct at application startup, that addresses the discoverability problem. If you create helpers like env.MustGetInt() which panic if an environment variable isn't set, that helps to eliminate bugs that exist due to missing environment variables. And you can work around some of the environment variable isolation problems in development by using a .env file — but at that point, it might be worth considering whether a configuration file might be more appropriate. One of the big reasons that I like to use command-line flags is that you get a lot of stuff for free. You get automatic -help text, automatic type conversions, the ability to set defaults, and it handles invalid inputs and undefined flags nicely. Also, it's always very clear what configuration values are being used — you either explicitly pass the values when starting the application, or the default values hardcoded into your Go codebase are used. On top of that, most other gophers will be familiar with the flag package and you don't need any third-party dependencies. When I'm using command-line flags, I typically set the default values to things that are appropriate for a development environment. This is mainly so I don't have to keep typing long commands to run the application when actively developing it. In terms of application secrets, like I mentioned earlier, there's nothing stopping you from storing a specific secret in an environment variable and using it in conjunction with a command-line flag if you want. For example, if you store a password for your database user in a DB_PASSWORD environment variable, you can include it as a command-line flag value when starting the application like so: $ go run main.go -db-user=web -db-password=$DB_PASSWORD Or, although it is a bit more 'magical', you could even use the environment variable as the default value: flag.StringVar(&cfg.db.password, "db-password", os.Getenv("DB_PASSWORD"), "Database user password") So, for all these reasons, I tend to prefer using command-line flags for configuration. The big exception to this is when there are a lot of configuration settings, and it would be awkward to pass them all via command-line flags, or the settings have a deeply nested 'structure' to them. In these cases, I think it can be more practical and maintainable to store the settings in a TOML or JSON configuration file, and load them on application startup like we demonstrated earlier.

• Organize your Go middleware without dependencies
  Apr 26, 2025

  For many years, I've used third-party packages to help organize and manage middleware in my Go web applications. In small projects, I often used alice to create middleware 'chains' that I could reuse across multiple routes. And for larger applications, with lots of middleware and routes, I typically used a router like chi or flow to create nested route 'groups' with per-group middleware. But since Go 1.22 introduced the new pattern matching functionality for http.ServeMux, where possible I've tried to drop third-party dependencies from my routing logic and shift to using just the standard library. But going all-in on the standard library leaves a good question: how should we organize and manage middleware without using any third-party packages? Note: If you're not familiar with middleware in Go, I recommend reading this introduction to making and using middleware before continuing. Why is managing middleware a problem? If you have an application with only a few routes and middleware functions, the simplest thing to do is to wrap your handler functions with the necessary middleware on a route-by-route basis. A bit like this: // No middleware on this route. mux.Handle("GET /static/", http.FileServerFS(ui.Files)) // Both these routes use the requestID and logRequest middleware. mux.Handle("GET /", requestID(logRequest(http.HandlerFunc(home)))) mux.Handle("GET /article/{id}", requestID(logRequest(http.HandlerFunc(showArticle)))) // This route has the additional authenticateUser and requireAdminUser middleware. mux.Handle("GET /admin", requestID(logRequest(authenticateUser(requireAdminUser(http.HandlerFunc(showAdminDashboard)))))) This works, and requires no external dependencies, but you can probably imagine the downsides as the number of routes grows: There's repetition in the route declarations. It's a bit difficult to read and see which routes are using the same middleware at a glance. It feels error-prone — in a large application if you need to add, remove or reorder middleware across many routes it could be easy to miss out one of the routes and not spot the mistake. An alternative to alice As I briefly mentioned above, the alice package allows you to declare and reuse 'chains' of middleware. We could rewrite the example code above to use alice like so: mux := http.NewServeMux() // Create a base middleware chain. baseChain := alice.New(requestID, logRequest) // Extend the base chain with auth middleware for admin-only routes. adminChain := baseChain.Append(authenticateUser, requireAdminUser) // No middleware on this route. mux.Handle("GET /static/", http.FileServerFS(ui.Files)) // Public routes using the base middleware. mux.Handle("GET /", baseChain.ThenFunc(home)) mux.Handle("GET /article/{id}", baseChain.ThenFunc(showArticle)) // Admin routes with the additional auth middleware. mux.Handle("GET /admin", adminChain.ThenFunc(showAdminDashboard)) To me, this code feels quite a lot cleaner, and it largely mitigates the three problems that we talked about above. But if you don't want to introduce alice as a dependency, it's possible to leverage the slices.Backward function introduced in Go 1.23 and create your own chain type in a few simple lines of code: type chain []func(http.Handler) http.Handler func (c chain) thenFunc(h http.HandlerFunc) http.Handler { return c.then(h) } func (c chain) then(h http.Handler) http.Handler { for _, mw := range slices.Backward(c) { h = mw(h) } return h } You can then use this chain type in your route declarations like so: mux := http.NewServeMux() // Create a base middleware chain. baseChain := chain{requestID, logRequest} // Extend the base chain with auth middleware for admin-only routes. adminChain := append(baseChain, authenticateUser, requireAdminUser) mux.Handle("GET /static/", http.FileServerFS(ui.Files)) mux.Handle("GET /", baseChain.thenFunc(home)) mux.Handle("GET /article/{id}", baseChain.thenFunc(showArticle)) mux.Handle("GET /admin", adminChain.thenFunc(showAdminDashboard)) The syntax in this code isn't exactly the same as using alice, but it's pretty close, and in terms of behavior it's functionally the same. If you're interested in using this approach in your own codebase, I've made tests for the chain type available in this gist. An alternative to chi and similar routers In large applications, when I have lots-of-different-middleware being used on lots-of-different-routes, I've always found the route grouping functionality provided by routers like chi and flow to be a huge help. They basically allow you to create route groups with specific middleware, and these groups can be nested, with child groups 'inheriting' and extending the middleware of their parent groups. Let's take a look at an example using chi, which I think was the first router to support this style of route grouping functionality. r := chi.NewRouter() // No middleware on this route. r.Method("GET", "/static/", http.FileServerFS(ui.Files)) // Create a route group. r.Group(func(r chi.Router) { // Add the middleware for the group. r.Use(requestID) r.Use(logRequest) // The routes declared in the group will use this middleware. r.Get("/", home) r.Get("/article/{id}", showArticle) // Create a nested route group. Any routes in this group will use the // middleware declared in the group *and* the parent groups. r.Group(func(r chi.Router) { r.Use(authenticateUser) r.Use(requireAdminUser) r.Get("/admin", showAdminDashboard) }) }) But if you want to stick with the standard library, it doesn't take much to create your own router implementation that wraps http.ServeMux and supports middleware groups in a similar style: type Router struct { chain []func(http.Handler) http.Handler *http.ServeMux } func NewRouter() *Router { return &Router{ServeMux: http.NewServeMux()} } func (r *Router) Use(mw ...func(http.Handler) http.Handler) { r.chain = append(r.chain, mw...) } func (r *Router) Group(fn func(r *Router)) { subRouter := &Router{chain: slices.Clone(r.chain), ServeMux: r.ServeMux} fn(subRouter) } func (r *Router) HandleFunc(pattern string, h http.HandlerFunc) { r.Handle(pattern, h) } func (r *Router) Handle(pattern string, h http.Handler) { for _, mw := range slices.Backward(r.chain) { h = mw(h) } r.ServeMux.Handle(pattern, h) } And then you can use the Router type in your code like so: r := NewRouter() r.Handle("GET /static/", http.FileServerFS(ui.Files)) r.Group(func(r *Router) { r.Use(requestID) r.Use(logRequest) r.HandleFunc("GET /", home) r.HandleFunc("GET /article/{id}", showArticle) r.Group(func(r *Router) { r.Use(authenticateUser) r.Use(requireAdminUser) r.HandleFunc("GET /admin", showAdminDashboard) }) }) Again, complete tests for the Router type are available in this gist.

• When is it OK to panic in Go?
  Mar 31, 2025

  If you've been working with Go for a while, you might be familiar with the Go proverb "don't panic". It's a pithy way of saying: "handle errors gracefully, or return them to the caller to handle gracefully, instead of passing errors to the built-in panic() function". And while "don't panic" is a great guideline that you should follow, sometimes it's taken to mean that you should no-way, never, ever call panic(). And I don't think that's true. The panic() function is a tool, and there are some rare times when it might be the appropriate tool for the job. In this post we'll talk through what panic() does and why it's generally better to avoid using it, discuss some scenarios where panicking can be appropriate, and finish with a few real-world examples. Panicking vs. returning errors Let's begin by creating a timeIn() function, which takes a IANA time zone name and returns the current time in that zone. In Go, if the timeIn() function encounters an error, the normal and idiomatic way to deal with it would be to return the error to the caller. Like so: package main import ( "fmt" "os" "time" ) func timeIn(zone string) (time.Time, error) { loc, err := time.LoadLocation(zone) if err != nil { return time.Time{}, err // Return any error from time.LoadLocation() } return time.Now().In(loc), nil } func main() { tz := "Europe/Wonderland" t, err := timeIn(tz) if err != nil { fmt.Println("Error:", err) os.Exit(1) } fmt.Println("Current time in", tz, "is", t) } $ go run main.go Error: unknown time zone Europe/Wonderland exit status 1 In theory, you could handle the potential error inside timeIn() by passing it to panic() — instead of returning it. Like this: package main import ( "fmt" "time" ) func timeIn(zone string) time.Time { loc, err := time.LoadLocation(zone) if err != nil { panic(err) // Call panic() with the error as the argument } return time.Now().In(loc) } func main() { tz := "Europe/Wonderland" t := timeIn(tz) fmt.Println("Current time in", tz, "is", t) } $ go run main.go panic: unknown time zone Europe/Wonderland goroutine 1 [running]: main.timeIn({0x4c2c7e?, 0x7d40fe626108?}) /tmp/main.go:11 +0xc5 main.main() /tmp/main.go:20 +0x2b exit status 2 When you call panic() in your Go code, it will do the following four things: Immediately stop normal execution of the code in the current function. Nothing after the call to panic() will be executed. Run any deferred functions for the current goroutine in reverse (LIFO) order. Print out panic: and the value you passed to the panic() function to os.Stderr, along with a stack trace for the current goroutine at the point panic() was called, . Terminate the program with exit code 2. Note: It's possible to recover panics by using the recover() function inside a deferred function, in which case step 2 will only be executed until the point of recovery, and steps 3 and 4 in the list above won't automatically happen. Explaining how recover() works is outside the scope of this blog post, but here's a good introduction and I also recommend watching this video for a discussion of some intricacies. Also note: The official documentation for panic() only goes as far as saying "the program is terminated with a non-zero exit code". As far as I can see, at the time of writing, the exit code following an unrecovered panic is always 2, but the documentation deliberately says "non-zero" to give wiggle-room for a potential change to a different exit code in the future. There's a discussion about this here. Why is panicking considered bad? The panic() function itself isn't intrinsically bad. In fact, what it does for you is really quite nice — the running of deferred functions, the printing of the stack trace... this is good stuff. It's more that returning errors is normally better. When you call panic() in a function, it always sets off the same fixed chain of events that we described above. Whereas if the function returns the error, the caller has full control over how that error is managed. It could be logged, presented to a user, the function could be retried, or the error could even be ignored. Alternatively, the error could be propagated again back up the call stack to the grandparent caller to manage. It all depends on the use case. When you return an error, the caller has control and flexibility to handle it in the most appropriate way. There are also some other benefits of returning errors: When propagating errors back up the call stack, you can optionally wrap them to provide additional context at each step. This extra context can make errors more informative and useful, and potentially make debugging easier than relying solely on the stack trace from a panic(). It's easier to write unit tests for a function when it returns errors. It's certainly not impossible to verify that a function panics when you expect it to during a test, but it is more awkward and less clear than just checking an error return value. If you're creating a package for other people to import and use, it's polite to return errors instead of panicking. Remember: a panic will terminate the running application, which people using your package may not expect or appreciate! It's better to return an error, and leave it up to the caller to decide what to do next. They can always call panic() with the error if they want. Finally, it's just the Go way. Errors are normally returned — it's what the Go standard library mostly does, and it's what other Gophers have come to expect as standard. By sticking with this convention, your code is more predictable and easier for other people to follow. So, returning errors (or handling them gracefully then-and-there) is almost always better. Which leaves us with the question, when is panicking the better option? When is panicking appropriate? To answer this, it's helpful to distinguish between what I'll call "operational errors" and "programmer errors" for the purpose of this post. By operational errors, we're talking about errors that you might reasonably expect to happen during the operation of your program. Some examples are errors caused by a database or network resource being temporarily unavailable, the permissions on a file being wrong, a timeout on a long-running operation, or invalid user input. These errors don't necessarily mean there is a problem with your program itself — in fact they're often caused by things outside the control of your program. Operational errors are to be expected. And because you know there's a chance they'll occur during normal operation, you should endeavour to return them to the caller and gracefully handle them in a way that makes the most sense for your program. Don't use panic() to manage them. By programmer errors, we're talking about errors which should "never" happen during the operation of your program — the kind of error that stems from a developer mistake, a logical flaw in your codebase, or trying to use another piece of code in an unsupported way. Ideally, you'd spot programmer errors during development or testing, rather than having them surface in production. And (hopefully!) they should be relatively rare. When you encounter a programmer error, it means that your program finds itself in an unexpected state. And in this scenario, calling panic() is much more commonly accepted as an appropriate thing to do. For all the good reasons that we talked about above, if it is possible to safely and gracefully manage the error by returning it up the call stack, then you should default to doing that still. But using panic() can be a good and appropriate choice when either: The error is truly unrecoverable (that is, there is no reasonable way to safely continue operating and handle the error more gracefully); or Returning the error would add an unacceptable amount of complexity or additional error handling code to the rest of your codebase — all for something that you never expect to see in production. You can see this logic play out in some of the Go standard library operations that trigger a panic. For instance: Dividing an integer or float by 0 Accessing an out-of-bounds index in slice or array Dereferencing a nil pointer Trying to use a nil map Unlocking a mutex that isn't locked Sending on a closed channel Defining two flags with the same name in the same flag.FlagSet Passing an integer or > 999 to http.ResponseWriter.WriteHeader() When a sync.WaitGroup counter drops below zero What do all these have in common? First, they're programmer errors. If any of these things happen, it's due to a logical mistake in your codebase or you trying to use a language feature or function in an unsupported way. These things shouldn't happen during normal operation in production. And if they returned an error, it would add an arguably unacceptable amount of extra error handling to everyone's Go code. Just imagine if you had to check for an error return value every time you use the / operator, access a value in a slice, or unlock a mutex. It would add a lot of overhead. So, in summary, it can be appropriate to use panic() to deal with programmer errors that are either unrecoverable or where returning an error would add an unacceptable amount of extra error handling to the rest of your codebase. Exactly what constitutes "an unacceptable amount" is your judgement call, based on your experience and particular codebase. And that's OK. There's no exact right or wrong answer here. On top of this, there are a couple of other scenarios where I think calling panic() can be appropriate: In a last-ditch 'guard clause' to prevent a particular operation happening when it shouldn't. If the panic ever gets executed, it indicates a bug in your program or violation of some internal business logic. When you don't want the program to continue and there are no better options for dealing with the error beyond calling panic(). Real-world examples and discussion By now I hope it's clear that panic() should be used sparingly and only when it really makes sense. Personally, probably about half of the codebases I work on don't call panic() at all, and even when they do, it's only in a few places. So with that said, here are a few real-life examples from recent codebases I've worked on. Example one Here's an example from a web application, where we have some code to retrieve a user value from the HTTP request context. type contextKey string const userContextKey = contextKey("user") func contextGetUser(r *http.Request) user.User { user, ok := r.Context().Value(userContextKey).(user.User) if !ok { panic("missing user value in request context") } return user } In this particular application, the code is structured in such a way that the contextGetUser() function is only ever called when we logically expect there to be a user value in the request context. In this application, a missing value is firmly an programmer error and indicates that there is something wrong with the codebase. Yes, contextGetUser() could return an error instead of panicking. The error is certainly recoverable — the caller could cease further operations, log the error and send the user a 500 Internal Server Error response. But this function gets called a lot, and it felt like returning an error would introduce excessive error handling for something that we should never see during normal operation. On balance, using panic() here felt appropriate. Example two Here's another example from the same application: func getEnvInt(key string, defaultValue int) int { value, exists := os.LookupEnv(key) if !exists { return defaultValue } intValue, err := strconv.Atoi(value) if err != nil { panic(err) } return intValue } In this application, getEnvInt() is a helper function used to read a value from an environment variable and convert it to an int. If the conversion fails, then it panics. At first glance, this might not seem like a suitable place to use panic(). An error when trying to convert a specific environment variable to an int seems like something outside of our program's control — an operational error. And it is. But in this case, the getEnvInt function is used (and only used) right at the start of the program to load configuration settings from the environment, like so: httpPort := getEnvInt("HTTP_PORT", 3939) At this early stage of the program, the logger (which also happens to rely on environment settings) hasn't been initialized. Since the program can't run without valid configuration values, and there's no proper logger available yet to handle errors gracefully, there aren't any other good options on the table for managing this error. Resorting to panic() feels like a reasonable choice. It fits the scenario of you don't want the program to continue and there are no better options for dealing with the error. Note: I could have made the getEnvInt() function return an error, and had the caller itself call panic(). But it would have generated additional error handling for basically the same end result, so on balance it made sense to panic from within getEnvInt(). Example three This is an example of where I've previously used panic() in a guard clause. var safeChars = regexp.MustCompile("^[a-z0-9_]+$") type SortValues struct { Column string Ascending bool } func (sv *SortValues) OrderBySQL() string { if !safeChars.MatchString(sv.Column) { panic("unsafe sort column: " + sv.Column) } if sv.Ascending { return fmt.Sprintf("ORDER BY %s ASC", sv.Column) } return fmt.Sprintf("ORDER BY %s DESC", sv.Column) } In this particular application, there was a need to generate SQL queries with dynamic ORDER BY parameters based on user input. Unfortunately, SQL doesn't support placeholder parameters in ORDER BY clauses, so we have to use string interpolation to insert the column name and sort direction into the query instead. The SortValues type holds the user-provided column name and sort direction, and its OrderBySQL() method returns a string like ORDER BY title ASC. By the time that the OrderBySQL() method is called, one of the upstream functions should have already validated the SortValues.Column value against a whitelist of allowed column names. But if a bug, or oversight, ever caused that validation step to be missed, the application would be vulnerable to a SQL injection attack via the user-provided column name. So, as a last-ditch mitigation, we use a panicking guard clause in OrderBySQL() to ensure that the SortValues.Column value only contains 'safe' characters (a to z, 0 to 9, and underscores). We never expect this check to fail, so returning an error from OrderBySQL() seems like overkill. But if it ever did happen, it feels better to trigger a panic than risk compromising the database. Summary So, let's answer the title of this post: When is it OK to panic in Go? Your default should always be to return errors to the caller — or handle them gracefully then-and-there. "Don't panic" is a good guideline to almost always follow. But panic() isn't inherently bad, and using it is appropriate when: Your program encounters a programmer error and there is no way to manage it safely in a more graceful way. Your program encounters a programmer error and returning it to the caller would add an unacceptable amount of complexity or error handling to the rest of your codebase. You have a last-ditch 'guard clause' to prevent a particular operation happening when it shouldn't. Your program can't continue and there are simply no better options for dealing with the error in a more graceful way.

• How to manage tool dependencies in Go 1.24+
  Feb 20, 2025

  One of my favourite features of Go 1.24 is the new functionality for managing developer tooling dependencies. By this, I mean tooling that you use to assist with development, testing, build, or deployment – such as staticcheck for static code analysis, govulncheck for vulnerability scanning, or air for live-reloading applications. Historically, managing these dependencies — especially in a team setting — has been tricky. The previous solutions have been to use a tools.go file or the go run pattern, but while these approaches work, they’ve always felt like workarounds with some downsides. With Go 1.24, there’s finally a better way. Adding tools to your module Using tools Listing tools Verifying tools Vendoring tools Upgrading and downgrading tools Removing tools Using a separate modfile for tools A quick example To demonstrate the new functionality, let's scaffold a simple module and add some application code. $ go mod init example.com go: creating new go.mod: module example.com $ touch main.go File: main.go package main import ( "fmt" "github.com/kr/text" ) func main() { wrapped := text.Wrap("This is an informational message that should be wrapped.", 30) fmt.Println(wrapped) } Now fetch the github.com/kr/text package and run the code. The output should look like this: $ go get github.com/kr/text go: downloading github.com/kr/text v0.2.0 go: added github.com/kr/text v0.2.0 $ go run . This is an informational message that should be wrapped. Adding tools to your module Go 1.24 introduces the -tool flag for go get, which you can use like this: go get -tool import_path@version This command will download the package specified by the import path (along with any child dependencies), store them in your module cache, and record them in your go.mod file. The @version part is optional – if you omit it, the latest version will be downloaded. Let's use this to add the latest versions of stringer and govulncheck to our module as developer tools, along with staticcheck version 0.5.1. $ go get -tool golang.org/x/tools/cmd/stringer go: downloading golang.org/x/tools v0.30.0 go: downloading golang.org/x/sync v0.11.0 go: downloading golang.org/x/mod v0.23.0 go: added golang.org/x/mod v0.23.0 go: added golang.org/x/sync v0.11.0 go: added golang.org/x/tools v0.30.0 $ go get -tool golang.org/x/vuln/cmd/govulncheck go: downloading golang.org/x/vuln v1.1.4 go: downloading golang.org/x/telemetry v0.0.0-20240522233618-39ace7a40ae7 go: downloading golang.org/x/sys v0.30.0 go: upgraded golang.org/x/telemetry v0.0.0-20240521205824-bda55230c457 => v0.0.0-20240522233618-39ace7a40ae7 go: added golang.org/x/vuln v1.1.4 $ go get -tool honnef.co/go/tools/cmd/staticcheck@v0.5.1 go: downloading honnef.co/go/tools v0.5.1 go: downloading golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa go: added github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: added golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 go: added honnef.co/go/tools v0.5.1 After running these, your go.mod file will now include a tool (...) section listing the tools you've added. The corresponding module paths and versions for all the dependencies will appear in the require (...) section and be marked as indirect: File: go.mod module example.com go 1.24.0 require ( github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c // indirect github.com/kr/text v0.2.0 // indirect golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 // indirect golang.org/x/mod v0.23.0 // indirect golang.org/x/sync v0.11.0 // indirect golang.org/x/sys v0.30.0 // indirect golang.org/x/telemetry v0.0.0-20240522233618-39ace7a40ae7 // indirect golang.org/x/tools v0.30.0 // indirect golang.org/x/vuln v1.1.4 // indirect honnef.co/go/tools v0.5.1 // indirect ) tool ( golang.org/x/tools/cmd/stringer golang.org/x/vuln/cmd/govulncheck honnef.co/go/tools/cmd/staticcheck ) Using tools Once added, you can run tools using the go tool command. From the command line To run a specific tool from the command line within your module, you can use go tool followed by the last non-major-version segment of the import path for the tool (which is, normally, just the name for the tool). For example: $ go tool staticcheck -version staticcheck 2024.1.1 (0.5.1) $ go tool govulncheck No vulnerabilities found. In a makefile The go tool command also works nicely if you want to execute tools from your scripts or Makefiles. To illustrate, let's create a Makefile with an audit task that runs staticcheck and govulncheck on the codebase. $ touch Makefile .PHONY: audit audit: go vet ./... go tool staticcheck ./... go tool govulncheck If you run make audit, you should see that all the checks complete successfully. $ make audit go vet ./... go tool staticcheck ./... go tool govulncheck No vulnerabilities found. With go:generate Let's also take a look at an example where we use the stringer tool in conjunction with go:generate to generate String() methods for some iota constants. File: main.go package main import ( "fmt" "github.com/kr/text" ) //go:generate go tool stringer -type=Level type Level int const ( Info Level = iota Error Fatal ) func main() { wrapped := text.Wrap("This is an informational message that should be wrapped.", 30) fmt.Printf("%s: %s\n", Info, wrapped) } The important thing here is the //go:generate line. When you run go generate on this file, it will in turn use go tool to execute the version of the stringer tool listed in your go.mod file. Let's try it out: $ go generate . $ ls go.mod go.sum level_string.go main.go Makefile You should see that a new level_string.go file is created, and running the application should result in some output that looks like this: $ go run . Info: This is an informational message that should be wrapped. Listing tools You can check which tools have been added to a module by running go list tool, like so: $ go list tool honnef.co/go/tools/cmd/staticcheck golang.org/x/tools/cmd/stringer golang.org/x/vuln/cmd/govulncheck Verifying tools Because the tools are included in your go.mod file as dependencies, if you want to check that the code for the tools stored in your module cache has not changed you can simply run go mod verify: $ go mod verify all modules verified This will check that the code in your module cache exactly matches the corresponding checksums in your go.sum file. Vendoring tools If you run go mod vendor, the code for tooling dependencies will be included in the vendor folder and the vendor/modules.txt manifest alongside your non-tool dependencies. $ go mod vendor $ tree -L 3 . ├── go.mod ├── go.sum ├── main.go ├── Makefile └── vendor ├── github.com │ ├── BurntSushi │ └── kr ├── golang.org │ └── x ├── honnef.co │ └── go └── modules.txt When tools are vendored in this way, running go tool will execute the corresponding code in the vendor directory. Note that go mod verify does not work on vendored code. Upgrading and downgrading tools To upgrade or downgrade a specific tool to a specific version, you can use the same go get -tool import_path@version command that you did for adding the tool originally. For example: $ go get -tool honnef.co/go/tools/cmd/staticcheck@v0.5.0 To upgrade to the latest version of a specific tool, omit the @version suffix. $ go get -tool honnef.co/go/tools/cmd/staticcheck You can also upgrade all tools to their latest version by running go get tool. Note: tool is a sub-command here, not a flag. $ go get tool If your tool dependencies are vendored, you will need to re-run go mod vendor after any upgrades or downgrades. At the time of writing, I'm not aware of any easy way to specifically list the tools that have upgrades available – if you know of one please let me know! Removing tools To remove the tool completely from your module, use go get -tool with the special version tag @none. $ go get -tool honnef.co/go/tools/cmd/staticcheck@none Again, if you're vendoring, make sure to run go mod vendor after removing a tool. Using a separate modfile for tools A Reddit commenter mentioned the potential for problems if your tools share dependencies with your application code. For example, let's say that your application code depends on golang.org/x/sync version v0.11.0, and is tested and known to work with that version. Then if you add a tool that relies on a newer version of golang.org/x/sync, the version number in your go.mod file will be bumped to the newer version and your application code will use that newer version too. In theory, this shouldn't be a problem so long as all your dependencies and their child dependencies are stable, follow strict semantic versioning, and don't make backwards-incompatible changes without a major version increment. But, of course, the real world is messy and backwards-incompatible changes might happen, which could unexpectedly break your application code. It's worth noting that this issue isn't limited to tool dependencies – the same thing can happen if your application code and a non-tool dependency both rely on the same package. However, including tools in go.mod increases the risk. To reduce this risk, you can use a separate modfile for tool dependencies instead of including them in your main go.mod. You can do this with the -modfile flag, specifying an alternative file such as go.tool.mod, like so: # Initialize a go.tool.mod modfile $ go mod init -modfile=go.tool.mod example.com # Add a tool to the module $ go get -tool -modfile=go.tool.mod golang.org/x/vuln/cmd/govulncheck # Run the tool from the command line $ go tool -modfile=go.tool.mod govulncheck # List all tools added to the module $ go list -modfile=go.tool.mod tool # Verify the integrity of the tool dependencies $ go mod verify -modfile=go.tool.mod # Upgrade or downgrade a tool to a specific version $ go get -tool -modfile=go.tool.mod golang.org/x/vuln/cmd/govulncheck@v1.1.2 # Upgrade all tools to their latest version $ go get -modfile=go.tool.mod tool # Remove a tool from the module $ go get -tool -modfile=go.tool.mod golang.org/x/vuln/cmd/govulncheck@none

• Eleven tips for structuring your Go projects
  Jan 22, 2025

  When working with Go, you have three main building blocks to help organize your code: files, packages and modules. But as Go developers, one of the common challenges we have is knowing how to best combine these building blocks to structure a codebase. In this post, I'll share a mix of mindset tips and practical advice that I hope will help, especially if you're new to the language. Different projects, different structures Aim for effective, not perfect Forget conventions from other languages or frameworks Don't use directories just to organize files Use one of the standard layouts as a skeleton … And then let it evolve If you're unsure, begin with two files Keep related things close Big files aren't necessarily bad Create packages judiciously Look out for warning signs 1. Different projects, different structures I'd like to start by emphasizing that there's no single "right" way to structure a Go codebase. If you're using a specific framework or tool to scaffold your project, then you might be given a fixed directory structure to work with. But outside of that, there are relatively few conventions widely-adopted by the Go community, and the answer to "how should I structure my codebase?" is almost always "it depends". It depends on what you're building, your business needs, your testing approach, your team, your dependencies or tooling, and any internal conventions you choose to follow. Take a look at GitHub, and you'll find thousands of examples of successful Go projects — with quite different structures. For example, mkcert and Kubernetes are both excellent Go projects, but they differ significantly in scale and purpose. And these differences mean that their repository structures also look quite different. A structure that works well for your current project might not be the same as the structures that you've used before or seen elsewhere — and that's perfectly fine. 2. Aim for effective, not perfect If you're a perfectionist, this might be easier said than done, but try not to stress too much about making your codebase structure perfect. If you find yourself obsessing over the "perfect" way to organize your code, try to let go of that. Instead, aim for a structure that works effectively enough for your specific project. By "effective enough," I mean that your code is easy to find and navigate, the logic is straightforward to follow, changes can be made with confidence, and you're not running into the kind of warning signs I'll cover later in this post. 3. Forget conventions from other languages or frameworks Don't feel guilty if your codebase structure doesn't follow the conventions or best practices you're used to from other languages or frameworks. If it works effectively for your Go project, that's what matters. For example, if you're an experienced Ruby on Rails or Django developer building your first Go web application, you might be tempted to recreate the familiar directory structure from those frameworks. But while you probably could make it work if you tried, it's unlikely to be the most effective or simple solution for your Go project. 4. Don't use directories just to organize files This is a subtle but important point, especially if you're new to Go. You shouldn't create new directories just to organize your .go files. In Go, creating a directory creates a new package, and placing a file in that directory makes it part of that package. Create a directory only when you have a specific reason to create a new package – not because you want a neater/cleaner/clearer directory structure for your files. 5. Use one of the standard layouts as a skeleton The official Go documentation has a great article describing some standard project layouts. I use one of these layouts as the high level "skeleton structure" in pretty much every Go project I work on nowadays, and recommend that you do too. Small projects For small projects, consider using the basic layout where you just put everything in the project's root directory, like this: ├── main.go ├── foo.go ├── bar.go ├── go.mod └── README.md A couple of real-life examples of projects that use this layout are mkcert and flow. Small projects with supporting packages For projects where you need to break out some code out into supporting packages, use the supporting packages layout. In this pattern, the supporting packages live within an internal directory in the project root, and your main package files and other project assets continue to live in the root directory. ├── internal │ └── foo │ └── foo.go ├── main.go ├── bar.go ├── go.mod └── README.md Note: The directory name internal carries a special meaning and behavior in Go: any package which lives under this directory can only be imported by code inside the parent of the internal directory. So if you put your supporting packages in an internal directory, like above, they cannot be imported by code outside of your project – even if the project code is publicly available somewhere like GitHub. That's often useful, because it means you can confidently refactor the code within the internal folder without inadvertently breaking something for other people. Larger projects For larger projects I generally recommend using the server project layout, especially if: Your project will have a lot non .go assets (like template files, SQL migrations, tool configurations and Makefiles); or Your project will contain more than one main package (e.g. main packages for a web application and a CLI tool) In this layout: Your executable main package files live in sub-directories under a cmd directory The rest of your Go packages live in an internal directory All other project assets remain in the root of the project directory Like so: ├── cmd │ └── foo │ ├── main.go │ └── bar.go ├── internal │ └── baz │ └── baz.go ├── go.mod ├── Makefile └── README.md For a more complete example, here's the directory structure from a recent project I worked on – including main packages for a web server and CLI application, along with various non-Go assets. 6. … And then let it evolve Use one of the standard project layouts as your high-level 'skeleton', but beyond that, I recommend letting the rest of the structure within that skeleton evolve naturally as development progresses. In other words, don't decide your directory structure or what .go files you will have upfront, and then shoehorn in your Go code into that. Instead, let the code you're writing guide the files and packages that you create. 7. If you're unsure, begin with two files If you're in any doubt, start with the basic layout and just a go.mod and main.go file in the root of your project directory. Then, as your project evolves, add additional files and packages as needed. Starting this way is perfectly OK. Personally, about half of the new projects I work on begin with just these two files — and nothing more. 8. Keep related things close This one feels pretty obvious – especially if you're an experienced developer – but it's still worth saying. As a general rule, keep related things close to each other – in the same .go file or in the same package. Here are a few examples: Constants, variables, custom types and utility functions (which are not reused by multiple packages) should be declared close to the code they support, in the same .go file or package. It may make sense to group utility functions that are related to each other and used in multiple places into a single reusable package. If you have a custom struct type, define any methods for it directly below the struct declaration in the same .go file. In a web application or API, define all routing rules together in a single function or .go file. There will probably be times when it makes sense for you to break the 'keep related things close' rule in your code, and that's OK, but it's a good principle to default to. 9. Big files aren't necessarily bad So long as it doesn't cause you practical problems during development or maintenance, file size in Go doesn't matter. It's OK to have .go files that contain a couple of lines, or thousands. Neither of these things is automatically considered an anti-pattern in Go. To give you an idea of some big files, the runtime/proc.go file from the Go standard library contains 6,548 lines of code. And /pkg/apis/core/validation/validation.go from the Kubernetes repository contains 8,606 lines of code (it's corresponding _test.go file also has over 26,000 lines). I'm not saying that your .go files should be big. More that if – on balance – it makes sense to have a big file… then it makes sense. Don't feel guilty about it, and don't feel like you need to break it into smaller files unless there's a good reason to. 10. Create packages judiciously In a similar vein, big packages aren't necessarily bad. In fact, I'd say that one of the more common mistakes in Go is splitting up your code into too many small packages. The problem with having lots of small packages is that it can add complexity to your application, especially when you need to share state, configuration, or dependencies across package boundaries. It also increases the likelihood of encountering import cycle problems. As a rule of thumb, only create additional packages when you have a demonstrable need or good reason to. For example: You have some code that you want to reuse. Putting the code in a standalone package facilitates this because you can then import the package and use it in different files throughout your project, or even copy-and-paste the package directory straight into another codebase. You want to isolate or enforce a boundary between the package code and the rest of your project. For example, you might use packages as an architectural tool to create lightweight decoupled 'layers' in your project code, or to isolate part of the codebase so it's easier for another person or team to work on separately. You have some code that acts as a 'black box' and moving it to a standalone package will reduce cognitive overhead and make your codebase clearer overall. 11. Look out for warning signs It can be hard to know exactly when your project structure is working effectively… instead it's probably easier to spot the signs that it isn't working effectively in practice. Some things to look out for are: You keep running into import cycle problems. It's hard to find things in the codebase, especially after time away or for new contributors. Relatively small changes often impact multiple packages or .go files. The flow of control is overly "jumpy" and hard to follow when debugging. There's a lot of duplication that's difficult to refactor out (note: some duplication is not always bad.) You're finding it difficult to manage errors appropriately. You feel like your are 'fighting the language', or you resort to using language features in a way that is not intended or idiomatic. It feels like a single file or package is doing too much and that there isn't a clear separation of responsibilities within it, and this is having a negative effect on the clarity of your code. If you spot these warning signs, it might be worth taking a step back and considering if tweaking the structure your codebase and packages will help to fix the problem.

• Implementing an in-memory cache in Go
  Dec 22, 2023

  In almost all web applications that I build, I end up needing to persist some data – either for a short period of time (such as caching the result of an expensive database query), or for the lifetime of the running application until it is restarted. When your application is a single binary running on a single machine, a simple, effective, and no-dependency way to do this is by persisting the data in memory using a mutex-protected map. And since generics was introduced in Go 1.18, it's possible to write a generic implementation that you can use to persist various different data types in a type-safe way. Note: The code for this post can be found in this gist. Long-lived cache If you want to persist data for the lifetime of the running application (or until you deliberately delete the data), you can create a generic Cache type like this: package cache import ( "sync" "time" ) // Cache is a basic in-memory key-value cache implementation. type Cache[K comparable, V any] struct { items map[K]V // The map storing key-value pairs. mu sync.Mutex // Mutex for controlling concurrent access to the cache. } // New creates a new Cache instance. func New[K comparable, V any]() *Cache[K, V] { return &Cache[K, V]{ items: make(map[K]V), } } // Set adds or updates a key-value pair in the cache. func (c *Cache[K, V]) Set(key K, value V) { c.mu.Lock() defer c.mu.Unlock() c.items[key] = value } // Get retrieves the value associated with the given key from the cache. The bool // return value will be false if no matching key is found, and true otherwise. func (c *Cache[K, V]) Get(key K) (V, bool) { c.mu.Lock() defer c.mu.Unlock() value, found := c.items[key] return value, found } // Remove deletes the key-value pair with the specified key from the cache. func (c *Cache[K, V]) Remove(key K) { c.mu.Lock() defer c.mu.Unlock() delete(c.items, key) } // Pop removes and returns the value associated with the specified key from the cache. func (c *Cache[K, V]) Pop(key K) (V, bool) { c.mu.Lock() defer c.mu.Unlock() value, found := c.items[key] // If the key is found, delete the key-value pair from the cache. if found { delete(c.items, key) } return value, found } And you can use it like this: package main import ( "fmt" "time" "path/to/cache" ) func main() { // Create a new Cache instance myCache := cache.New[string, int]() // Set key-value pairs in the cache myCache.Set("one", 1) myCache.Set("two", 2) myCache.Set("three", 3) // Retrieve values from the cache value, found := myCache.Get("two") if found { fmt.Printf("Value for key 'two': %v\n", value) } else { fmt.Println("Key 'two' not found in the cache") } // Pop a key from the cache poppedValue, found := myCache.Pop("three") if found { fmt.Printf("Popped value for key 'three': %v\n", poppedValue) } else { fmt.Println("Key 'three' not found in the cache") } // Remove a key from the cache myCache.Remove("one") // Try to retrieve a removed key removedValue, found := myCache.Get("one") if found { fmt.Printf("Value for key 'one': %v\n", removedValue) } else { fmt.Println("Key 'one' not found in the cache (after removal)") } } Expiring cache You can extend this idea to associate an expiry time with every value in the cache, and launch a background goroutine to periodically remove expired entries. Like so: package cache import ( "sync" "time" ) // item represents a cache item with a value and an expiration time. type item[V any] struct { value V expiry time.Time } // isExpired checks if the cache item has expired. func (i item[V]) isExpired() bool { return time.Now().After(i.expiry) } // TTLCache is a generic cache implementation with support for time-to-live // (TTL) expiration. type TTLCache[K comparable, V any] struct { items map[K]item[V] // The map storing cache items. mu sync.Mutex // Mutex for controlling concurrent access to the cache. } // NewTTL creates a new TTLCache instance and starts a goroutine to periodically // remove expired items every 5 seconds. func NewTTL[K comparable, V any]() *TTLCache[K, V] { c := &TTLCache[K, V]{ items: make(map[K]item[V]), } go func() { for range time.Tick(5 * time.Second) { c.mu.Lock() // Iterate over the cache items and delete expired ones. for key, item := range c.items { if item.isExpired() { delete(c.items, key) } } c.mu.Unlock() } }() return c } // Set adds a new item to the cache with the specified key, value, and // time-to-live (TTL). func (c *TTLCache[K, V]) Set(key K, value V, ttl time.Duration) { c.mu.Lock() defer c.mu.Unlock() c.items[key] = item[V]{ value: value, expiry: time.Now().Add(ttl), } } // Get retrieves the value associated with the given key from the cache. func (c *TTLCache[K, V]) Get(key K) (V, bool) { c.mu.Lock() defer c.mu.Unlock() item, found := c.items[key] if !found { // If the key is not found, return the zero value for V and false. return item.value, false } if item.isExpired() { // If the item has expired, remove it from the cache and return the // value and false. delete(c.items, key) return item.value, false } // Otherwise return the value and true. return item.value, true } // Remove removes the item with the specified key from the cache. func (c *TTLCache[K, V]) Remove(key K) { c.mu.Lock() defer c.mu.Unlock() // Delete the item with the given key from the cache. delete(c.items, key) } // Pop removes and returns the item with the specified key from the cache. func (c *TTLCache[K, V]) Pop(key K) (V, bool) { c.mu.Lock() defer c.mu.Unlock() item, found := c.items[key] if !found { // If the key is not found, return the zero value for V and false. return item.value, false } // If the key is found, delete the item from the cache. delete(c.items, key) if item.isExpired() { // If the item has expired, return the value and false. return item.value, false } // Otherwise return the value and true. return item.value, true } And you can use this in much the same way: package main import ( "fmt" "time" "path/to/cache" ) func main() { // Create a new TTLCache instance myTTLCache := cache.NewTTL[string, int]() // Set key-value pairs with TTL in the cache myTTLCache.Set("one", 1, 5*time.Second) myTTLCache.Set("two", 2, 10*time.Second) myTTLCache.Set("three", 3, 15*time.Second) // Retrieve values from the cache value, found := myTTLCache.Get("two") if found { fmt.Printf("Value for key 'two': %v\n", value) } else { fmt.Println("Key 'two' not found in the cache or has expired") } // Wait for a while to allow some items to expire time.Sleep(7 * time.Second) // Try to retrieve an expired key expiredValue, found := myTTLCache.Get("one") if found { fmt.Printf("Value for key 'one': %v\n", expiredValue) } else { fmt.Println("Key 'one' not found in the cache or has expired") } // Pop a key from the cache poppedValue, found := myTTLCache.Pop("two") if found { fmt.Printf("Popped value for key 'two': %v\n", poppedValue) } else { fmt.Println("Key 'two' not found in the cache or has expired") } // Remove a key from the cache myTTLCache.Remove("three") }

• A complete guide to working with Cookies in Go
  Sep 28, 2022

  In this post we're going to run through how to use cookies in your Go web application to persist data between HTTP requests for a specific client. We'll start simple, and slowly build up a working application which covers the following topics: Basic reading and writing of cookies Encoding special characters and maximum length Using tamper-proof (signed) cookies Using confidential (encrypted) and tamper-proof cookies Storing custom data types in cookies Hint: If you're new to web development and need a general introduction to what cookies are and how they work, I recommend reading this MDN article before continuing. If you just want the final code, rather than the explanations, you can find it in this gist. Basic use The first thing to know is that cookies in Go are represented by the http.Cookie type. This is a struct which looks like this: type Cookie struct { Name string Value string Path string Domain string Expires time.Time RawExpires string // MaxAge=0 means no 'Max-Age' attribute specified. // MaxAge0 means Max-Age attribute present and given in seconds MaxAge int Secure bool HttpOnly bool SameSite SameSite Raw string Unparsed []string } Name is the cookie name. It can contain any US-ASCII characters except ( ) @ , ; : \ " / [ ? ] = { } and space, tab and control characters. It is a mandatory field. Value contains the data that you want to persist. It can contain any US-ASCII characters except , ; \ " and space, tab and control characters. It is a mandatory field. Path, Domain, Expires, MaxAge, Secure, HttpOnly and SameSite map directly to the respective cookie attributes. All of these are optional fields. If set, the value of the SameSite field should be one of the SameSite constants from the net/http package. The RawExpires, Raw and Unparsed fields are only used when your Go program is acting as a client (rather than a server) and parsing the cookies from a HTTP response. Most of the time you won't need to use these fields. Cookies can be written in a HTTP response using the http.SetCookie() function, and read from a HTTP request using the *Request.Cookie() method. Let's jump in and use these things in a working example. If you'd like to follow along, please run the following commands to set up a basic project scaffold: $ mkdir example-project $ cd example-project/ $ go mod init example.com/example-project go: creating new go.mod: module example.com/example-project $ touch main.go In the main.go file we're going to create a simple web application with two endpoints: GET /set which writes a new cookie along with the HTTP response. GET /get which reads the cookie sent with the HTTP request and then echoes out the cookie value in the response. Go ahead and add the following code to main.go: File: main.gopackage main import ( "errors" "log" "net/http" ) func main() { // Start a web server with the two endpoints. mux := http.NewServeMux() mux.HandleFunc("/set", setCookieHandler) mux.HandleFunc("/get", getCookieHandler) log.Print("Listening...") err := http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func setCookieHandler(w http.ResponseWriter, r *http.Request) { // Initialize a new cookie containing the string "Hello world!" and some // non-default attributes. cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello world!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Use the http.SetCookie() function to send the cookie to the client. // Behind the scenes this adds a `Set-Cookie` header to the response // containing the necessary cookie data. http.SetCookie(w, &cookie) // Write a HTTP response as normal. w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Retrieve the cookie from the request using its name (which in our case is // "exampleCookie"). If no matching cookie is found, this will return a // http.ErrNoCookie error. We check for this, and return a 400 Bad Request // response to the client. cookie, err := r.Cookie("exampleCookie") if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } // Echo out the cookie value in the response body. w.Write([]byte(cookie.Value)) } Note: As an aside, we have set the Secure attribute on the cookie to true. This indicates to the client (usually a web browser) that the cookie should only be used for 'secure' connections. Generally speaking this means that the cookie should only be used for encrypted connections (i.e. HTTPS), but many modern browsers (including Firefox and Chrome) also consider unencrypted connections to localhost to be 'secure'. This means that the cookie should work on localhost even if our web application is only using HTTP. OK, let's try this out. Go ahead and run the application: $ go run . 2022/09/25 10:44:11 Listening... And then open http://localhost:3000/set in your web browser. You should see the "cookie set!" response and, if you have developer tools open, you should also see the Set-Cookie header containing the data in the HTTP response headers. Then if you visit http://localhost:3000/get, our exampleCookie cookie should be passed back along with the HTTP request, and our getCookieHandler will retrieve the cookie value and print it in the response. Like so: If you want, you can also make a request to http://localhost:3000/set using curl to see the contents of the Set-Cookie header. Like so: $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie="Hello world!"; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Sun, 25 Sep 2022 08:45:02 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! Encoding special characters and maximum length So far, so good! But there are a couple of important things to be aware of when writing cookies. As mentioned briefly above, cookie values must only contain a subset of the US-ASCII characters. If you try to use an unsupported character, Go will strip it out before setting the Set-Cookie header. Let's try this out by adapting our setCookieHandler to write a cookie value containing a non US-ASCII character like "Hello Zoë!" (notice the umlauted ë character): File: main.go package main ... func setCookieHandler(w http.ResponseWriter, r *http.Request) { cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } http.SetCookie(w, &cookie) w.Write([]byte("cookie set!")) } ... Then when you make a request to http://localhost:3000/set, you'll see that the cookie value has been stripped down to "Hello Zo!". $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie="Hello Zo!"; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Sun, 25 Sep 2022 09:00:03 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! A good way to avoid this kind of problem is to base64-encode your cookie values before writing them. Because the base64 character set is a subset of the US-ASCII characters supported in cookies, we can be confident that nothing will be stripped from the cookie value. Another thing to be aware of is that web browsers impose a maximum size limit on cookies. But this limit — and how the cookie size is calculated — depends on the browser version being used. To prevent problems, a good rule-of-thumb is to keep the total size of the cookie (including all attributes) to no more than 4096 bytes. If you try to send a cookie larger than 4096 bytes, Go will write the Set-Cookie header without any problems (it won't be truncated), but there is a risk that the client may truncate or reject the cookie. To help with these two potential problems, let's create an internal/cookies package containing a couple of helper functions: A Write() function which encodes a cookie value to base64 and checks that the total length of the cookie is no more than 4096 bytes before writing it. A Read() function which reads a cookie from the current request and decodes the cookie value from base64. $ mkdir -p internal/cookies $ touch internal/cookies/cookies.go File: internal/cookies/cookies.gopackage cookies import ( "encoding/base64" "errors" "net/http" ) var ( ErrValueTooLong = errors.New("cookie value too long") ErrInvalidValue = errors.New("invalid cookie value") ) func Write(w http.ResponseWriter, cookie http.Cookie) error { // Encode the cookie value using base64. cookie.Value = base64.URLEncoding.EncodeToString([]byte(cookie.Value)) // Check the total length of the cookie contents. Return the ErrValueTooLong // error if it's more than 4096 bytes. if len(cookie.String()) > 4096 { return ErrValueTooLong } // Write the cookie as normal. http.SetCookie(w, &cookie) return nil } func Read(r *http.Request, name string) (string, error) { // Read the cookie as normal. cookie, err := r.Cookie(name) if err != nil { return "", err } // Decode the base64-encoded cookie value. If the cookie didn't contain a // valid base64-encoded value, this operation will fail and we return an // ErrInvalidValue error. value, err := base64.URLEncoding.DecodeString(cookie.Value) if err != nil { return "", ErrInvalidValue } // Return the decoded cookie value. return string(value), nil } Then we can update our main.go file to use these new helpers, like so: File: main.gopackage main import ( "errors" "log" "net/http" "example.com/example-project/internal/cookies" // Import the internal/cookies package. ) ... func setCookieHandler(w http.ResponseWriter, r *http.Request) { // Initialize the cookie as normal. cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Write the cookie. If there is an error (due to an encoding failure or it // being too long) then log the error and send a 500 Internal Server Error // response. err := cookies.Write(w, cookie) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Use the Read() function to retrieve the cookie value, additionally // checking for the ErrInvalidValue error and handling it as necessary. value, err := cookies.Read(r, "exampleCookie") if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } w.Write([]byte(value)) } If you restart your web application and make a request to http://localhost:3000/set followed by http://localhost:3000/get in your browser, you should now successfully see the message "Hello Zoë!" in full. Likewise, if you make a request to http://localhost:3000/set using curl, you should see that the cookie value is SGVsbG8gWm_DqyE= — which is the base64 encoding of Hello Zoë!. $ curl -i localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie=SGVsbG8gWm_DqyE=; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Sun, 25 Sep 2022 09:14:18 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set $ echo "SGVsbG8gWm_DqyE=" | base64url --decode Hello Zoë! Tamper-proof (signed) cookies By default, you shouldn't trust cookie data. Because cookies are stored on the client, it's fairly straightforward for a user to edit them (in fact, many web browser extensions exist for exactly this purpose). So if you're performing actions in your web application based on the value of a cookie, it's important to first verify that the cookie hasn't been edited and contains the original name and value that you set. A good way to do this is to generate a HMAC signature of the cookie name and value, and then prepend this signature to the cookie value before sending it to the client. So that the final value is in this format: cookie.Value = "{HMAC signature}{original value}" When we receive the cookie back from the client, we can recalculate the HMAC signature from the cookie name and original value, and check that the recalculated HMAC signature matches the signature at the start of the received cookie. If they match, it confirms the integrity of the name and value — and we know that it hasn't been edited by the client. Let's update the internal/cookies/cookies.go file to include some WriteSigned() and ReadSigned() functions which do exactly that. File: internal/cookies/cookies.gopackage cookies import ( "crypto/hmac" "crypto/sha256" "encoding/base64" "errors" "net/http" ) ... func WriteSigned(w http.ResponseWriter, cookie http.Cookie, secretKey []byte) error { // Calculate a HMAC signature of the cookie name and value, using SHA256 and // a secret key (which we will create in a moment). mac := hmac.New(sha256.New, secretKey) mac.Write([]byte(cookie.Name)) mac.Write([]byte(cookie.Value)) signature := mac.Sum(nil) // Prepend the cookie value with the HMAC signature. cookie.Value = string(signature) + cookie.Value // Call our Write() helper to base64-encode the new cookie value and write // the cookie. return Write(w, cookie) } func ReadSigned(r *http.Request, name string, secretKey []byte) (string, error) { // Read in the signed value from the cookie. This should be in the format // "{signature}{original value}". signedValue, err := Read(r, name) if err != nil { return "", err } // A SHA256 HMAC signature has a fixed length of 32 bytes. To avoid a potential // 'index out of range' panic in the next step, we need to check sure that the // length of the signed cookie value is at least this long. We'll use the // sha256.Size constant here, rather than 32, just because it makes our code // a bit more understandable at a glance. if len(signedValue) Alright, let's update our main.go file to include a secret key and use the new WriteSigned() and ReadSigned() functions. The secret key should be generated using a cryptographically secure random number generator (CSRNG), should be unique to your application, and should ideally have at least 32 bytes of entropy. For the purpose of this example, we'll use a random 64 character hex string and decode it to give us a byte slice containing 32 random bytes. File: main.gopackage main import ( "encoding/hex" "errors" "log" "net/http" "example.com/example-project/internal/cookies" ) // Declare a global variable to hold the secret key. var secretKey []byte func main() { var err error // Decode the random 64-character hex string to give us a slice containing // 32 random bytes. For simplicity, I've hardcoded this hex string but in a // real application you should read it in at runtime from a command-line // flag or environment variable. secretKey, err = hex.DecodeString("13d6b4dff8f84a10851021ec8608f814570d562c92fe6b5ec4c9f595bcb3234b") if err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/set", setCookieHandler) mux.HandleFunc("/get", getCookieHandler) log.Print("Listening...") err = http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func setCookieHandler(w http.ResponseWriter, r *http.Request) { cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Use the WriteSigned() function, passing in the secret key as the final // argument. err := cookies.WriteSigned(w, cookie, secretKey) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Use the ReadSigned() function, passing in the secret key as the final // argument. value, err := cookies.ReadSigned(r, "exampleCookie", secretKey) if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } w.Write([]byte(value)) } If you visit http://localhost:3000/set in your web browser followed by http://localhost:3000/get, you should still successfully see the message "Hello Zoë!". If you like, you can also use a browser extension to change the cookie value (search for "cookie editor" in your browser extension store). If you do this and visit http://localhost:3000/get again, you should now receive a 400 Bad Request response and the "invalid cookie" message. Before we move on, let's also make a request to http://localhost:3000/set using curl: $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie=1lYrR9MfMsu6Dm39EgfbOuFTUbZm3_5tmWsF943HN4hIZWxsbyBab8OrIQ==; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Wed, 28 Sep 2022 09:28:55 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! In my case we can see that the signed cookie value is: 1lYrR9MfMsu6Dm39EgfbOuFTUbZm3_5tmWsF943HN4hIZWxsbyBab8OrIQ== Let's base64-decode this: $ echo "1lYrR9MfMsu6Dm39EgfbOuFTUbZm3_5tmWsF943HN4hIZWxsbyBab8OrIQ==" | base64url --decode �V+G�2˺m��:�SQ�f��m�k���7�Hello Zoë! The first part of the decoded value is the HMAC signature (which looks like gibberish), followed by our original cookie value in plaintext. Confidential (encrypted) and tamper-proof cookies The HMAC signing pattern above is great for times when you want to confirm that a cookie has not been edited by a client, and you're not worried about the client being able to read the cookie data (i.e. the cookie doesn't contain any secret or confidential information). But if you do want to prevent the client from being able to read the cookie data, we need to encrypt the data before writing it. A good way to encrypt the data in cookies is to use AES-GCM (AES with Galois/Counter Mode) encryption. AES-GCM is a type of authenticated encryption, which is good because it both encrypts and authenticates the data. The encryption ensures confidentiality of the data, and the authentication ensures the integrity of the data (i.e. that the data hasn't been changed). Effectively, encrypting our cookie data using AES-GCM is a relatively easy way to give us confidential, tamper-proof, cookies in a single step. Let's create two new helper functions, WriteEncrypted() and ReadEncrypted(), which use this. Like so: File: internal/cookies/cookies.gopackage cookies import ( "crypto/aes" "crypto/cipher" "crypto/hmac" "crypto/rand" "crypto/sha256" "encoding/base64" "errors" "fmt" "io" "net/http" "strings" ) ... func WriteEncrypted(w http.ResponseWriter, cookie http.Cookie, secretKey []byte) error { // Create a new AES cipher block from the secret key. block, err := aes.NewCipher(secretKey) if err != nil { return err } // Wrap the cipher block in Galois Counter Mode. aesGCM, err := cipher.NewGCM(block) if err != nil { return err } // Create a unique nonce containing 12 random bytes. nonce := make([]byte, aesGCM.NonceSize()) _, err = io.ReadFull(rand.Reader, nonce) if err != nil { return err } // Prepare the plaintext input for encryption. Because we want to // authenticate the cookie name as well as the value, we make this plaintext // in the format "{cookie name}:{cookie value}". We use the : character as a // separator because it is an invalid character for cookie names and // therefore shouldn't appear in them. plaintext := fmt.Sprintf("%s:%s", cookie.Name, cookie.Value) // Encrypt the data using aesGCM.Seal(). By passing the nonce as the first // parameter, the encrypted data will be appended to the nonce — meaning // that the returned encryptedValue variable will be in the format // "{nonce}{encrypted plaintext data}". encryptedValue := aesGCM.Seal(nonce, nonce, []byte(plaintext), nil) // Set the cookie value to the encryptedValue. cookie.Value = string(encryptedValue) // Write the cookie as normal. return Write(w, cookie) } func ReadEncrypted(r *http.Request, name string, secretKey []byte) (string, error) { // Read the encrypted value from the cookie as normal. encryptedValue, err := Read(r, name) if err != nil { return "", err } // Create a new AES cipher block from the secret key. block, err := aes.NewCipher(secretKey) if err != nil { return "", err } // Wrap the cipher block in Galois Counter Mode. aesGCM, err := cipher.NewGCM(block) if err != nil { return "", err } // Get the nonce size. nonceSize := aesGCM.NonceSize() // To avoid a potential 'index out of range' panic in the next step, we // check that the length of the encrypted value is at least the nonce // size. if len(encryptedValue) Then we can switch our main.go file to use these new helpers like so: File: main.gopackage main ... func setCookieHandler(w http.ResponseWriter, r *http.Request) { cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } err := cookies.WriteEncrypted(w, cookie, secretKey) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { value, err := cookies.ReadEncrypted(r, "exampleCookie", secretKey) if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } w.Write([]byte(value)) } Note: When using AES-GCM encryption, it's important that your secret key is exactly 32 bytes long. Otherwise you will get a runtime error like crypto/aes: invalid key size . Again, you can visit http://localhost:3000/set and http://localhost:3000/get in your browser, and you should still successfully see the message "Hello Zoë!". And if you edit the exampleCookie cookie using a browser extension, you should find that any subsequent requests result in an "invalid cookie" response. Let's take a look at the Set-Cookie header now using curl. $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie=hBGecbVJ2cI0yAwrbMYd5sv7qslxBJoGnk7LBLHVR9rKrqh1cTVs2IuWHZUOkl2fdYeIYmY=; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Wed, 28 Sep 2022 10:37:23 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! In my case the encrypted cookie value is: hBGecbVJ2cI0yAwrbMYd5sv7qslxBJoGnk7LBLHVR9rKrqh1cTVs2IuWHZUOkl2fdYeIYmY= If we base64-decode this value, we should now just see gibberish and our original "Hello Zoë!" value should no longer be visible. $ echo "hBGecbVJ2cI0yAwrbMYd5sv7qslxBJoGnk7LBLHVR9rKrqh1cTVs2IuWHZUOkl2fdYeIYmY=" | base64url --decode ��q�I��4�+l������q��N���G�ʮ�uq5l؋���]�u��bf Great! The encryption has worked! Storing custom data types So far we've just been storing simple string data in our cookies. But what if we want to store something more complicated, like the data for a user represented as a struct in Go? type User struct { Name string Age int } The good news is that the Go standard library includes the encoding/gob package, which we can use to encode/decode a Go value to and from a byte slice. It's kind of like "pickling" in Python, "marshaling" in Ruby, or "serializing" in PHP. To help demonstrate how to use this, let's update our main.go file to gob-encode a User struct and store it in a cookie: package main import ( "bytes" "encoding/gob" "encoding/hex" "errors" "fmt" "log" "net/http" "strings" "example.com/example-project/internal/cookies" ) var secret []byte // Declare the User type. type User struct { Name string Age int } func main() { // Importantly, we need to tell the encoding/gob package about the Go type // that we want to encode. We do this my passing *an instance* of the type // to gob.Register(). In this case we pass a pointer to an initialized (but // empty) instance of the User struct. gob.Register(&User{}) var err error secret, err = hex.DecodeString("13d6b4dff8f84a10851021ec8608f814570d562c92fe6b5ec4c9f595bcb3234b") if err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/set", setCookieHandler) mux.HandleFunc("/get", getCookieHandler) log.Print("Listening...") err = http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func setCookieHandler(w http.ResponseWriter, r *http.Request) { // Initialize a User struct containing the data that we want to store in the // cookie. user := User{Name: "Alice", Age: 21} // Initialize a buffer to hold the gob-encoded data. var buf bytes.Buffer // Gob-encode the user data, storing the encoded output in the buffer. err := gob.NewEncoder(&buf).Encode(&user) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } // Call buf.String() to get the gob-encoded value as a string and set it as // the cookie value. cookie := http.Cookie{ Name: "exampleCookie", Value: buf.String(), Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Write an encrypted cookie containing the gob-encoded data as normal. err = cookies.WriteEncrypted(w, cookie, secret) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Read the gob-encoded value from the encrypted cookie, handling any errors // as necessary. gobEncodedValue, err := cookies.ReadEncrypted(r, "exampleCookie", secret) if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } // Create a new instance of a User type. var user User // Create an strings.Reader containing the gob-encoded value. reader := strings.NewReader(gobEncodedValue) // Decode it into the User type. Notice that we need to pass a *pointer* to // the Decode() target here? if err := gob.NewDecoder(reader).Decode(&user); err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } // Print the user information in the response. fmt.Fprintf(w, "Name: %q\n", user.Name) fmt.Fprintf(w, "Age: %d\n", user.Age) } If you want, restart your application and visit localhost:3000/set followed by localhost:3000/get in your web browser. You should see a response similar to this: Note: In the example above, we've gob-encoded the cookie data and then written the cookie using WriteEncrypted(), but you could equally write the cookie using the Write() or WriteSigned() helper functions that we made earlier too.

• Flow: A tiny but powerful HTTP router for Go
  May 25, 2022

  Last year I wrote a new HTTP router for Go called Flow. I've been using it in production on this site and in a couple of other projects since, and I'm pretty happy with how it's working out so decided to share it a bit more widely. My aim with Flow was to bring together my favourite features from other popular routers that I frequently used. It has: A very small and readable codebase (approx. 160 LOC) with pattern-matching logic similar to matryer/way. Middleware management like chi — including the ability to create route 'groups' which use different middleware. Optional regexp support for tighter pattern matching, similar to chi and gorilla/mux. Automatic handling of OPTIONS requests, like julienschmidt/httprouter. Automatic handling of HEAD requests, like bmizerany/pat. An Allow header is automatically set on all OPTIONS and 405 Method Not Allowed responses, like julienschmidt/httprouter. Ability to map multiple HTTP methods to the same handler in one declaration, like gorilla/mux. Additionally: It has a very small API (see the Go docs) so there's not much to learn. It's designed to work nicely with http.Handler, http.HandlerFunc, and the standard Go middleware pattern. The handlers for 404 Not Found and 405 Method Not Allowed responses are customizable. Conflicting routes are permitted (e.g. /posts/:id and posts/new), with routes matched in the order that they are declared. It has zero dependencies. Below is a quick example of the syntax, and if you like the look of it you can check out the full README on GitHub. mux := flow.New() // The Use() method can be used to register middleware. Middleware declared at // the top level will used on all routes (including error handlers and OPTIONS // responses). mux.Use(exampleMiddleware1) // Routes can use multiple HTTP methods. mux.HandleFunc("/profile/:name", exampleHandlerFunc1, "GET", "POST") // Optionally, regular expressions can be used to enforce a specific pattern // for a named parameter. mux.HandleFunc("/profile/:name/:age|^[0-9]{1,3}$", exampleHandlerFunc2, "GET") // The wildcard ... can be used to match the remainder of a request path. // Notice that HTTP methods are also optional (if not provided, all HTTP // methods will match the route). mux.Handle("/static/...", exampleHandler) // You can create route 'groups'. mux.Group(func(mux *flow.Mux) { // Middleware declared within in the group will only be used on the routes // in the group. mux.Use(exampleMiddleware2) mux.HandleFunc("/admin", exampleHandlerFunc3, "GET") // Groups can be nested. mux.Group(func(mux *flow.Mux) { mux.Use(exampleMiddleware3) mux.HandleFunc("/admin/passwords", exampleHandlerFunc4, "GET") }) }) A note on performance I haven't done any benchmarking against other routers, so I can't speak about the relative performance of Flow. What I can say it has been plenty fast enough for all of my use-cases so far and not a hot spot when profiling my applications under load.

• How to use go run to manage tool dependencies
  May 10, 2022

  When you're working on a project it's common for there to be some developer tooling dependencies. These aren't code dependencies, but rather tools that you run as part of the development, testing, build or deployment processes. For example, you might use golang.org/x/text/cmd/gotext in conjunction with go:generate to generate message catalogs for translation, or honnef.co/go/tools/cmd/staticcheck to perform static analysis on your code before committing a change. This raises a couple of interesting questions — especially in a team environment. How do you make sure that everyone has the necessary tools installed on their machines? And that the tools they are using are all the same version? Until Go 1.17, the convention for managing this was to create a tools.go file in your project containing import statements for the different tools and a //go:build tools build constraint. If you're not already familiar with this approach, it's described in the official Go Wiki. But since Go 1.17 there is an alternative approach you can take. It has pros and cons compared to the tools.go approach, but it's worth knowing about and may be a good fit for some projects. It hinges on the fact that go run now allows you to execute a specific version of a remote package. From the 1.17 release notes: go run now accepts arguments with version suffixes (for example, go run example.com/cmd@v1.0.0). This causes go run to build and run packages in module-aware mode, ignoring the go.mod file in the current directory or any parent directory, if there is one. In other words, you can use go run package@version to execute a remote package when you are outside of a module, or inside of a module even if the package isn't listed in the go.mod file. It's also useful as a quick way to run an executable package without installing it. Instead of this: $ go install honnef.co/go/tools/cmd/staticcheck@v0.3.1 $ staticcheck ./... You can now just do this: $ go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... Important: When you execute go run package@version the necessary modules will be downloaded and cached on your machine in the module cache. So when you execute the same go run command later, the cache will be used (rather than everything being downloaded again) and it should complete faster. Using with go:generate Let's take a look at an example where we use the golang.org/x/tools/cmd/stringer tool in conjunction with go:generate to generate String() methods for some iota constants. If you'd like to follow along, please run the following commands: $ mkdir tools $ go mod init example.com/tools $ touch main.go And then add the following code to main.go: File: main.go package main import "fmt" //go:generate go run golang.org/x/tools/cmd/stringer@v0.1.10 -type=Level type Level int const ( Info Level = iota Error Fatal ) func main() { fmt.Printf("%s: Hello world!\n", Info) } The important thing here is the //go:generate line. When you run go generate on this file, it will in turn use go run to execute v0.1.10 of the golang.org/x/tools/cmd/stringer package. Let's try it out: $ go generate . go: downloading golang.org/x/tools v0.1.10 go: downloading golang.org/x/sys v0.0.0-20211019181941-9d821ace8654 go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 You should see that the necessary modules are downloaded and then the go:generate command finishes executing successfully — resulting in a new level_string.go file being generated and a working application. Like so: $ ls go.mod level_string.go main.go $ go run . Info: Hello world! Using in a Makefile You can also use the go run package@version pattern to execute tools from your scripts or Makefiles. To illustrate, let's create a Makefile with an audit task that executes a specific version of the staticcheck tool. $ touch Makefile File: Makefile .PHONY: audit audit: go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... If you run make audit, the necessary modules will be downloaded and the staticcheck tool should complete its checks successfully. $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... go: downloading honnef.co/go/tools v0.3.1 go: downloading golang.org/x/tools v0.1.11-0.20220316014157-77aa08bb151a go: downloading golang.org/x/exp/typeparams v0.0.0-20220218215828-6cf2b201936e go: downloading github.com/BurntSushi/toml v0.4.1 If you run it for a second time, you'll see that the module cache is used and it should finish much faster. $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... Pros and cons In terms of positives, go run package@version has a couple of nice advantages over the tools.go approach: It's simpler to set up and requires less code — no tools.go file is needed, there are no build constraints, and no aliased imports. It avoids polluting your dependency graph with things that your binaries do not actually depend on. In terms of negatives: If you have the same go run package@version command in multiple places throughout your codebase and want to upgrade to a newer version, then you need to update all of the commands manually (or use sed or find-and-replace). With the tools.go approach you only need to update your go.mod file by running go get package@newversion. With the tools.go approach it's possible to verify that cached code in your module cache hasn't been changed by running go mod verify. I'm not aware of an equivalent check for go run package@version (if you know of a way to do this, please let me know!). From my limited testing, it seems to be possible to edit the cached code in the module cache on your machine, and go run package@version will use this edited code without complaining. If you are working offline, then go run package@version may fail with a dial tcp: lookup proxy.golang.org: Temporary failure in name resolution error because it can't reach the Go module mirror — even if there is a copy already in your local module cache. Similar to this: $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... go: honnef.co/go/tools/cmd/staticcheck@v0.3.1: honnef.co/go/tools/cmd/staticcheck@v0.3.1: Get "https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.3.1.info": dial tcp: lookup proxy.golang.org: Temporary failure in name resolution make: *** [Makefile:4: audit] Error 1 As far as I can see this isn't a problem when you use the tools.go approach, although you can work around it fairly easily by setting the GOPROXY environment variable to direct while you are offline. Doing this will force go run to bypass the Go module mirror and use the cached module on your machine straight away. $ export GOPROXY=direct $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./...

• Change URL query params in Go
  Nov 28, 2021

  In this short post we're going to discuss how to add, modify or delete URL query string parameters in Go. To illustrate, we'll look at how to change this URL: https://example.com?name=alice&age=28&gender=female To this: https://example.com?name=alice&age=29&occupation=carpenter If you want to change the URL query string in place: // Use url.Parse() to parse a string into a *url.URL type. If your URL is // already a url.URL type you can skip this step. urlA, err := url.Parse("https://example.com?name=alice&age=28&gender=female") if err != nil { log.Fatal(err) } // Use the Query() method to get the query string params as a url.Values map. values := urlA.Query() // Make the changes that you want using the Add(), Set() and Del() methods. If // you want to retrieve or check for a specific parameter you can use the Get() // and Has() methods respectively. values.Add("occupation", "carpenter") values.Del("gender") values.Set("age", strconv.Itoa(29)) // Use the Encode() method to transform the url.Values map into a URL-encoded // string (like "age=29&name=alice...") and assign it back to the URL. Note // that the encoded values will be sorted alphabetically based on the parameter // name. urlA.RawQuery = values.Encode() fmt.Printf("urlA: %s", urlA.String()) Running this will print out: urlA: https://example.com?age=29&name=alice&occupation=carpenter If you want to create a clone of the URL but with a different query string, while leaving the original URL unchanged, you need to create a copy of the original url.URL struct first. There are a couple of ways to do this. You can either re-parse the URL, or you can dereference the original url.URL and make a copy, like so: // This is equivalent to: var newUrl url.URL = *originalUrl newUrl := *originalUrl When you do this, you create a new newURL variable of type url.URL which is initialized to the (dereferenced) value of *originalURL. This means that newURL has a different address in memory to originalURL. Putting this together, the pattern for creating a new URL with different parameters is: urlA, err := url.Parse("https://example.com?name=alice&age=28&gender=female") if err != nil { log.Fatal(err) } // Make a copy of the original url.URL. urlB := *urlA // Make the param changes to the new url.URL type... values := urlB.Query() values.Add("occupation", "carpenter") values.Del("gender") values.Set("age", strconv.Itoa(29)) urlB.RawQuery = values.Encode() fmt.Printf("urlA: %s\n", urlA.String()) // This will be unchanged. fmt.Printf("urlB: %s\n", urlB.String()) // This will have the new params. Running this will print out: urlA: https://example.com?name=alice&age=28&gender=female urlB: https://example.com?age=29&name=alice&occupation=carpenter As a side note, you can use this technique any time you want to 'clone' a URL and make changes to it. For example to create a clone of a URL with a different path, you can do this: urlA, err := url.Parse("https://example.com/foo") if err != nil { log.Fatal(err) } urlB := *urlA urlB.Path = "/bar" fmt.Printf("%s\n", urlA.String()) // Prints https://example.com/foo fmt.Printf("%s\n", urlB.String()) // Prints https://example.com/bar

• I18n in Go: Managing translations
  Aug 25, 2021

  Recently I've been building a fully internationalized (i18n) and localized (l10n) web application for the first time with Go's golang.org/x/text packages. I've found that the packages and tools that live under golang.org/x/text are really effective and well designed, although it's been a bit of a challenge to figure out how to put it all together in a real application. Note: Just in case you're not already aware, the packages that live under golang.org/x are part of the official Go Project but outside the main Go standard library tree. They are held to looser standards that the standard library packages, which means they aren't subject to the Go compatibility promise (i.e. their APIs might change), and documentation may not always be complete. In this tutorial I want to explain how you can use golang.org/x/text packages to manage translations in your application. Specifically: How to use the golang.org/x/text/language and golang.org/x/text/message packages to print translated messages from your Go code. How to use the gotext tool to automatically extract messages for translation from your code into JSON files. How to use gotext to parse translated JSON files and create a catalog containing translated messages. How to manage variables in messages and provided pluralized versions of translations. What we'll be building To help put this into context, we're going to create a simple pre-launch website for an imaginary online bookstore. We'll start off slowly and build up the code step-by-step. Our application will have just a single home page, and we'll localize the page content based on a locale identifier at the start of the URL path. We'll set up our application to support three different locales: the United Kingdom, Germany, and the French-speaking part of Switzerland. URL Localized for localhost:4018/en-gb United Kingdom localhost:4018/de-de Germany localhost:4018/fr-ch Switzerland (French-speaking) We're going to follow a common convention and use BCP 47 language tags as the locale identifier in our URLs. Simplifying things hugely for the sake of this tutorial, BCP 47 language tags typically take the format {language}-{region}. The language part is a ISO 639-1 code and the region is a two-letter country code from ISO_3166-1. It's conventional to uppercase the region (like en-GB), but BCP 47 tags are technically case-insensitive and it's OK for us to use all-lowercase versions in our URLs. Scaffolding a web application If you'd like to follow along with the application build, go ahead and run the following commands to setup a new project directory. $ mkdir bookstore $ cd bookstore $ go mod init bookstore.example.com go: creating new go.mod: module bookstore.example.com At this point, you should have a go.mod file in the root of the project directory with the module path bookstore.example.com. Next create a new cmd/www directory to hold the code for the bookstore web application, and add main.go and handlers.go files like so: $ mkdir -p cmd/www $ touch cmd/www/main.go cmd/www/handlers.go Your project directory should now look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go └── go.mod Let's begin in the cmd/www/main.go file and add the code to declare our application routes and start a HTTP server. Because our application URL paths will always use a (dynamic) locale as a prefix — like /en-gb/bestsellers or /fr-ch/bestsellers — it's simplest if our application uses a third-party router which supports dynamic values in URL path segments. I'm going to use pat, but feel free to use an alternative like chi or gorilla/mux if you prefer. Note: If you're not sure which router to use in your project, you might like to take a look at my comparison of Go routers blog post. OK, open up the main.go file and add the following code: File: cmd/www/main.go package main import ( "log" "net/http" "github.com/bmizerany/pat" ) func main() { // Initialize a router and add the path and handler for the homepage. mux := pat.New() mux.Get("/:locale", http.HandlerFunc(handleHome)) // Start the HTTP server using the router. log.Print("starting server on :4018...") err := http.ListenAndServe(":4018", mux) log.Fatal(err) } Then in the cmd/www/handlers.go file, add a handleHome() function which extracts the locale identifer from the URL path and echoes it in the HTTP response. File: cmd/www/handlers.go package main import ( "fmt" "net/http" ) func handleHome(w http.ResponseWriter, r *http.Request) { // Extract the locale from the URL path. This line of code is likely to // be different for you if you are using an alternative router. locale := r.URL.Query().Get(":locale") // If the locale matches one of our supported values, echo the locale // in the response. Otherwise send a 404 Not Found response. switch locale { case "en-gb", "de-de", "fr-ch": fmt.Fprintf(w, "The locale is %s\n", locale) default: http.NotFound(w, r) } } Once that's done, run go mod tidy to tidy your go.mod file and download any necessary dependencies, and then run the web application. $ go mod tidy go: finding module for package github.com/bmizerany/pat go: found github.com/bmizerany/pat in github.com/bmizerany/pat v0.0.0-20210406213842-e4b6760bdd6f $ go run ./cmd/www/ 2021/08/21 21:22:57 starting server on :4018... If you make some requests to the application using curl, you should find that the appropriate locale is echoed back to you like so: $ curl localhost:4018/en-gb The locale is en-gb $ curl localhost:4018/de-de The locale is de-de $ curl localhost:4018/fr-ch The locale is fr-ch $ curl localhost:4018/da-DK 404 page not found Extracting and translating text content Now that we've laid the groundwork for our web application, let's get into the core of this tutorial and update the handleHome() function so that it renders a "Welcome!" message translated for the specific locale. In this project we'll use British English (en-GB) as the default 'source' or 'base' language in our application, but we'll want to render a translated version of the welcome message in German and French for the other locales. To do this, we'll need to import the golang.org/x/text/language and golang.org/x/text/message packages and update our handleHome() function to do the following two things: Construct a language.Tag which identifies the target language that we want to translate the message in to. The language package contains some pre-defined tags for common language variants, but I find that it's easier to use the language.MustParse() function to create a tag. This let's you create a language.Tag for any valid BCP 47 value, like language.MustParse("fr-CH"). Once you have a language tag, you can use the message.NewPrinter() function to create a message.Printer instance that prints out messages in that specific language. If you're following along, please go ahead and update your cmd/www/handlers.go file to contain the following code: File: cmd/www/handlers.go package main import ( "net/http" "golang.org/x/text/language" "golang.org/x/text/message" ) func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") // Declare variable to hold the target language tag. var lang language.Tag // Use language.MustParse() to assign the appropriate language tag // for the locale. switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } // Initialize a message.Printer which uses the target language. p := message.NewPrinter(lang) // Print the welcome message translated into the target language. p.Fprintf(w, "Welcome!\n") } Again, run go mod tidy to download the necessary dependencies… $ go mod tidy go: finding module for package golang.org/x/text/message go: finding module for package golang.org/x/text/language go: downloading golang.org/x/text v0.3.7 go: found golang.org/x/text/language in golang.org/x/text v0.3.7 go: found golang.org/x/text/message in golang.org/x/text v0.3.7 And then run the application: $ go run ./cmd/www/ 2021/08/21 21:33:52 starting server on :4018... When you make a request to any of the supported URLs, you should now see the (untranslated) welcome message like this: $ curl localhost:4018/en-gb Welcome! $ curl localhost:4018/de-de Welcome! $ curl localhost:4018/fr-ch Welcome! So in all cases we're seeing the "Welcome!" message in our en-GB source language. That's because we still need to provide Go's message package with the actual translations that we want to use. Without the actual translations, it falls back to displaying the message in the source language. There are a number of ways to provide Go's message package with translations, but for most non-trivial applications it's probably sensible to use some automated tooling to help you manage the task. Fortunately, Go provides the gotext tool to assist with this. Note: The gotext tool we're using is the one from golang.org/x/text/cmd/gotext. It shouldn't be confused with the github.com/leonelquinteros/gotext package (which is designed to work with GNU gettext utilities and PO/MO files). If you're following along, please use go install to install the gotext executable on your machine: $ go install golang.org/x/text/cmd/gotext@latest All being well, the tool should be installed to your $GOBIN directory on your system path and you can run it like so: $ which gotext /home/alex/go/bin/gotext $ gotext gotext is a tool for managing text in Go source code. Usage: gotext command [arguments] The commands are: update merge translations and generate catalog extract extracts strings to be translated from code rewrite rewrites fmt functions to use a message Printer generate generates code to insert translated messages Use "gotext help [command]" for more information about a command. Additional help topics: Use "gotext help [topic]" for more information about that topic. I really like the gotext tool — it's functionality is excellent — but there are a couple of important things to point out before we carry on. The first thing is that go text is designed to work in conjunction with go generate, not as a standalone command-line tool. You can run it as a standalone tool, but weird things happen and it's a lot smoother if you use it in the way it's intended. The other thing is that documentation and help functionality is basically non-existent. The best guidance on how to use it are the examples in the repository and, probably, this article that you're reading right now. There is an open issue about the lack of help functionality, and hopefully this is something that will improve in the future. In this tutorial, we're going to store the all the code relating to translations in a new internal/translations package. We could keep all the translation code for our web application under cmd/www instead, but in my (limited) experience I've found that using a separate internal/translations package is better. It helps separate concerns and also makes it possible to reuse the same translations across different applications in the same project. YMMV. If you're following along, go ahead and create that new directory and a translations.go file like so: $ mkdir -p internal/translations $ touch internal/translations/translations.go At this point, your project structure should look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go ├── go.mod ├── go.sum └── internal └── translations └── translations.go Next, let's open up the internal/translations/translations.go file and add a go generate command which uses gotext to extract the messages for translation from our application. File: internal/translations/translations.go package translations //go:generate gotext -srclang=en-GB update -out=catalog.go -lang=en-GB,de-DE,fr-CH bookstore.example.com/cmd/www There's a lot going on in this command, so let's quickly break it down. The -srclang flag contains the BCP 47 tag for the source (or 'base') language that we are using in the application. In our case, the source language is en-GB. update is thegotext function that we want to execute. As well as update there are extract, rewrite and generate functions, but in the translation workflow for a web application the only one you actually need is update. The -out flag contains the path that you want the message catalog to be output to. This path should be relative to the file containing the go generate command. In our case, we've set the value to catalog.go, which means that the message catalog will be output to a new internal/translations/catalog.go file. We'll talk more about message catalogs and explain what they are shortly. The -lang flag contains a comma-separated list of the BCP 47 tags that you want to create translations for. You don't need to include the source language here, but (as we'll demonstrate later in this article) it can be helpful for dealing with pluralization of text content. Lastly, we have the fully-qualified module path for the package(s) that you want to create translations for (in this case bookstore.example.com/cmd/www). You can list multiple packages if necessary, separated by a whitespace character. When we execute this go generate command, gotext will walk the code for the cmd/www application and look for all calls to a message.Printer†. It then extracts the relevant message strings and outputs them to some JSON files for translation. † Important: It's critical to note when gotext walks your code it actually only looks for calls to message.Printer.Printf(), Fprintf() and Sprintf() — basically the three methods that end with an f. It ignores all other methods such as Sprint() or Println(). You can see this behavior in the gotext implementation here. OK, let's put this into action and call go generate on our translations.go file. In turn, this will execute the gotext command that we included at the top of that file. $ go generate ./internal/translations/translations.go de-DE: Missing entry for "Welcome!". fr-CH: Missing entry for "Welcome!". Cool, this looks like we're getting somewhere. We've got some useful feedback to indicate that we are missing the necessary German and French translations for our "Welcome!" message. If you take a look at the directory structure for your project, it should now look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go ├── go.mod ├── go.sum └── internal └── translations ├── catalog.go ├── locales │ ├── de-DE │ │ └── out.gotext.json │ ├── en-GB │ │ └── out.gotext.json │ └── fr-CH │ └── out.gotext.json └── translations.go We can see that the go generate command has automatically generated an internal/translations/catalog.go file for us (which we'll look at in a minute), and a locales folder containing out.gotext.json files for each of our target languages. Let's take a look at the internal/translations/locales/de-DE/out.gotext.json file: File: internal/translations/locales/de-DE/out.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "" } ] } In this JSON file, the relevant BCP 47 language tag is defined at the top of the file, followed by a JSON array of the messages which require translation. The message value is the text for translation in the source language, and the (currently empty) translation value is where we should enter appropriate German translation. It's important to emphasize that you don't edit this file in place. Instead, the workflow for adding a translation goes like this: You generate the out.gotext.json files containing the messages which need to be translated (which we've just done). You send these files to a translator, who edits the JSON to include the necessary translations. They then send the updated files back to you. You then save these updated files with the name messages.gotext.json in the folder for the appropriate language. For demonstration purposes, let's quickly simulate this workflow by copying the out.gotext.json files to messages.gotext.json files, and updating them to include the translated messages like so: $ cp internal/translations/locales/de-DE/out.gotext.json internal/translations/locales/de-DE/messages.gotext.json $ cp internal/translations/locales/fr-CH/out.gotext.json internal/translations/locales/fr-CH/messages.gotext.json File: internal/translations/locales/de-DE/messages.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" } ] } File: internal/translations/locales/fr-CH/messages.gotext.json { "language": "fr-CH", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Bienvenu !" } ] } If you like, you can also take a look at the out.gotext.json file for our en-GB source language. You'll see that the translation value for the message has been auto-filled for us. File: internal/translations/locales/en-GB/messages.gotext.json { "language": "en-GB", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Welcome!", "translatorComment": "Copied from source.", "fuzzy": true } ] } The next step is to run our go generate command again. This time, it should execute without any warning messages about missing translations. $ go generate ./internal/translations/translations.go Now it's a good time to take a look at the internal/translations/catalog.go file, which is automatically generated for us by the gotext update command. This file contains a message catalog, which is — very roughly speaking — a mapping of messages and their relevant translations for each target language. Let's take a quick look inside the file: File: internal/translations/catalog.go // Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT. package translations import ( "golang.org/x/text/language" "golang.org/x/text/message" "golang.org/x/text/message/catalog" ) type dictionary struct { index []uint32 data string } func (d *dictionary) Lookup(key string) (data string, ok bool) { p, ok := messageKeyToIndex[key] if !ok { return "", false } start, end := d.index[p], d.index[p+1] if start == end { return "", false } return d.data[start:end], true } func init() { dict := map[string]catalog.Dictionary{ "de_DE": &dictionary{index: de_DEIndex, data: de_DEData}, "en_GB": &dictionary{index: en_GBIndex, data: en_GBData}, "fr_CH": &dictionary{index: fr_CHIndex, data: fr_CHData}, } fallback := language.MustParse("en-GB") cat, err := catalog.NewFromMap(dict, catalog.Fallback(fallback)) if err != nil { panic(err) } message.DefaultCatalog = cat } var messageKeyToIndex = map[string]int{ "Welcome!\n": 0, } var de_DEIndex = []uint32{ // 2 elements 0x00000000, 0x00000011, } // Size: 32 bytes const de_DEData string = "\x04\x00\x01\n\f\x02Willkommen!" var en_GBIndex = []uint32{ // 2 elements 0x00000000, 0x0000000e, } // Size: 32 bytes const en_GBData string = "\x04\x00\x01\n\t\x02Welcome!" var fr_CHIndex = []uint32{ // 2 elements 0x00000000, 0x00000010, } // Size: 32 bytes const fr_CHData string = "\x04\x00\x01\n\v\x02Bienvenu !" // Total table size 143 bytes (0KiB); checksum: 385F6E56 I don't want to dwell on the details here, because it's OK for use to treat this file as something of a 'black box', and — as warned by the comment at the top of the file — we shouldn't make any changes to it directly. But the most important thing to point out is that this file contains an init() function which, when called, initializes a new message catalog containing all our translations and mappings. It then sets this as the default message catalog by assigning it to the message.DefaultCatalog global variable. When we call one of the message.Printer functions, the printer will lookup the relevant translation from the default message catalog for printing. This is really nice, because it means that all our translations are stored in memory at runtime, and any lookups are very fast and efficient. So, if we take a step back for a moment, we can see that the gotext update command that we're using with go generate actually does two things. One — it walks the code in our cmd/www application and extracts the necessary strings for translation into the out.gotext.json files; and two — it also parses any messages.gotext.json files (if present) and updates the message catalog accordingly. The final step in getting this working is to import the internal/translations package in our cmd/www/handlers.go file. This will ensure that the init() function in internal/translations/translations.go is called, and the default message catalog is updated to be the one containing our translations. Because we won't actually be referencing anything in the internal/translations package directly, we'll need to alias the import path to the blank identifer _ to prevent the Go compiler from complaining. Go ahead and do that now: File: cmd/www/handlers.go package main import ( "net/http" // Import the internal/translations package, so that its init() // function is called. _ "bookstore.example.com/internal/translations" "golang.org/x/text/language" "golang.org/x/text/message" ) func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") var lang language.Tag switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") } Alright, let's try this out! When your restart the application and try making some requests, you should now see the "Welcome!" message translated into the appropriate language. $ curl localhost:4018/en-GB Welcome! $ curl localhost:4018/de-de Willkommen! $ curl localhost:4018/fr-ch Bienvenu ! Using variables in translations Now that we've got the basic translations working in our application, let's move on to something a bit more advanced and look at how to manage translations with interpolated variables in them. To demonstrate, we'll update the HTTP response from our handleHome() function to include a "{N} books available" line, where {N} is an integer containing the number of books in our imaginary bookstore. File: cmd/www/handlers.go package main ... func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") var lang language.Tag switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } // Define a variable to hold the number of books. In a real application // this would probably be retrieved by making a database query or // something similar. var totalBookCount = 1_252_794 p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") // Use the Fprintf() function to include the new message in the HTTP // response, with the book count as in interpolated integer value. p.Fprintf(w, "%d books available\n", totalBookCount) } Save the changes, then use go generate to output some new out.gotext.json files. You should see warning messages for the new missing translations like so: $ go generate ./internal/translations/translations.go de-DE: Missing entry for "{TotalBookCount} books available". fr-CH: Missing entry for "{TotalBookCount} books available". Let's take a look at the de-DE/out.gotext.json file: File: internal/translations/locales/de-DE/out.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": "", "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } The first thing to point out here is that the translation for our "Welcome!" message has been persisted across the workflow and is already present in the out.gotext.json file. This is obviously really important, because it means that when we send the file to the translator they won't need to provide the translation again. The second thing is that there is now an entry for our new message. We can see that this has the form "{TotalBookCount} books available", with the (capitalized) variable name from our Go code being used as the placeholder parameter. You should keep this in mind when writing your code, and try to use sensible and descriptive variable names that will make sense to your translators. The placeholders array also provides additional information about each placeholder value, the most useful part probably being the type value (which in this case tells the translator that the TotalBookCount value is an integer). So the next step is to send these new out.gotext.json files off to a translator for translation. Again, we'll simulate that here by copying them to messages.gotext.json files and adding the translations like so: $ cp internal/translations/locales/de-DE/out.gotext.json internal/translations/locales/de-DE/messages.gotext.json $ cp internal/translations/locales/fr-CH/out.gotext.json internal/translations/locales/fr-CH/messages.gotext.json File: internal/translations/locales/de-DE/messages.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": "{TotalBookCount} Bücher erhältlich", "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } File: internal/translations/locales/fr-CH/messages.gotext.json { "language": "fr-CH", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Bienvenu !" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": "{TotalBookCount} livres disponibles", "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } Make sure that both messages.gotext.json files are saved, and then run go generate to update our message catalog. This should run without any warnings. $ go generate ./internal/translations/translations.go When you restart the cmd/www application and make some HTTP requests again, you should now see the new translated messages like so: $ curl localhost:4018/en-GB Welcome! 1,252,794 books available $ curl localhost:4018/de-de Willkommen! 1.252.794 Bücher erhältlich $ curl localhost:4018/fr-ch Bienvenu ! 1 252 794 livres disponibles Now this is really cool. As we'll as the translations being applied by our message.Printer, it's also smart enough to output the interpolated integer value with the correct number formatting for each language. We can see here that our en-GB locale uses the "," character as a thousands separator, whereas de-DE uses "." and fr-CH uses the whitespace " ". A similar thing is done for decimal separators too. Dealing with pluralization's This is working nicely, but what happens if there is only 1 book available in our bookstore? Let's update the handleHome() function so that the totalBookCount value is 1: File: cmd/www/handlers.go package main ... func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") var lang language.Tag switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } // Set the total book count to 1. var totalBookCount = 1 p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") p.Fprintf(w, "%d books available\n", totalBookCount) } (I know this is a bit of a tenuous example, but it helps illustrate Go's pluralization functionality without much extra code, so bear with me!) You can probably imagine what happens when we restart the application and make a request to localhost:4018/en-gb now. $ curl localhost:4018/en-gb Welcome! 1 books available That's right, we see the message "1 books available", which isn't correct English because of the plural noun books. It would be better if this message read 1 book available or — even better — One book available instead. Happily, it's possible for us to specify alternative translations based on the value of an interpolated variable in our messages.gotext.json files. Let's start by demonstrating this for our en-GB locale. If you're following along, copy the en-GB/out.gotext.json file to en-GB/messages.gotext.json: $ cp internal/translations/locales/en-GB/out.gotext.json internal/translations/locales/en-GB/messages.gotext.json And then update it like so: File: internal/translations/locales/en-GB/messages.gotext.json { "language": "en-GB", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Welcome!", "translatorComment": "Copied from source.", "fuzzy": true }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": { "select": { "feature": "plural", "arg": "TotalBookCount", "cases": { "=1": { "msg": "One book available" }, "other": { "msg": "{TotalBookCount} books available" } } } }, "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } Now, rather than the translation value being a simple string we have set it to a JSON object that instructs the message catalog to use different translations depending on the value of the TotalBookCount placeholder. The key part here is the cases value, which contains the translations to use for different values of the placeholder. The supported case rules are: Case Description "=x" Where x is an integer that equals the value of the placeholder " Where x is an integer that is larger than the value of the placeholder "other" All other cases (a bit like default in a Go switch statement) Note: If you look at the documentation for the golang.org/x/text/feature/plural package (which is what gotext uses behind the scenes when generating the message catalog), you'll see that it also mentions the case rules "zero", "one", "two", "few", and "many". However, these rules aren't supported for all possible target languages, and you may get an error like gotext: generation failed: error: plural: form "many" not supported for language "de-DE" if you try to use them. It seems to be safer to stick with the three case rules in the table above. Additionally, it's important to be aware that the range of allowed values for x in the "=x" and " case rules is 0 to 32767. Trying to use something outside of that range will result in an error. There's an open issue about these behaviors here. Let's complete work this by updating the messages.gotext.json files for our de-DE and fr-CH languages to include the appropriate pluralized variations, like so: File: internal/translations/locales/de-DE/messages.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": { "select": { "feature": "plural", "arg": "TotalBookCount", "cases": { "=1": { "msg": "Ein Buch erhältlich" }, "other": { "msg": "{TotalBookCount} Bücher erhältlich" } } } }, "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } File: internal/translations/locales/fr-CH/messages.gotext.json { "language": "fr-CH", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Bienvenu !" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": { "select": { "feature": "plural", "arg": "TotalBookCount", "cases": { "=1": { "msg": "Un livre disponible" }, "other": { "msg": "{TotalBookCount} livres disponibles" } } } }, "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } Once those files are saved, use go generate again to update the message catalog: $ go generate ./internal/translations/translations.go And if you restart the web application and make some HTTP requests, you should now see the appropriate message for 1 book: $ curl localhost:4018/en-GB Welcome! One book available $ curl localhost:4018/de-de Willkommen! Ein Buch erhältlich $ curl localhost:4018/fr-ch Bienvenu ! Un livre disponible If you like, you can revert the totalBookCount variable back to a larger number... File: cmd/www/handlers.go package main ... func handleHome(w http.ResponseWriter, r *http.Request) { ... // Revert the total book count. var totalBookCount = 1_252_794 p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") p.Fprintf(w, "%d books available\n", totalBookCount) } And when you restart the application and make another request, you should see the "other" version of our message: $ curl localhost:4018/de-de Willkommen! 1.252.794 Bücher erhältlich Creating a localizer abstraction In the final part of this article we're going to create a new internal/localizer package which abstracts all our code for dealing with languages, printers and translations. If you're following along, go ahead and create a new internal/localizer directory containing a localizer.go file. $ mkdir -p internal/localizer $ touch internal/localizer/localizer.go At this point, your project structure should look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go ├── go.mod ├── go.sum └── internal ├── localizer │ └── localizer.go └── translations ├── catalog.go ├── locales │ ├── de-DE │ │ ├── messages.gotext.json │ │ └── out.gotext.json │ ├── en-GB │ │ ├── messages.gotext.json │ │ └── out.gotext.json │ └── fr-CH │ ├── messages.gotext.json │ └── out.gotext.json └── translations.go And then add the following code to the new localizer.go file: File: internal/localizer/localizer.go package localizer import ( // Import the internal/translations so that it's init() function // is run. It's really important that we do this here so that the // default message catalog is updated to use our translations // *before* we initialize the message.Printer instances below. _ "bookstore.example.com/internal/translations" "golang.org/x/text/language" "golang.org/x/text/message" ) // Define a Localizer type which stores the relevant locale ID (as used // in our URLs) and a (deliberately unexported) message.Printer instance // for the locale. type Localizer struct { ID string printer *message.Printer } // Initialize a slice which holds the initialized Localizer types for // each of our supported locales. var locales = []Localizer{ { // Germany ID: "de-de", printer: message.NewPrinter(language.MustParse("de-DE")), }, { // Switzerland (French speaking) ID: "fr-ch", printer: message.NewPrinter(language.MustParse("fr-CH")), }, { // United Kingdom ID: "en-gb", printer: message.NewPrinter(language.MustParse("en-GB")), }, } // The Get() function accepts a locale ID and returns the corresponding // Localizer for that locale. If the locale ID is not supported then // this returns `false` as the second return value. func Get(id string) (Localizer, bool) { for _, locale := range locales { if id == locale.ID { return locale, true } } return Localizer{}, false } // We also add a Translate() method to the Localizer type. This acts // as a wrapper around the unexported message.Printer's Sprintf() // function and returns the appropriate translation for the given // message and arguments. func (l Localizer) Translate(key message.Reference, args ...interface{}) string { return l.printer.Sprintf(key, args...) } Note: Notice here that we're initializing a single message.Printer for each locale at startup, and these will be used concurrently by our web application handlers. Although the golang.org/x/text/message documentation doesn't say that message.Printer is safe for concurrent use, I checked with Marcel van Lohuizen (the lead developer of the golang.org/x/text packages) and he confirmed that message.Printer is intended to be used concurrently and is concurrency safe (so long as access to any write destination is synchronized). Next let's update the cmd/www/handlers.go file to use our new Localizer type, and — while we're at it — let's also make our handleHome() function render an additional "Launching soon!" message. File: cmd/www/handlers.go package main import ( "fmt" // New import "net/http" "bookstore.example.com/internal/localizer" // New import ) func handleHome(w http.ResponseWriter, r *http.Request) { // Initialize a new Localizer based on the locale ID in the URL. l, ok := localizer.Get(r.URL.Query().Get(":locale")) if !ok { http.NotFound(w, r) return } var totalBookCount = 1_252_794 // Update these to use the new Translate() method. fmt.Fprintln(w, l.Translate("Welcome!")) fmt.Fprintln(w, l.Translate("%d books available", totalBookCount)) // Add an additional "Launching soon!" message. fmt.Fprintln(w, l.Translate("Launching soon!")) } It's worth pointing out that our use of the Translate() method here isn't just some syntactic sugar. You might remember earlier that I wrote the following warning: It's critical to note when gotext walks your code it actually only looks for calls to message.Printer.Printf(), Fprintf() and Sprintf() — basically the three methods that end with an f. It ignores all other methods such as Sprint() or Println(). By having all our translations go through the Translate() method — which uses Sprintf() behind-the-scenes — we avoid the scenario where you accidentally use a method like Sprint() or Println() and gotext doesn't extract the message to the out.gotext.json files. Let's try this out and run go generate again: $ go generate ./internal/translations/translations.go de-DE: Missing entry for "Launching soon!". fr-CH: Missing entry for "Launching soon!". So this is really smart. We can see that gotext has been clever enough to walk our entire codebase and identify what strings need to be translated, even when we abstract the message.Printer.Sprintf() call to a helper function in a different package. This is awesome, and one of the things that I really appreciate about the gotext tool. If you're following along, please go ahead and copy the out.gotext.json files to message.gotext.json files, and add the necessary translations for the new "Launching soon!" message. Then remember to run go generate again and restart the web application. When you make some HTTP requests again now, your responses should look similar to this: $ curl localhost:4018/en-gb Welcome! 1,252,794 books available Launching soon! $ curl localhost:4018/de-de Willkommen! 1.252.794 Bücher erhältlich Bald verfügbar! $ curl localhost:4018/fr-ch Bienvenu ! 1 252 794 livres disponibles Bientôt disponible ! Additional information Conflicting routes At this start of this post I'd deliberately didn't recommending using httprouter, despite it being an excellent and popular router. This is because using a dynamic locale as the first part of a URL path is likely to result in conflicts with other application routes which don't require a locale prefix, like /static/css/main.css or /admin/login. The httprouter package doesn't allow conflicting routes, which makes using it awkward in this scenario. If you do want to use httprouter, or want to avoid conflicting routes in your application, you could pass the locale as a query string parameter instead like /category/travel?locale=gb.

• How to correctly use Basic Authentication in Go
  Jun 21, 2021

  When searching for examples of HTTP basic authentication with Go, every result I found unfortunately contained code which was either out-of-date (i.e. doesn't use the r.BasicAuth() functionality that was introduced in Go 1.4) or failed to mitigate the risk of timing attacks. So in this post, I'd like to discuss how to use it correctly in your Go applications. We'll start with a bit of background information, but if you're not interested in that you can skip straight to the code. What is basic authentication? When should I use it? As a developer, you're probably already familiar with the prompt that web browsers show when you visit a URL that is protected with basic authentication. When you input a username and password into this prompt, the web browser will send a HTTP request to the server containing an Authorization header — similar to this: Authorization: Basic YWxpY2U6cGE1NXdvcmQ= The Authorization header value is made up of the string Basic followed by the username and password in the format username:password and base-64 encoded. In this specific example, YWxpY2U6cGE1NXdvcmQ= is the base-64 encoding of the value alice:pa55word. When the server receives this request, it can decode the username and password from the Authorization header and check that they are valid. If the credentials are not valid, the server can return a 401 Unauthorized response and the browser can redisplay the prompt. Basic authentication can be used in lots of different scenarios, but it's often a good fit for when you have a low-value resource and want a quick and easy way to protect it from prying eyes. To help keep things secure you should: Only ever use it over HTTPS connections. If you don't use HTTPS, the Authorization header can potentially be intercepted and decoded by an attacker, who can then use the username and password to gain access to your protected resources. Use a strong password that is difficult for attackers to guess or brute-force. Consider adding rate limiting to your application, to make it harder for an attacker to brute-force the credentials. On the client side, basic auth is supported out-of-the-box by most programming languages and command-line tools such as curl and wget, as well as web browsers. Protecting a web application Probably the simplest way to use basic authentication in your application is to create some middleware. In this middleware we want to do two things: Extract the provided username and password from the request Authorization header, if it exists. The best way to do this is with the r.BasicAuth() method. Compare the provided username and password against the values that you expect. If the username and password are not correct, or the request didn't contain a valid Authorization header, then the middleware should send a 401 Unauthorized response and set a WWW-Authenticate header to inform the client that basic authentication should be used to gain access. Otherwise, the middleware should allow the request to proceed and call the next handler in the chain. When comparing the provided username and password against the expected values, to eliminate the risk of a timing attack you should use Go's subtle.ConstantTimeCompare() function instead of the == operator. Note: In Go (like most languages) the normal == comparison operator will return as soon as it finds a difference between two strings. So if the first character is different, it will return after just looking at one character. In theory, this opens the opportunity for a timing attack where an attacker could make lots of requests to your application, and look at discrepancies in the average response time. The time it takes for them receive a 401 Unauthorized response effectively tells them how many characters they got right. With enough requests, they could build up a picture of the complete username and password. Realistically though, string comparison is so fast that network jitter will obscure any differences in timing, meaning that it is probably impossible to pull off this attack successfully. But... there is some evidence that remote timing attacks are feasible, and given that we can quite easily eliminate this risk completely by using subtle.ConstantTimeCompare(), I think it makes sense to do so. It's also important to be aware that using subtle.ConstantTimeCompare() can leak information about username and password length. To prevent this, we should hash both the provided and expected username and password values using a fast cryptographic hash function like SHA-256 before comparing them. This ensures that both the provided and expected values that we are comparing are equal in length and prevents subtle.ConstantTimeCompare() itself from returning early. Putting that together, the pattern for implementing some middleware looks like this: func basicAuth(next http.HandlerFunc) http.HandlerFunc { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Extract the username and password from the request // Authorization header. If no Authorization header is present // or the header value is invalid, then the 'ok' return value // will be false. username, password, ok := r.BasicAuth() if ok { // Calculate SHA-256 hashes for the provided and expected // usernames and passwords. usernameHash := sha256.Sum256([]byte(username)) passwordHash := sha256.Sum256([]byte(password)) expectedUsernameHash := sha256.Sum256([]byte("your expected username")) expectedPasswordHash := sha256.Sum256([]byte("your expected password")) // Use the subtle.ConstantTimeCompare() function to check if // the provided username and password hashes equal the // expected username and password hashes. ConstantTimeCompare // will return 1 if the values are equal, or 0 otherwise. // Importantly, we should do the work to evaluate both the // username and password before checking the return values to // avoid leaking information. usernameMatch := (subtle.ConstantTimeCompare(usernameHash[:], expectedUsernameHash[:]) == 1) passwordMatch := (subtle.ConstantTimeCompare(passwordHash[:], expectedPasswordHash[:]) == 1) // If the username and password are correct, then call // the next handler in the chain. Make sure to return // afterwards, so that none of the code below is run. if usernameMatch && passwordMatch { next.ServeHTTP(w, r) return } } // If the Authentication header is not present, is invalid, or the // username or password is wrong, then set a WWW-Authenticate // header to inform the client that we expect them to use basic // authentication and send a 401 Unauthorized response. w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`) http.Error(w, "Unauthorized", http.StatusUnauthorized) }) } Important: If you're looking at the code above and thinking "I thought you should never use SHA-256 for hashing passwords...", it's important to emphasize that the username and password are not being hashed for the purpose of storage, they are only being hashed in order to get two equal-length byte slices that can be compared in constant-time. Low collision risk is the important thing here, and a hash like SHA-256 is a good fit for this purpose. You might also be wondering what the realm value is and why we are setting it to "restricted" in the WWW-Authenticate response header. Basically, the realm value is a string which allows you to create partitions of protected space in your application. So, for example, an application could have a "documents" realm and an "admin area" realm, which require different credentials. A web browser (or other type of client) can cache and automatically reuse the same username and password for any requests within the same realm, so that the prompt doesn't need to be shown for every single request. If you don't require multiple partitions for your application, you can set the realm to a single hardcoded value like "restricted", like we have in the code above. For the sake of security and/or flexibility, you may also prefer to store the expected username and password values in environment variables or pass them as command-line flag values when starting the application, rather than hard-coding them into your application. A working example Let's take a quick look at this in the context of a small — but fully functioning — web application. If you'd like to follow along, create a new basic-auth-example directory on your computer, add a main.go file, initialize a module, and create a pair of locally-trusted TLS certificates using the mkcert tool. Like so: $ mkdir basic-auth-example $ cd basic-auth-example $ touch main.go $ go mod init example.com/basic-auth-example go: creating new go.mod: module example.com/basic-auth-example $ mkcert localhost Created a new certificate valid for the following names 📜 - "localhost" The certificate is at "./localhost.pem" and the key at "./localhost-key.pem" ✅ It will expire on 21 September 2023 🗓 $ ls go.mod localhost-key.pem localhost.pem main.go Then add the following code to the main.go file, so that the application reads the expected username and password from environment variables and uses the middleware pattern that we described above. package main import ( "crypto/sha256" "crypto/subtle" "fmt" "log" "net/http" "os" "time" ) type application struct { auth struct { username string password string } } func main() { app := new(application) app.auth.username = os.Getenv("AUTH_USERNAME") app.auth.password = os.Getenv("AUTH_PASSWORD") if app.auth.username == "" { log.Fatal("basic auth username must be provided") } if app.auth.password == "" { log.Fatal("basic auth password must be provided") } mux := http.NewServeMux() mux.HandleFunc("GET /unprotected", app.unprotectedHandler) mux.HandleFunc("GET /protected", app.basicAuth(app.protectedHandler)) srv := &http.Server{ Addr: ":4000", Handler: mux, IdleTimeout: time.Minute, ReadTimeout: 10 * time.Second, WriteTimeout: 30 * time.Second, } log.Printf("starting server on %s", srv.Addr) err := srv.ListenAndServeTLS("./localhost.pem", "./localhost-key.pem") log.Fatal(err) } func (app *application) protectedHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "This is the protected handler") } func (app *application) unprotectedHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "This is the unprotected handler") } func (app *application) basicAuth(next http.HandlerFunc) http.HandlerFunc { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { username, password, ok := r.BasicAuth() if ok { usernameHash := sha256.Sum256([]byte(username)) passwordHash := sha256.Sum256([]byte(password)) expectedUsernameHash := sha256.Sum256([]byte(app.auth.username)) expectedPasswordHash := sha256.Sum256([]byte(app.auth.password)) usernameMatch := (subtle.ConstantTimeCompare(usernameHash[:], expectedUsernameHash[:]) == 1) passwordMatch := (subtle.ConstantTimeCompare(passwordHash[:], expectedPasswordHash[:]) == 1) if usernameMatch && passwordMatch { next.ServeHTTP(w, r) return } } w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`) http.Error(w, "Unauthorized", http.StatusUnauthorized) }) } You should then be able to start the application, using a pair of temporary AUTH_USERNAME and AUTH_PASSWORD environment variables. Like so: $ AUTH_USERNAME=alice AUTH_PASSWORD=p8fnxeqj5a7zbrqp go run . 2021/06/20 16:09:21 starting server on :4000 At this point, if you open your web browser and visit https://localhost:4000/protected you should be greeted by the basic authentication prompt. Alternatively, you can make some requests using curl to verify that the authentication checks are working correctly. $ curl -i https://localhost:4000/unprotected HTTP/2 200 content-type: text/plain; charset=utf-8 content-length: 32 date: Sun, 20 Jun 2021 14:09:56 GMT This is the unprotected handler $ curl -i https://localhost:4000/protected HTTP/2 401 content-type: text/plain; charset=utf-8 www-authenticate: Basic realm="restricted", charset="UTF-8" x-content-type-options: nosniff content-length: 13 date: Sun, 20 Jun 2021 14:09:59 GMT Unauthorized $ curl -i -u alice:p8fnxeqj5a7zbrqp https://localhost:4000/protected HTTP/2 200 content-type: text/plain; charset=utf-8 content-length: 30 date: Sun, 20 Jun 2021 14:10:14 GMT This is the protected handler $ curl -i -u alice:wrongPa55word https://localhost:4000/protected HTTP/2 401 content-type: text/plain; charset=utf-8 www-authenticate: Basic realm="restricted", charset="UTF-8" x-content-type-options: nosniff content-length: 13 date: Sun, 20 Jun 2021 14:15:30 GMT Unauthorized Making a request to a protected resource Finally, if you need to access a protected resource from your Go code as a client, all you need to do is call the r.SetBasicAuth() method on your request before executing it. Like so: package main import ( "fmt" "io" "log" "net/http" "time" ) func main() { client := http.Client{Timeout: 5 * time.Second} req, err := http.NewRequest(http.MethodGet, "https://localhost:4000/protected", http.NoBody) if err != nil { log.Fatal(err) } req.SetBasicAuth("alice", "p8fnxeqj5a7zbrqp") res, err := client.Do(req) if err != nil { log.Fatal(err) } defer res.Body.Close() resBody, err := io.ReadAll(res.Body) if err != nil { log.Fatal(err) } fmt.Printf("Status: %d\n", res.StatusCode) fmt.Printf("Body: %s\n", string(resBody)) }

Julia Evans

• New zine: The Secret Rules of the Terminal
  Jun 26, 2025

  Hello! After many months of writing deep dive blog posts about the terminal, on Tuesday I released a new zine called “The Secret Rules of the Terminal”! You can get it for $12 here: https://wizardzines.com/zines/terminal , or get an 15-pack of all my zines here . Here’s the cover: the table of contents Here’s the table of contents: why the terminal? I’ve been using the terminal every day for 20 years but even though I’m very confident in the terminal, I’ve always had a bit of an uneasy feeling about it. Usually things work fine, but sometimes something goes wrong and it just feels like investigating it is impossible, or at least like it would open up a huge can of worms. So I started trying to write down a list of weird problems I’ve run into in terminal and I realized that the terminal has a lot of tiny inconsistencies like: sometimes you can use the arrow keys to move around, but sometimes pressing the arrow keys just prints ^[[D sometimes you can use the mouse to select text, but sometimes you can’t sometimes your commands get saved to a history when you run them, and sometimes they don’t some shells let you use the up arrow to see the previous command, and some don’t If you use the terminal daily for 10 or 20 years, even if you don’t understand exactly why these things happen, you’ll probably build an intuition for them. But having an intuition for them isn’t the same as understanding why they happen. When writing this zine I actually had to do a lot of work to figure out exactly what was happening in the terminal to be able to talk about how to reason about it. the rules aren’t written down anywhere It turns out that the “rules” for how the terminal works (how do you edit a command you type in? how do you quit a program? how do you fix your colours?) are extremely hard to fully understand, because “the terminal” is actually made of many different pieces of software (your terminal emulator, your operating system, your shell, the core utilities like grep , and every other random terminal program you’ve installed) which are written by different people with different ideas about how things should work. So I wanted to write something that would explain: how the 4 pieces of the terminal (your shell, terminal emulator, programs, and TTY driver) fit together to make everything work some of the core conventions for how you can expect things in your terminal to work lots of tips and tricks for how to use terminal programs this zine explains the most useful parts of terminal internals Terminal internals are a mess. A lot of it is just the way it is because someone made a decision in the 80s and now it’s impossible to change, and honestly I don’t think learning everything about terminal internals is worth it. But some parts are not that hard to understand and can really make your experience in the terminal better, like: if you understand what your shell is responsible for, you can configure your shell (or use a different one!) to access your history more easily, get great tab completion, and so much more if you understand escape codes , it’s much less scary when cat ing a binary to stdout messes up your terminal, you can just type reset and move on if you understand how colour works, you can get rid of bad colour contrast in your terminal so you can actually read the text I learned a surprising amount writing this zine When I wrote How Git Works , I thought I knew how Git worked, and I was right. But the terminal is different. Even though I feel totally confident in the terminal and even though I’ve used it every day for 20 years, I had a lot of misunderstandings about how the terminal works and (unless you’re the author of tmux or something) I think there’s a good chance you do too. A few things I learned that are actually useful to me: I understand the structure of the terminal better and so I feel more confident debugging weird terminal stuff that happens to me (I was even able to suggest a small improvement to fish!). Identifying exactly which piece of software is causing a weird thing to happen in my terminal still isn’t easy but I’m a lot better at it now. you can write a shell script to copy to your clipboard over SSH how reset works under the hood (it does the equivalent of stty sane; sleep 1; tput reset ) – basically I learned that I don’t ever need to worry about remembering stty sane or tput reset and I can just run reset instead how to look at the invisible escape codes that a program is printing out (run unbuffer program > out; less out ) why the builtin REPLs on my Mac like sqlite3 are so annoying to use (they use libedit instead of readline ) blog posts I wrote along the way As usual these days I wrote a bunch of blog posts about various side quests: How to add a directory to your PATH “rules” that terminal problems follow why pipes sometimes get “stuck”: buffering some terminal frustrations ASCII control characters in my terminal on “what’s the deal with Ctrl+A, Ctrl+B, Ctrl+C, etc?” entering text in the terminal is complicated what’s involved in getting a “modern” terminal setup? reasons to use your shell’s job control standards for ANSI escape codes , which is really me trying to figure out if I think the terminfo database is serving us well today people who helped with this zine A long time ago I used to write zines mostly by myself but with every project I get more and more help. I met with Marie Claire LeBlanc Flanagan every weekday from September to June to work on this one. The cover is by Vladimir Kašiković, Lesley Trites did copy editing, Simon Tatham (who wrote PuTTY ) did technical review, our Operations Manager Lee did the transcription as well as a million other things, and Jesse Luehrs (who is one of the very few people I know who actually understands the terminal’s cursed inner workings) had so many incredibly helpful conversations with me about what is going on in the terminal. get the zine Here are some links to get the zine again: get The Secret Rules of the Terminal get a 15-pack of all my zines here . As always, you can get either a PDF version to print at home or a print version shipped to your house. The only caveat is print orders will ship in August – I need to wait for orders to come in to get an idea of how many I should print before sending it to the printer.

• Using `make` to compile C programs (for non-C-programmers)
  Jun 10, 2025

  I have never been a C programmer but every so often I need to compile a C/C++ program from source. This has been kind of a struggle for me: for a long time, my approach was basically “install the dependencies, run make , if it doesn’t work, either try to find a binary someone has compiled or give up”. “Hope someone else has compiled it” worked pretty well when I was running Linux but since I’ve been using a Mac for the last couple of years I’ve been running into more situations where I have to actually compile programs myself. So let’s talk about what you might have to do to compile a C program! I’ll use a couple of examples of specific C programs I’ve compiled and talk about a few things that can go wrong. Here are three programs we’ll be talking about compiling: paperjam sqlite qf (a pager you can run to quickly open files from a search with rg -n THING | qf ) step 1: install a C compiler This is pretty simple: on an Ubuntu system if I don’t already have a C compiler I’ll install one with: sudo apt-get install build-essential This installs gcc , g++ , and make . The situation on a Mac is more confusing but it’s something like “install xcode command line tools”. step 2: install the program’s dependencies Unlike some newer programming languages, C doesn’t have a dependency manager. So if a program has any dependencies, you need to hunt them down yourself. Thankfully because of this, C programmers usually keep their dependencies very minimal and often the dependencies will be available in whatever package manager you’re using. There’s almost always a section explaining how to get the dependencies in the README, for example in paperjam ’s README, it says: To compile PaperJam, you need the headers for the libqpdf and libpaper libraries (usually available as libqpdf-dev and libpaper-dev packages). You may need a2x (found in AsciiDoc ) for building manual pages. So on a Debian-based system you can install the dependencies like this. sudo apt install -y libqpdf-dev libpaper-dev If a README gives a name for a package (like libqpdf-dev ), I’d basically always assume that they mean “in a Debian-based Linux distro”: if you’re on a Mac brew install libqpdf-dev will not work. I still have not 100% gotten the hang of developing on a Mac yet so I don’t have many tips there yet. I guess in this case it would be brew install qpdf if you’re using Homebrew. step 3: run ./configure (if needed) Some C programs come with a Makefile and some instead come with a script called ./configure . For example, if you download sqlite’s source code , it has a ./configure script in it instead of a Makefile. My understanding of this ./configure script is: You run it, it prints out a lot of somewhat inscrutable output, and then it either generates a Makefile or fails because you’re missing some dependency The ./configure script is part of a system called autotools that I have never needed to learn anything about beyond “run it to generate a Makefile ”. I think there might be some options you can pass to get the ./configure script to produce a different Makefile but I have never done that. step 4: run make The next step is to run make to try to build a program. Some notes about make : Sometimes you can run make -j8 to parallelize the build and make it go faster It usually prints out a million compiler warnings when compiling the program. I always just ignore them. I didn’t write the software! The compiler warnings are not my problem. compiler errors are often dependency problems Here’s an error I got while compiling paperjam on my Mac: /opt/homebrew/Cellar/qpdf/12.0.0/include/qpdf/InputSource.hh:85:19: error: function definition does not declare parameters 85 | qpdf_offset_t last_offset{0}; | ^ Over the years I’ve learned it’s usually best not to overthink problems like this: if it’s talking about qpdf , there’s a good change it just means that I’ve done something wrong with how I’m including the qpdf dependency. Now let’s talk about some ways to get the qpdf dependency included in the right way. the world’s shortest introduction to the compiler and linker Before we talk about how to fix dependency problems: building C programs is split into 2 steps: Compiling the code into object files (with gcc or clang ) Linking those object files into a final binary (with ld ) It’s important to know this when building a C program because sometimes you need to pass the right flags to the compiler and linker to tell them where to find the dependencies for the program you’re compiling. make uses environment variables to configure the compiler and linker If I run make on my Mac to install paperjam , I get this error: c++ -o paperjam paperjam.o pdf-tools.o parse.o cmds.o pdf.o -lqpdf -lpaper ld: library 'qpdf' not found This is not because qpdf is not installed on my system (it actually is!). But the compiler and linker don’t know how to find the qpdf library. To fix this, we need to: pass "-I/opt/homebrew/include" to the compiler (to tell it where to find the header files) pass "-L/opt/homebrew/lib -liconv" to the linker (to tell it where to find library files and to link in iconv ) And we can get make to pass those extra parameters to the compiler and linker using environment variables! To see how this works: inside paperjam ’s Makefile you can see a bunch of environment variables, like LDLIBS here: paperjam: $(OBJS) $(LD) -o $@ $^ $(LDLIBS) Everything you put into the LDLIBS environment variable gets passed to the linker ( ld ) as a command line argument. secret environment variable: CPPFLAGS Makefiles sometimes define their own environment variables that they pass to the compiler/linker, but make also has a bunch of “implicit” environment variables which it will automatically pass to the C compiler and linker. There’s a full list of implicit environment variables here , but one of them is CPPFLAGS , which gets automatically passed to the C compiler. (technically it would be more normal to use CXXFLAGS for this, but this particular Makefile hardcodes CXXFLAGS so setting CPPFLAGS was the only way I could find to set the compiler flags without editing the Makefile ) As an aside: it took me a long time to realize how closely tied to C/C++ `make` is -- I used to think that `make` was just a general build system (and of course you can use it for anything!) but it has a lot of affordances for building C/C++ programs that it doesn't have for building any other kind of program. two ways to pass environment variables to make I learned thanks to @zwol that there are actually two ways to pass environment variables to make : CXXFLAGS=xyz make (the usual way) make CXXFLAGS=xyz The difference between them is that make CXXFLAGS=xyz will override the value of CXXFLAGS set in the Makefile but CXXFLAGS=xyz make won’t. I’m not sure which way is the norm but I’m going to use the first way in this post. how to use CPPFLAGS and LDLIBS to fix this compiler error Now that we’ve talked about how CPPFLAGS and LDLIBS get passed to the compiler and linker, here’s the final incantation that I used to get the program to build successfully! CPPFLAGS="-I/opt/homebrew/include" LDLIBS="-L/opt/homebrew/lib -liconv" make paperjam This passes -I/opt/homebrew/include to the compiler and -L/opt/homebrew/lib -liconv to the linker. Also I don’t want to pretend that I “magically” knew that those were the right arguments to pass, figuring them out involved a bunch of confused Googling that I skipped over in this post. I will say that: the -I compiler flag tells the compiler which directory to find header files in, like /opt/homebrew/include/qpdf/QPDF.hh the -L linker flag tells the linker which directory to find libraries in, like /opt/homebrew/lib/libqpdf.a the -l linker flag tells the linker which libraries to link in, like -liconv means “link in the iconv library”, or -lm means “link math ” tip: how to just build 1 specific file: make $FILENAME Yesterday I discovered this cool tool called qf which you can use to quickly open files from the output of ripgrep . qf is in a big directory of various tools, but I only wanted to compile qf . So I just compiled qf , like this: make qf Basically if you know (or can guess) the output filename of the file you’re trying to build, you can tell make to just build that file by running make $FILENAME tip: you don’t need a Makefile I sometimes write 5-line C programs with no dependencies, and I just learned that if I have a file called blah.c , I can just compile it like this without creating a Makefile : make blah It gets automaticaly expanded to cc -o blah blah.c , which saves a bit of typing. I have no idea if I’m going to remember this (I might just keep typing gcc -o blah blah.c anyway) but it seems like a fun trick. tip: look at how other packaging systems built the same C program If you’re having trouble building a C program, maybe other people had problems building it too! Every Linux distribution has build files for every package that they build, so even if you can’t install packages from that distribution directly, maybe you can get tips from that Linux distro for how to build the package. Realizing this (thanks to my friend Dave) was a huge ah-ha moment for me. For example, this line from the nix package for paperjam says: env.NIX_LDFLAGS = lib.optionalString stdenv.hostPlatform.isDarwin "-liconv"; This is basically saying “pass the linker flag -liconv to build this on a Mac”, so that’s a clue we could use to build it. That same file also says env.NIX_CFLAGS_COMPILE = "-DPOINTERHOLDER_TRANSITION=1"; . I’m not sure what this means, but when I try to build the paperjam package I do get an error about something called a PointerHolder , so I guess that’s somehow related to the “PointerHolder transition”. step 5: installing the binary Once you’ve managed to compile the program, probably you want to install it somewhere! Some Makefile s have an install target that let you install the tool on your system with make install . I’m always a bit scared of this (where is it going to put the files? what if I want to uninstall them later?), so if I’m compiling a pretty simple program I’ll often just manually copy the binary to install it instead, like this: cp qf ~/bin step 6: maybe make your own package! Once I figured out how to do all of this, I realized that I could use my new make knowledge to contribute a paperjam package to Homebrew! Then I could just brew install paperjam on future systems. The good thing is that even if the details of how all of the different packaging systems, they fundamentally all use C compilers and linkers. it can be useful to understand a little about C even if you’re not a C programmer I think all of this is an interesting example of how it can useful to understand some basics of how C programs work (like “they have header files”) even if you’re never planning to write a nontrivial C program if your life. It feels good to have some ability to compile C/C++ programs myself, even though I’m still not totally confident about all of the compiler and linker flags and I still plan to never learn anything about how autotools works other than “you run ./configure to generate the Makefile ”. Two things I left out of this post: LD_LIBRARY_PATH / DYLD_LIBRARY_PATH (which you use to tell the dynamic linker at runtime where to find dynamically linked files) because I can’t remember the last time I ran into an LD_LIBRARY_PATH issue and couldn’t find an example. pkg-config , which I think is important but I don’t understand yet

• Standards for ANSI escape codes
  Mar 07, 2025

  Hello! Today I want to talk about ANSI escape codes. For a long time I was vaguely aware of ANSI escape codes (“that’s how you make text red in the terminal and stuff”) but I had no real understanding of where they were supposed to be defined or whether or not there were standards for them. I just had a kind of vague “there be dragons” feeling around them. While learning about the terminal this year, I’ve learned that: ANSI escape codes are responsible for a lot of usability improvements in the terminal (did you know there’s a way to copy to your system clipboard when SSHed into a remote machine?? It’s an escape code called OSC 52 !) They aren’t completely standardized, and because of that they don’t always work reliably. And because they’re also invisible, it’s extremely frustrating to troubleshoot escape code issues. So I wanted to put together a list for myself of some standards that exist around escape codes, because I want to know if they have to feel unreliable and frustrating, or if there’s a future where we could all rely on them with more confidence. what’s an escape code? ECMA-48 xterm control sequences terminfo should programs use terminfo? is there a “single common set” of escape codes? some reasons to use terminfo some more documents/standards why I think this is interesting what’s an escape code? Have you ever pressed the left arrow key in your terminal and seen ^[[D ? That’s an escape code! It’s called an “escape code” because the first character is the “escape” character, which is usually written as ESC , \x1b , \E , \033 , or ^[ . Escape codes are how your terminal emulator communicates various kinds of information (colours, mouse movement, etc) with programs running in the terminal. There are two kind of escape codes: input codes which your terminal emulator sends for keypresses or mouse movements that don’t fit into Unicode. For example “left arrow key” is ESC[D , “Ctrl+left arrow” might be ESC[1;5D , and clicking the mouse might be something like ESC[M :3 . output codes which programs can print out to colour text, move the cursor around, clear the screen, hide the cursor, copy text to the clipboard, enable mouse reporting, set the window title, etc. Now let’s talk about standards! ECMA-48 The first standard I found relating to escape codes was ECMA-48 , which was originally published in 1976. ECMA-48 does two things: Define some general formats for escape codes (like “CSI” codes, which are ESC[ + something and “OSC” codes, which are ESC] + something) Define some specific escape codes, like how “move the cursor to the left” is ESC[D , or “turn text red” is ESC[31m . In the spec, the “cursor left” one is called CURSOR LEFT and the one for changing colours is called SELECT GRAPHIC RENDITION . The formats are extensible, so there’s room for others to define more escape codes in the future. Lots of escape codes that are popular today aren’t defined in ECMA-48: for example it’s pretty common for terminal applications (like vim, htop, or tmux) to support using the mouse, but ECMA-48 doesn’t define escape codes for the mouse. xterm control sequences There are a bunch of escape codes that aren’t defined in ECMA-48, for example: enabling mouse reporting (where did you click in your terminal?) bracketed paste (did you paste that text or type it in?) OSC 52 (which terminal applications can use to copy text to your system clipboard) I believe (correct me if I’m wrong!) that these and some others came from xterm, are documented in XTerm Control Sequences , and have been widely implemented by other terminal emulators. This list of “what xterm supports” is not a standard exactly, but xterm is extremely influential and so it seems like an important document. terminfo In the 80s (and to some extent today, but my understanding is that it was MUCH more dramatic in the 80s) there was a huge amount of variation in what escape codes terminals actually supported. To deal with this, there’s a database of escape codes for various terminals called “terminfo”. It looks like the standard for terminfo is called X/Open Curses , though you need to create an account to view that standard for some reason. It defines the database format as well as a C library interface (“curses”) for accessing the database. For example you can run this bash snippet to see every possible escape code for “clear screen” for all of the different terminals your system knows about: for term in $(toe -a | awk '{print $1}') do echo $term infocmp -1 -T "$term" 2>/dev/null | grep 'clear=' | sed 's/clear=//g;s/,//g' done On my system (and probably every system I’ve ever used?), the terminfo database is managed by ncurses. should programs use terminfo? I think it’s interesting that there are two main approaches that applications take to handling ANSI escape codes: Use the terminfo database to figure out which escape codes to use, depending on what’s in the TERM environment variable. Fish does this, for example. Identify a “single common set” of escape codes which works in “enough” terminal emulators and just hardcode those. Some examples of programs/libraries that take approach #2 (“don’t use terminfo”) include: kakoune python-prompt-toolkit linenoise libvaxis chalk I got curious about why folks might be moving away from terminfo and I found this very interesting and extremely detailed rant about terminfo from one of the fish maintainers , which argues that: [the terminfo authors] have done a lot of work that, at the time, was extremely important and helpful. My point is that it no longer is. I’m not going to do it justice so I’m not going to summarize it, I think it’s worth reading. is there a “single common set” of escape codes? I was just talking about the idea that you can use a “common set” of escape codes that will work for most people. But what is that set? Is there any agreement? I really do not know the answer to this at all, but from doing some reading it seems like it’s some combination of: The codes that the VT100 supported (though some aren’t relevant on modern terminals) what’s in ECMA-48 (which I think also has some things that are no longer relevant) What xterm supports (though I’d guess that not everything in there is actually widely supported enough) and maybe ultimately “identify the terminal emulators you think your users are going to use most frequently and test in those”, the same way web developers do when deciding which CSS features are okay to use I don’t think there are any resources like Can I use…? or Baseline for the terminal though. (in theory terminfo is supposed to be the “caniuse” for the terminal but it seems like it often takes 10+ years to add new terminal features when people invent them which makes it very limited) some reasons to use terminfo I also asked on Mastodon why people found terminfo valuable in 2025 and got a few reasons that made sense to me: some people expect to be able to use the TERM environment variable to control how programs behave (for example with TERM=dumb ), and there’s no standard for how that should work in a post-terminfo world even though there’s less variation between terminal emulators than there was in the 80s, there’s far from zero variation: there are graphical terminals, the Linux framebuffer console, the situation you’re in when connecting to a server via its serial console, Emacs shell mode, and probably more that I’m missing there is no one standard for what the “single common set” of escape codes is, and sometimes programs use escape codes which aren’t actually widely supported enough terminfo & user agent detection The way that ncurses uses the TERM environment variable to decide which escape codes to use reminds me of how webservers used to sometimes use the browser user agent to decide which version of a website to serve. It also seems like it’s had some of the same results – the way iTerm2 reports itself as being “xterm-256color” feels similar to how Safari’s user agent is “Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15”. In both cases the terminal emulator / browser ends up changing its user agent to get around user agent detection that isn’t working well. On the web we ended up deciding that user agent detection was not a good practice and to instead focus on standardization so we can serve the same HTML/CSS to all browsers. I don’t know if the same approach is the future in the terminal though – I think the terminal landscape today is much more fragmented than the web ever was as well as being much less well funded. some more documents/standards A few more documents and standards related to escape codes, in no particular order: the Linux console_codes man page documents escape codes that Linux supports how the VT 100 handles escape codes & control sequences the kitty keyboard protocol OSC 8 for links in the terminal (and notes on adoption ) A summary of ANSI standards from tmux this terminal features reporting specification from iTerm sixel graphics why I think this is interesting I sometimes see people saying that the unix terminal is “outdated”, and since I love the terminal so much I’m always curious about what incremental changes might make it feel less “outdated”. Maybe if we had a clearer standards landscape (like we do on the web!) it would be easier for terminal emulator developers to build new features and for authors of terminal applications to more confidently adopt those features so that we can all benefit from them and have a richer experience in the terminal. Obviously standardizing ANSI escape codes is not easy (ECMA-48 was first published almost 50 years ago and we’re still not there!). I don’t even know what all of the challenges are. But the situation with HTML/CSS/JS used to be extremely bad too and now it’s MUCH better, so maybe there’s hope.

• How to add a directory to your PATH
  Feb 13, 2025

  I was talking to a friend about how to add a directory to your PATH today. It’s something that feels “obvious” to me since I’ve been using the terminal for a long time, but when I searched for instructions for how to do it, I actually couldn’t find something that explained all of the steps – a lot of them just said “add this to ~/.bashrc ”, but what if you’re not using bash? What if your bash config is actually in a different file? And how are you supposed to figure out which directory to add anyway? So I wanted to try to write down some more complete directions and mention some of the gotchas I’ve run into over the years. Here’s a table of contents: step 1: what shell are you using? step 2: find your shell’s config file a note on bash’s config file step 3: figure out which directory to add step 3.1: double check it’s the right directory step 4: edit your shell config step 5: restart your shell problems: problem 1: it ran the wrong program problem 2: the program isn’t being run from your shell problem 3: duplicate PATH entries making it harder to debug problem 4: losing your history after updating your PATH notes: a note on source a note on fish_add_path step 1: what shell are you using? If you’re not sure what shell you’re using, here’s a way to find out. Run this: ps -p $$ -o pid,comm= if you’re using bash , it’ll print out 97295 bash if you’re using zsh , it’ll print out 97295 zsh if you’re using fish , it’ll print out an error like “In fish, please use $fish_pid” ( $$ isn’t valid syntax in fish, but in any case the error message tells you that you’re using fish, which you probably already knew) Also bash is the default on Linux and zsh is the default on Mac OS (as of 2024). I’ll only cover bash, zsh, and fish in these directions. step 2: find your shell’s config file in zsh, it’s probably ~/.zshrc in bash, it might be ~/.bashrc , but it’s complicated, see the note in the next section in fish, it’s probably ~/.config/fish/config.fish (you can run echo $__fish_config_dir if you want to be 100% sure) a note on bash’s config file Bash has three possible config files: ~/.bashrc , ~/.bash_profile , and ~/.profile . If you’re not sure which one your system is set up to use, I’d recommend testing this way: add echo hi there to your ~/.bashrc Restart your terminal If you see “hi there”, that means ~/.bashrc is being used! Hooray! Otherwise remove it and try the same thing with ~/.bash_profile You can also try ~/.profile if the first two options don’t work. (there are a lot of elaborate flow charts out there that explain how bash decides which config file to use but IMO it’s not worth it to internalize them and just testing is the fastest way to be sure) step 3: figure out which directory to add Let’s say that you’re trying to install and run a program called http-server and it doesn’t work, like this: $ npm install -g http-server $ http-server bash: http-server: command not found How do you find what directory http-server is in? Honestly in general this is not that easy – often the answer is something like “it depends on how npm is configured”. A few ideas: Often when setting up a new installer (like cargo , npm , homebrew , etc), when you first set it up it’ll print out some directions about how to update your PATH. So if you’re paying attention you can get the directions then. Sometimes installers will automatically update your shell’s config file to update your PATH for you Sometimes just Googling “where does npm install things?” will turn up the answer Some tools have a subcommand that tells you where they’re configured to install things, like: Node/npm: npm config get prefix (then append /bin/ ) Go: go env GOPATH (then append /bin/ ) asdf: asdf info | grep ASDF_DIR (then append /bin/ and /shims/ ) step 3.1: double check it’s the right directory Once you’ve found a directory you think might be the right one, make sure it’s actually correct! For example, I found out that on my machine, http-server is in ~/.npm-global/bin . I can make sure that it’s the right directory by trying to run the program http-server in that directory like this: $ ~/.npm-global/bin/http-server Starting up http-server, serving ./public It worked! Now that you know what directory you need to add to your PATH , let’s move to the next step! step 4: edit your shell config Now we have the 2 critical pieces of information we need: Which directory you’re trying to add to your PATH (like ~/.npm-global/bin/ ) Where your shell’s config is (like ~/.bashrc , ~/.zshrc , or ~/.config/fish/config.fish ) Now what you need to add depends on your shell: bash instructions: Open your shell’s config file, and add a line like this: export PATH=$PATH:~/.npm-global/bin/ (obviously replace ~/.npm-global/bin with the actual directory you’re trying to add) zsh instructions: You can do the same thing as in bash, but zsh also has some slightly fancier syntax you can use if you prefer: path=( $path ~/.npm-global/bin ) fish instructions: In fish, the syntax is different: set PATH $PATH ~/.npm-global/bin (in fish you can also use fish_add_path , some notes on that further down ) step 5: restart your shell Now, an extremely important step: updating your shell’s config won’t take effect if you don’t restart it! Two ways to do this: open a new terminal (or terminal tab), and maybe close the old one so you don’t get confused Run bash to start a new shell (or zsh if you’re using zsh, or fish if you’re using fish) I’ve found that both of these usually work fine. And you should be done! Try running the program you were trying to run and hopefully it works now. If not, here are a couple of problems that you might run into: problem 1: it ran the wrong program If the wrong version of a program is running, you might need to add the directory to the beginning of your PATH instead of the end. For example, on my system I have two versions of python3 installed, which I can see by running which -a : $ which -a python3 /usr/bin/python3 /opt/homebrew/bin/python3 The one your shell will use is the first one listed . If you want to use the Homebrew version, you need to add that directory ( /opt/homebrew/bin ) to the beginning of your PATH instead, by putting this in your shell’s config file (it’s /opt/homebrew/bin/:$PATH instead of the usual $PATH:/opt/homebrew/bin/ ) export PATH=/opt/homebrew/bin/:$PATH or in fish: set PATH ~/.cargo/bin $PATH problem 2: the program isn’t being run from your shell All of these directions only work if you’re running the program from your shell . If you’re running the program from an IDE, from a GUI, in a cron job, or some other way, you’ll need to add the directory to your PATH in a different way, and the exact details might depend on the situation. in a cron job Some options: use the full path to the program you’re running, like /home/bork/bin/my-program put the full PATH you want as the first line of your crontab (something like PATH=/bin:/usr/bin:/usr/local/bin:….). You can get the full PATH you’re using in your shell by running echo "PATH=$PATH" . I’m honestly not sure how to handle it in an IDE/GUI because I haven’t run into that in a long time, will add directions here if someone points me in the right direction. problem 3: duplicate PATH entries making it harder to debug If you edit your path and start a new shell by running bash (or zsh , or fish ), you’ll often end up with duplicate PATH entries, because the shell keeps adding new things to your PATH every time you start your shell. Personally I don’t think I’ve run into a situation where this kind of duplication breaks anything, but the duplicates can make it harder to debug what’s going on with your PATH if you’re trying to understand its contents. Some ways you could deal with this: If you’re debugging your PATH , open a new terminal to do it in so you get a “fresh” state. This should avoid the duplication. Deduplicate your PATH at the end of your shell’s config (for example in zsh apparently you can do this with typeset -U path ) Check that the directory isn’t already in your PATH when adding it (for example in fish I believe you can do this with fish_add_path --path /some/directory ) How to deduplicate your PATH is shell-specific and there isn’t always a built in way to do it so you’ll need to look up how to accomplish it in your shell. problem 4: losing your history after updating your PATH Here’s a situation that’s easy to get into in bash or zsh: Run a command (it fails) Update your PATH Run bash to reload your config Press the up arrow a couple of times to rerun the failed command (or open a new terminal) The failed command isn’t in your history! Why not? This happens because in bash, by default, history is not saved until you exit the shell. Some options for fixing this: Instead of running bash to reload your config, run source ~/.bashrc (or source ~/.zshrc in zsh). This will reload the config inside your current session. Configure your shell to continuously save your history instead of only saving the history when the shell exits. (How to do this depends on whether you’re using bash or zsh, the history options in zsh are a bit complicated and I’m not exactly sure what the best way is) a note on source When you install cargo (Rust’s installer) for the first time, it gives you these instructions for how to set up your PATH, which don’t mention a specific directory at all. This is usually done by running one of the following (note the leading DOT): . "$HOME/.cargo/env" # For sh/bash/zsh/ash/dash/pdksh source "$HOME/.cargo/env.fish" # For fish The idea is that you add that line to your shell’s config, and their script automatically sets up your PATH (and potentially other things) for you. This is pretty common (for example Homebrew suggests you eval brew shellenv ), and there are two ways to approach this: Just do what the tool suggests (like adding . "$HOME/.cargo/env" to your shell’s config) Figure out which directories the script they’re telling you to run would add to your PATH, and then add those manually. Here’s how I’d do that: Run . "$HOME/.cargo/env" in my shell (or the fish version if using fish) Run echo "$PATH" | tr ':' '\n' | grep cargo to figure out which directories it added See that it says /Users/bork/.cargo/bin and shorten that to ~/.cargo/bin Add the directory ~/.cargo/bin to PATH (with the directions in this post) I don’t think there’s anything wrong with doing what the tool suggests (it might be the “best way”!), but personally I usually use the second approach because I prefer knowing exactly what configuration I’m changing. a note on fish_add_path fish has a handy function called fish_add_path that you can run to add a directory to your PATH like this: fish_add_path /some/directory This is cool (it’s such a simple command!) but I’ve stopped using it for a couple of reasons: Sometimes fish_add_path will update the PATH for every session in the future (with a “universal variable”) and sometimes it will update the PATH just for the current session and it’s hard for me to tell which one it will do. In theory the docs explain this but I could not understand them. If you ever need to remove the directory from your PATH a few weeks or months later because maybe you made a mistake, it’s kind of hard to do (there are instructions in this comments of this github issue though ). that’s all Hopefully this will help some people. Let me know (on Mastodon or Bluesky) if you there are other major gotchas that have tripped you up when adding a directory to your PATH, or if you have questions about this post!

• Some terminal frustrations
  Feb 05, 2025

  A few weeks ago I ran a terminal survey (you can read the results here ) and at the end I asked: What’s the most frustrating thing about using the terminal for you? 1600 people answered, and I decided to spend a few days categorizing all the responses. Along the way I learned that classifying qualitative data is not easy but I gave it my best shot. I ended up building a custom tool to make it faster to categorize everything. As with all of my surveys the methodology isn’t particularly scientific. I just posted the survey to Mastodon and Twitter, ran it for a couple of days, and got answers from whoever happened to see it and felt like responding. Here are the top categories of frustrations! I think it’s worth keeping in mind while reading these comments that 40% of people answering this survey have been using the terminal for 21+ years 95% of people answering the survey have been using the terminal for at least 4 years These comments aren’t coming from total beginners. Here are the categories of frustrations! The number in brackets is the number of people with that frustration. I’m mostly writing this up for myself because I’m trying to write a zine about the terminal and I wanted to get a sense for what people are having trouble with. remembering syntax (115) People talked about struggles remembering: the syntax for CLI tools like awk, jq, sed, etc the syntax for redirects keyboard shortcuts for tmux, text editing, etc One example comment: There are just so many little “trivia” details to remember for full functionality. Even after all these years I’ll sometimes forget where it’s 2 or 1 for stderr, or forget which is which for > and >> . switching terminals is hard (91) People talked about struggling with switching systems (for example home/work computer or when SSHing) and running into: OS differences in keyboard shortcuts (like Linux vs Mac) systems which don’t have their preferred text editor (“no vim” or “only vim”) different versions of the same command (like Mac OS grep vs GNU grep) no tab completion a shell they aren’t used to (“the subtle differences between zsh and bash”) as well as differences inside the same system like pagers being not consistent with each other (git diff pagers, other pagers). One example comment: I got used to fish and vi mode which are not available when I ssh into servers, containers. color (85) Lots of problems with color, like: programs setting colors that are unreadable with a light background color finding a colorscheme they like (and getting it to work consistently across different apps) color not working inside several layers of SSH/tmux/etc not liking the defaults not wanting color at all and struggling to turn it off This comment felt relatable to me: Getting my terminal theme configured in a reasonable way between the terminal emulator and fish (I did this years ago and remember it being tedious and fiddly and now feel like I’m locked into my current theme because it works and I dread touching any of that configuration ever again). keyboard shortcuts (84) Half of the comments on keyboard shortcuts were about how on Linux/Windows, the keyboard shortcut to copy/paste in the terminal is different from in the rest of the OS. Some other issues with keyboard shortcuts other than copy/paste: using Ctrl-W in a browser-based terminal and closing the window the terminal only supports a limited set of keyboard shortcuts (no Ctrl-Shift- , no Super , no Hyper , lots of ctrl- shortcuts aren’t possible like Ctrl-, ) the OS stopping you from using a terminal keyboard shortcut (like by default Mac OS uses Ctrl+left arrow for something else) issues using emacs in the terminal backspace not working (2) other copy and paste issues (75) Aside from “the keyboard shortcut for copy and paste is different”, there were a lot of OTHER issues with copy and paste, like: copying over SSH how tmux and the terminal emulator both do copy/paste in different ways dealing with many different clipboards (system clipboard, vim clipboard, the “middle click” clipboard on Linux, tmux’s clipboard, etc) and potentially synchronizing them random spaces added when copying from the terminal pasting multiline commands which automatically get run in a terrifying way wanting a way to copy text without using the mouse discoverability (55) There were lots of comments about this, which all came down to the same basic complaint – it’s hard to discover useful tools or features! This comment kind of summed it all up: How difficult it is to learn independently. Most of what I know is an assorted collection of stuff I’ve been told by random people over the years. steep learning curve (44) A lot of comments about it generally having a steep learning curve. A couple of example comments: After 15 years of using it, I’m not much faster than using it than I was 5 or maybe even 10 years ago. and That I know I could make my life easier by learning more about the shortcuts and commands and configuring the terminal but I don’t spend the time because it feels overwhelming. history (42) Some issues with shell history: history not being shared between terminal tabs (16) limits that are too short (4) history not being restored when terminal tabs are restored losing history because the terminal crashed not knowing how to search history One example comment: It wasted a lot of time until I figured it out and still annoys me that “history” on zsh has such a small buffer; I have to type “history 0” to get any useful length of history. bad documentation (37) People talked about: documentation being generally opaque lack of examples in man pages programs which don’t have man pages Here’s a representative comment: Finding good examples and docs. Man pages often not enough, have to wade through stack overflow scrollback (36) A few issues with scrollback: programs printing out too much data making you lose scrollback history resizing the terminal messes up the scrollback lack of timestamps GUI programs that you start in the background printing stuff out that gets in the way of other programs’ outputs One example comment: When resizing the terminal (in particular: making it narrower) leads to broken rewrapping of the scrollback content because the commands formatted their output based on the terminal window width. “it feels outdated” (33) Lots of comments about how the terminal feels hampered by legacy decisions and how users often end up needing to learn implementation details that feel very esoteric. One example comment: Most of the legacy cruft, it would be great to have a green field implementation of the CLI interface. shell scripting (32) Lots of complaints about POSIX shell scripting. There’s a general feeling that shell scripting is difficult but also that switching to a different less standard scripting language (fish, nushell, etc) brings its own problems. Shell scripting. My tolerance to ditch a shell script and go to a scripting language is pretty low. It’s just too messy and powerful. Screwing up can be costly so I don’t even bother. more issues Some more issues that were mentioned at least 10 times: (31) inconsistent command line arguments: is it -h or help or –help? (24) keeping dotfiles in sync across different systems (23) performance (e.g. “my shell takes too long to start”) (20) window management (potentially with some combination of tmux tabs, terminal tabs, and multiple terminal windows. Where did that shell session go?) (17) generally feeling scared/uneasy (“The debilitating fear that I’m going to do some mysterious Bad Thing with a command and I will have absolutely no idea how to fix or undo it or even really figure out what happened”) (16) terminfo issues (“Having to learn about terminfo if/when I try a new terminal emulator and ssh elsewhere.”) (16) lack of image support (sixel etc) (15) SSH issues (like having to start over when you lose the SSH connection) (15) various tmux/screen issues (for example lack of integration between tmux and the terminal emulator) (15) typos & slow typing (13) the terminal getting messed up for various reasons (pressing Ctrl-S , cat ing a binary, etc) (12) quoting/escaping in the shell (11) various Windows/PowerShell issues n/a (122) There were also 122 answers to the effect of “nothing really” or “only that I can’t do EVERYTHING in the terminal” One example comment: Think I’ve found work arounds for most/all frustrations that’s all! I’m not going to make a lot of commentary on these results, but here are a couple of categories that feel related to me: remembering syntax & history (often the thing you need to remember is something you’ve run before!) discoverability & the learning curve (the lack of discoverability is definitely a big part of what makes it hard to learn) “switching systems is hard” & “it feels outdated” (tools that haven’t really changed in 30 or 40 years have many problems but they do tend to be always there no matter what system you’re on, which is very useful and makes them hard to stop using) Trying to categorize all these results in a reasonable way really gave me an appreciation for social science researchers’ skills.

• What's involved in getting a "modern" terminal setup?
  Jan 11, 2025

  Hello! Recently I ran a terminal survey and I asked people what frustrated them. One person commented: There are so many pieces to having a modern terminal experience. I wish it all came out of the box. My immediate reaction was “oh, getting a modern terminal experience isn’t that hard, you just need to….”, but the more I thought about it, the longer the “you just need to…” list got, and I kept thinking about more and more caveats. So I thought I would write down some notes about what it means to me personally to have a “modern” terminal experience and what I think can make it hard for people to get there. what is a “modern terminal experience”? Here are a few things that are important to me, with which part of the system is responsible for them: multiline support for copy and paste : if you paste 3 commands in your shell, it should not immediately run them all! That’s scary! ( shell , terminal emulator ) infinite shell history : if I run a command in my shell, it should be saved forever, not deleted after 500 history entries or whatever. Also I want commands to be saved to the history immediately when I run them, not only when I exit the shell session ( shell ) a useful prompt : I can’t live without having my current directory and current git branch in my prompt ( shell ) 24-bit colour : this is important to me because I find it MUCH easier to theme neovim with 24-bit colour support than in a terminal with only 256 colours ( terminal emulator ) clipboard integration between vim and my operating system so that when I copy in Firefox, I can just press p in vim to paste ( text editor , maybe the OS/terminal emulator too) good autocomplete : for example commands like git should have command-specific autocomplete ( shell ) having colours in ls ( shell config ) a terminal theme I like : I spend a lot of time in my terminal, I want it to look nice and I want its theme to match my terminal editor’s theme. ( terminal emulator , text editor ) automatic terminal fixing : If a programs prints out some weird escape codes that mess up my terminal, I want that to automatically get reset so that my terminal doesn’t get messed up ( shell ) keybindings : I want Ctrl+left arrow to work ( shell or application ) being able to use the scroll wheel in programs like less : ( terminal emulator and applications ) There are a million other terminal conveniences out there and different people value different things, but those are the ones that I would be really unhappy without. how I achieve a “modern experience” My basic approach is: use the fish shell. Mostly don’t configure it, except to: set the EDITOR environment variable to my favourite terminal editor alias ls to ls --color=auto use any terminal emulator with 24-bit colour support. In the past I’ve used GNOME Terminal, Terminator, and iTerm, but I’m not picky about this. I don’t really configure it other than to choose a font. use neovim , with a configuration that I’ve been very slowly building over the last 9 years or so (the last time I deleted my vim config and started from scratch was 9 years ago) use the base16 framework to theme everything A few things that affect my approach: I don’t spend a lot of time SSHed into other machines I’d rather use the mouse a little than come up with keyboard-based ways to do everything I work on a lot of small projects, not one big project some “out of the box” options for a “modern” experience What if you want a nice experience, but don’t want to spend a lot of time on configuration? Figuring out how to configure vim in a way that I was satisfied with really did take me like ten years, which is a long time! My best ideas for how to get a reasonable terminal experience with minimal config are: shell: either fish or zsh with oh-my-zsh terminal emulator: almost anything with 24-bit colour support, for example all of these are popular: linux: GNOME Terminal, Konsole, Terminator, xfce4-terminal mac: iTerm (Terminal.app doesn’t have 256-colour support) cross-platform: kitty, alacritty, wezterm, or ghostty shell config: set the EDITOR environment variable to your favourite terminal text editor maybe alias ls to ls --color=auto text editor: this is a tough one, maybe micro or helix ? I haven’t used either of them seriously but they both seem like very cool projects and I think it’s amazing that you can just use all the usual GUI editor commands ( Ctrl-C to copy, Ctrl-V to paste, Ctrl-A to select all) in micro and they do what you’d expect. I would probably try switching to helix except that retraining my vim muscle memory seems way too hard. Also helix doesn’t have a GUI or plugin system yet. Personally I wouldn’t use xterm, rxvt, or Terminal.app as a terminal emulator, because I’ve found in the past that they’re missing core features (like 24-bit colour in Terminal.app’s case) that make the terminal harder to use for me. I don’t want to pretend that getting a “modern” terminal experience is easier than it is though – I think there are two issues that make it hard. Let’s talk about them! issue 1 with getting to a “modern” experience: the shell bash and zsh are by far the two most popular shells, and neither of them provide a default experience that I would be happy using out of the box, for example: you need to customize your prompt they don’t come with git completions by default, you have to set them up by default, bash only stores 500 (!) lines of history and (at least on Mac OS) zsh is only configured to store 2000 lines, which is still not a lot I find bash’s tab completion very frustrating, if there’s more than one match then you can’t tab through them And even though I love fish , the fact that it isn’t POSIX does make it hard for a lot of folks to make the switch. Of course it’s totally possible to learn how to customize your prompt in bash or whatever, and it doesn’t even need to be that complicated (in bash I’d probably start with something like export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ ' , or maybe use starship ). But each of these “not complicated” things really does add up and it’s especially tough if you need to keep your config in sync across several systems. An extremely popular solution to getting a “modern” shell experience is oh-my-zsh . It seems like a great project and I know a lot of people use it very happily, but I’ve struggled with configuration systems like that in the past – it looks like right now the base oh-my-zsh adds about 3000 lines of config, and often I find that having an extra configuration system makes it harder to debug what’s happening when things go wrong. I personally have a tendency to use the system to add a lot of extra plugins, make my system slow, get frustrated that it’s slow, and then delete it completely and write a new config from scratch. issue 2 with getting to a “modern” experience: the text editor In the terminal survey I ran recently, the most popular terminal text editors by far were vim , emacs , and nano . I think the main options for terminal text editors are: use vim or emacs and configure it to your liking, you can probably have any feature you want if you put in the work use nano and accept that you’re going to have a pretty limited experience (for example I don’t think you can select text with the mouse and then “cut” it in nano) use micro or helix which seem to offer a pretty good out-of-the-box experience, potentially occasionally run into issues with using a less mainstream text editor just avoid using a terminal text editor as much as possible, maybe use VSCode, use VSCode’s terminal for all your terminal needs, and mostly never edit files in the terminal. Or I know a lot of people use code as their EDITOR in the terminal. issue 3: individual applications The last issue is that sometimes individual programs that I use are kind of annoying. For example on my Mac OS machine, /usr/bin/sqlite3 doesn’t support the Ctrl+Left Arrow keyboard shortcut. Fixing this to get a reasonable terminal experience in SQLite was a little complicated, I had to: realize why this is happening (Mac OS won’t ship GNU tools, and “Ctrl-Left arrow” support comes from GNU readline) find a workaround (install sqlite from homebrew, which does have readline support) adjust my environment (put Homebrew’s sqlite3 in my PATH) I find that debugging application-specific issues like this is really not easy and often it doesn’t feel “worth it” – often I’ll end up just dealing with various minor inconveniences because I don’t want to spend hours investigating them. The only reason I was even able to figure this one out at all is that I’ve been spending a huge amount of time thinking about the terminal recently. A big part of having a “modern” experience using terminal programs is just using newer terminal programs, for example I can’t be bothered to learn a keyboard shortcut to sort the columns in top , but in htop I can just click on a column heading with my mouse to sort it. So I use htop instead! But discovering new more “modern” command line tools isn’t easy (though I made a list here ), finding ones that I actually like using in practice takes time, and if you’re SSHed into another machine, they won’t always be there. everything affects everything else Something I find tricky about configuring my terminal to make everything “nice” is that changing one seemingly small thing about my workflow can really affect everything else. For example right now I don’t use tmux. But if I needed to use tmux again (for example because I was doing a lot of work SSHed into another machine), I’d need to think about a few things, like: if I wanted tmux’s copy to synchronize with my system clipboard over SSH, I’d need to make sure that my terminal emulator has OSC 52 support if I wanted to use iTerm’s tmux integration (which makes tmux tabs into iTerm tabs), I’d need to change how I configure colours – right now I set them with a shell script that I run when my shell starts, but that means the colours get lost when restoring a tmux session. and probably more things I haven’t thought of. “Using tmux means that I have to change how I manage my colours” sounds unlikely, but that really did happen to me and I decided “well, I don’t want to change how I manage colours right now, so I guess I’m not using that feature!”. It’s also hard to remember which features I’m relying on – for example maybe my current terminal does have OSC 52 support and because copying from tmux over SSH has always Just Worked I don’t even realize that that’s something I need, and then it mysteriously stops working when I switch terminals. change things slowly Personally even though I think my setup is not that complicated, it’s taken me 20 years to get to this point! Because terminal config changes are so likely to have unexpected and hard-to-understand consequences, I’ve found that if I change a lot of terminal configuration all at once it makes it much harder to understand what went wrong if there’s a problem, which can be really disorienting. So I usually prefer to make pretty small changes, and accept that changes can might take me a REALLY long time to get used to. For example I switched from using ls to eza a year or two ago and while I like it (because eza -l prints human-readable file sizes by default) I’m still not quite sure about it. But also sometimes it’s worth it to make a big change, like I made the switch to fish (from bash) 10 years ago and I’m very happy I did. getting a “modern” terminal is not that easy Trying to explain how “easy” it is to configure your terminal really just made me think that it’s kind of hard and that I still sometimes get confused. I’ve found that there’s never one perfect way to configure things in the terminal that will be compatible with every single other thing. I just need to try stuff, figure out some kind of locally stable state that works for me, and accept that if I start using a new tool it might disrupt the system and I might need to rethink things.

• "Rules" that terminal programs follow
  Dec 12, 2024

  Recently I’ve been thinking about how everything that happens in the terminal is some combination of: Your operating system ’s job Your shell ’s job Your terminal emulator ’s job The job of whatever program you happen to be running (like top or vim or cat ) The first three (your operating system, shell, and terminal emulator) are all kind of known quantities – if you’re using bash in GNOME Terminal on Linux, you can more or less reason about how how all of those things interact, and some of their behaviour is standardized by POSIX. But the fourth one (“whatever program you happen to be running”) feels like it could do ANYTHING. How are you supposed to know how a program is going to behave? This post is kind of long so here’s a quick table of contents: programs behave surprisingly consistently these are meant to be descriptive, not prescriptive it’s not always obvious which “rules” are the program’s responsibility to implement rule 1: noninteractive programs should quit when you press Ctrl-C rule 2: TUIs should quit when you press q rule 3: REPLs should quit when you press Ctrl-D on an empty line rule 4: don’t use more than 16 colours rule 5: vaguely support readline keybindings rule 5.1: Ctrl-W should delete the last word rule 6: disable colours when writing to a pipe rule 7: - means stdin/stdout these “rules” take a long time to learn programs behave surprisingly consistently As far as I know, there are no real standards for how programs in the terminal should behave – the closest things I know of are: POSIX, which mostly dictates how your terminal emulator / OS / shell should work together. I think it does specify a few things about how core utilities like cp should work but AFAIK it doesn’t have anything to say about how for example htop should behave. these command line interface guidelines But even though there are no standards, in my experience programs in the terminal behave in a pretty consistent way. So I wanted to write down a list of “rules” that in my experience programs mostly follow. these are meant to be descriptive, not prescriptive My goal here isn’t to convince authors of terminal programs that they should follow any of these rules. There are lots of exceptions to these and often there’s a good reason for those exceptions. But it’s very useful for me to know what behaviour to expect from a random new terminal program that I’m using. Instead of “uh, programs could do literally anything”, it’s “ok, here are the basic rules I expect, and then I can keep a short mental list of exceptions”. So I’m just writing down what I’ve observed about how programs behave in my 20 years of using the terminal, why I think they behave that way, and some examples of cases where that rule is “broken”. it’s not always obvious which “rules” are the program’s responsibility to implement There are a bunch of common conventions that I think are pretty clearly the program’s responsibility to implement, like: config files should go in ~/.BLAHrc or ~/.config/BLAH/FILE or /etc/BLAH/ or something --help should print help text programs should print “regular” output to stdout and errors to stderr But in this post I’m going to focus on things that it’s not 100% obvious are the program’s responsibility. For example it feels to me like a “law of nature” that pressing Ctrl-D should quit a REPL, but programs often need to explicitly implement support for it – even though cat doesn’t need to implement Ctrl-D support, ipython does . (more about that in “rule 3” below) Understanding which things are the program’s responsibility makes it much less surprising when different programs’ implementations are slightly different. rule 1: noninteractive programs should quit when you press Ctrl-C The main reason for this rule is that noninteractive programs will quit by default on Ctrl-C if they don’t set up a SIGINT signal handler, so this is kind of a “you should act like the default” rule. Something that trips a lot of people up is that this doesn’t apply to interactive programs like python3 or bc or less . This is because in an interactive program, Ctrl-C has a different job – if the program is running an operation (like for example a search in less or some Python code in python3 ), then Ctrl-C will interrupt that operation but not stop the program. As an example of how this works in an interactive program: here’s the code in prompt-toolkit (the library that iPython uses for handling input) that aborts a search when you press Ctrl-C . rule 2: TUIs should quit when you press q TUI programs (like less or htop ) will usually quit when you press q . This rule doesn’t apply to any program where pressing q to quit wouldn’t make sense, like tmux or text editors. rule 3: REPLs should quit when you press Ctrl-D on an empty line REPLs (like python3 or ed ) will usually quit when you press Ctrl-D on an empty line. This rule is similar to the Ctrl-C rule – the reason for this is that by default if you’re running a program (like cat ) in “cooked mode”, then the operating system will return an EOF when you press Ctrl-D on an empty line. Most of the REPLs I use (sqlite3, python3, fish, bash, etc) don’t actually use cooked mode, but they all implement this keyboard shortcut anyway to mimic the default behaviour. For example, here’s the code in prompt-toolkit that quits when you press Ctrl-D, and here’s the same code in readline . I actually thought that this one was a “Law of Terminal Physics” until very recently because I’ve basically never seen it broken, but you can see that it’s just something that each individual input library has to implement in the links above. Someone pointed out that the Erlang REPL does not quit when you press Ctrl-D , so I guess not every REPL follows this “rule”. rule 4: don’t use more than 16 colours Terminal programs rarely use colours other than the base 16 ANSI colours. This is because if you specify colours with a hex code, it’s very likely to clash with some users’ background colour. For example if I print out some text as #EEEEEE , it would be almost invisible on a white background, though it would look fine on a dark background. But if you stick to the default 16 base colours, you have a much better chance that the user has configured those colours in their terminal emulator so that they work reasonably well with their background color. Another reason to stick to the default base 16 colours is that it makes less assumptions about what colours the terminal emulator supports. The only programs I usually see breaking this “rule” are text editors, for example Helix by default will use a purple background which is not a default ANSI colour. It seems fine for Helix to break this rule since Helix isn’t a “core” program and I assume any Helix user who doesn’t like that colorscheme will just change the theme. rule 5: vaguely support readline keybindings Almost every program I use supports readline keybindings if it would make sense to do so. For example, here are a bunch of different programs and a link to where they define Ctrl-E to go to the end of the line: ipython ( Ctrl-E defined here ) atuin ( Ctrl-E defined here ) fzf ( Ctrl-E defined here ) zsh ( Ctrl-E defined here ) fish ( Ctrl-E defined here ) tmux’s command prompt ( Ctrl-E defined here ) None of those programs actually uses readline directly, they just sort of mimic emacs/readline keybindings. They don’t always mimic them exactly : for example atuin seems to use Ctrl-A as a prefix, so Ctrl-A doesn’t go to the beginning of the line. Also all of these programs seem to implement their own internal cut and paste buffers so you can delete a line with Ctrl-U and then paste it with Ctrl-Y . The exceptions to this are: some programs (like git , cat , and nc ) don’t have any line editing support at all (except for backspace, Ctrl-W , and Ctrl-U ) as usual text editors are an exception, every text editor has its own approach to editing text I wrote more about this “what keybindings does a program support?” question in entering text in the terminal is complicated . rule 5.1: Ctrl-W should delete the last word I’ve never seen a program (other than a text editor) where Ctrl-W doesn’t delete the last word. This is similar to the Ctrl-C rule – by default if a program is in “cooked mode”, the OS will delete the last word if you press Ctrl-W , and delete the whole line if you press Ctrl-U . So usually programs will imitate that behaviour. I can’t think of any exceptions to this other than text editors but if there are I’d love to hear about them! rule 6: disable colours when writing to a pipe Most programs will disable colours when writing to a pipe. For example: rg blah will highlight all occurrences of blah in the output, but if the output is to a pipe or a file, it’ll turn off the highlighting. ls --color=auto will use colour when writing to a terminal, but not when writing to a pipe Both of those programs will also format their output differently when writing to the terminal: ls will organize files into columns, and ripgrep will group matches with headings. If you want to force the program to use colour (for example because you want to look at the colour), you can use unbuffer to force the program’s output to be a tty like this: unbuffer rg blah | less -R I’m sure that there are some programs that “break” this rule but I can’t think of any examples right now. Some programs have an --color flag that you can use to force colour to be on, in the example above you could also do rg --color=always | less -R . rule 7: - means stdin/stdout Usually if you pass - to a program instead of a filename, it’ll read from stdin or write to stdout (whichever is appropriate). For example, if you want to format the Python code that’s on your clipboard with black and then copy it, you could run: pbpaste | black - | pbcopy ( pbpaste is a Mac program, you can do something similar on Linux with xclip ) My impression is that most programs implement this if it would make sense and I can’t think of any exceptions right now, but I’m sure there are many exceptions. these “rules” take a long time to learn These rules took me a long time for me to learn because I had to: learn that the rule applied anywhere at all (" Ctrl-C will exit programs") notice some exceptions (“okay, Ctrl-C will exit find but not less ”) subconsciously figure out what the pattern is (" Ctrl-C will generally quit noninteractive programs, but in interactive programs it might interrupt the current operation instead of quitting the program") eventually maybe formulate it into an explicit rule that I know A lot of my understanding of the terminal is honestly still in the “subconscious pattern recognition” stage. The only reason I’ve been taking the time to make things explicit at all is because I’ve been trying to explain how it works to others. Hopefully writing down these “rules” explicitly will make learning some of this stuff a little bit faster for others.

• Why pipes sometimes get "stuck": buffering
  Nov 29, 2024

  Here’s a niche terminal problem that has bothered me for years but that I never really understood until a few weeks ago. Let’s say you’re running this command to watch for some specific output in a log file: tail -f /some/log/file | grep thing1 | grep thing2 If log lines are being added to the file relatively slowly, the result I’d see is… nothing! It doesn’t matter if there were matches in the log file or not, there just wouldn’t be any output. I internalized this as “uh, I guess pipes just get stuck sometimes and don’t show me the output, that’s weird”, and I’d handle it by just running grep thing1 /some/log/file | grep thing2 instead, which would work. So as I’ve been doing a terminal deep dive over the last few months I was really excited to finally learn exactly why this happens. why this happens: buffering The reason why “pipes get stuck” sometimes is that it’s VERY common for programs to buffer their output before writing it to a pipe or file. So the pipe is working fine, the problem is that the program never even wrote the data to the pipe! This is for performance reasons: writing all output immediately as soon as you can uses more system calls, so it’s more efficient to save up data until you have 8KB or so of data to write (or until the program exits) and THEN write it to the pipe. In this example: tail -f /some/log/file | grep thing1 | grep thing2 the problem is that grep thing1 is saving up all of its matches until it has 8KB of data to write, which might literally never happen. programs don’t buffer when writing to a terminal Part of why I found this so disorienting is that tail -f file | grep thing will work totally fine, but then when you add the second grep , it stops working!! The reason for this is that the way grep handles buffering depends on whether it’s writing to a terminal or not. Here’s how grep (and many other programs) decides to buffer its output: Check if stdout is a terminal or not using the isatty function If it’s a terminal, use line buffering (print every line immediately as soon as you have it) Otherwise, use “block buffering” – only print data if you have at least 8KB or so of data to print So if grep is writing directly to your terminal then you’ll see the line as soon as it’s printed, but if it’s writing to a pipe, you won’t. Of course the buffer size isn’t always 8KB for every program, it depends on the implementation. For grep the buffering is handled by libc, and libc’s buffer size is defined in the BUFSIZ variable. Here’s where that’s defined in glibc . (as an aside: “programs do not use 8KB output buffers when writing to a terminal” isn’t, like, a law of terminal physics, a program COULD use an 8KB buffer when writing output to a terminal if it wanted, it would just be extremely weird if it did that, I can’t think of any program that behaves that way) commands that buffer & commands that don’t One annoying thing about this buffering behaviour is that you kind of need to remember which commands buffer their output when writing to a pipe. Some commands that don’t buffer their output: tail cat tee I think almost everything else will buffer output, especially if it’s a command where you’re likely to be using it for batch processing. Here’s a list of some common commands that buffer their output when writing to a pipe, along with the flag that disables block buffering. grep ( --line-buffered ) sed ( -u ) awk (there’s a fflush() function) tcpdump ( -l ) jq ( -u ) tr ( -u ) cut (can’t disable buffering) Those are all the ones I can think of, lots of unix commands (like sort ) may or may not buffer their output but it doesn’t matter because sort can’t do anything until it finishes receiving input anyway. Also I did my best to test both the Mac OS and GNU versions of these but there are a lot of variations and I might have made some mistakes. programming languages where the default “print” statement buffers Also, here are a few programming language where the default print statement will buffer output when writing to a pipe, and some ways to disable buffering if you want: C (disable with setvbuf ) Python (disable with python -u , or PYTHONUNBUFFERED=1 , or sys.stdout.reconfigure(line_buffering=False) , or print(x, flush=True) ) Ruby (disable with STDOUT.sync = true ) Perl (disable with $| = 1 ) I assume that these languages are designed this way so that the default print function will be fast when you’re doing batch processing. Also whether output is buffered or not might depend on how you print, for example in C++ cout << "hello\n" buffers when writing to a pipe but cout << "hello" << endl will flush its output. when you press Ctrl-C on a pipe, the contents of the buffer are lost Let’s say you’re running this command as a hacky way to watch for DNS requests to example.com , and you forgot to pass -l to tcpdump: sudo tcpdump -ni any port 53 | grep example.com When you press Ctrl-C , what happens? In a magical perfect world, what I would want to happen is for tcpdump to flush its buffer, grep would search for example.com , and I would see all the output I missed. But in the real world, what happens is that all the programs get killed and the output in tcpdump ’s buffer is lost. I think this problem is probably unavoidable – I spent a little time with strace to see how this works and grep receives the SIGINT before tcpdump anyway so even if tcpdump tried to flush its buffer grep would already be dead. After a little more investigation, there is a workaround: if you find tcpdump ’s PID and kill -TERM $PID , then tcpdump will flush the buffer so you can see the output. That’s kind of a pain but I tested it and it seems to work. redirecting to a file also buffers It’s not just pipes, this will also buffer: sudo tcpdump -ni any port 53 > output.txt Redirecting to a file doesn’t have the same “ Ctrl-C will totally destroy the contents of the buffer” problem though – in my experience it usually behaves more like you’d want, where the contents of the buffer get written to the file before the program exits. I’m not 100% sure whether this is something you can always rely on or not. a bunch of potential ways to avoid buffering Okay, let’s talk solutions. Let’s say you’ve run this command: tail -f /some/log/file | grep thing1 | grep thing2 I asked people on Mastodon how they would solve this in practice and there were 5 basic approaches. Here they are: solution 1: run a program that finishes quickly Historically my solution to this has been to just avoid the “command writing to pipe slowly” situation completely and instead run a program that will finish quickly like this: cat /some/log/file | grep thing1 | grep thing2 | tail This doesn’t do the same thing as the original command but it does mean that you get to avoid thinking about these weird buffering issues. (you could also do grep thing1 /some/log/file but I often prefer to use an “unnecessary” cat ) solution 2: remember the “line buffer” flag to grep You could remember that grep has a flag to avoid buffering and pass it like this: tail -f /some/log/file | grep --line-buffered thing1 | grep thing2 solution 3: use awk Some people said that if they’re specifically dealing with a multiple greps situation, they’ll rewrite it to use a single awk instead, like this: tail -f /some/log/file | awk '/thing1/ && /thing2/' Or you would write a more complicated grep , like this: tail -f /some/log/file | grep -E 'thing1.*thing2' ( awk also buffers, so for this to work you’ll want awk to be the last command in the pipeline) solution 4: use stdbuf stdbuf uses LD_PRELOAD to turn off libc’s buffering, and you can use it to turn off output buffering like this: tail -f /some/log/file | stdbuf -o0 grep thing1 | grep thing2 Like any LD_PRELOAD solution it’s a bit unreliable – it doesn’t work on static binaries, I think won’t work if the program isn’t using libc’s buffering, and doesn’t always work on Mac OS. Harry Marr has a really nice How stdbuf works post. solution 5: use unbuffer unbuffer program will force the program’s output to be a TTY, which means that it’ll behave the way it normally would on a TTY (less buffering, colour output, etc). You could use it in this example like this: tail -f /some/log/file | unbuffer grep thing1 | grep thing2 Unlike stdbuf it will always work, though it might have unwanted side effects, for example grep thing1 ’s will also colour matches. If you want to install unbuffer, it’s in the expect package. that’s all the solutions I know about! It’s a bit hard for me to say which one is “best”, I think personally I’m mostly likely to use unbuffer because I know it’s always going to work. If I learn about more solutions I’ll try to add them to this post. I’m not really sure how often this comes up I think it’s not very common for me to have a program that slowly trickles data into a pipe like this, normally if I’m using a pipe a bunch of data gets written very quickly, processed by everything in the pipeline, and then everything exits. The only examples I can come up with right now are: tcpdump tail -f watching log files in a different way like with kubectl logs the output of a slow computation what if there were an environment variable to disable buffering? I think it would be cool if there were a standard environment variable to turn off buffering, like PYTHONUNBUFFERED in Python. I got this idea from a couple of blog posts by Mark Dominus in 2018. Maybe NO_BUFFER like NO_COLOR ? The design seems tricky to get right; Mark points out that NETBSD has environment variables called STDBUF , STDBUF1 , etc which gives you a ton of control over buffering but I imagine most developers don’t want to implement many different environment variables to handle a relatively minor edge case. I’m also curious about whether there are any programs that just automatically flush their output buffers after some period of time (like 1 second). It feels like it would be nice in theory but I can’t think of any program that does that so I imagine there are some downsides. stuff I left out Some things I didn’t talk about in this post since these posts have been getting pretty long recently and seriously does anyone REALLY want to read 3000 words about buffering? the difference between line buffering and having totally unbuffered output how buffering to stderr is different from buffering to stdout this post is only about buffering that happens inside the program , your operating system’s TTY driver also does a little bit of buffering sometimes other reasons you might need to flush your output other than “you’re writing to a pipe”

• Importing a frontend Javascript library without a build system
  Nov 18, 2024

  I like writing Javascript without a build system and for the millionth time yesterday I ran into a problem where I needed to figure out how to import a Javascript library in my code without using a build system, and it took FOREVER to figure out how to import it because the library’s setup instructions assume that you’re using a build system. Luckily at this point I’ve mostly learned how to navigate this situation and either successfully use the library or decide it’s too difficult and switch to a different library, so here’s the guide I wish I had to importing Javascript libraries years ago. I’m only going to talk about using Javacript libraries on the frontend, and only about how to use them in a no-build-system setup. In this post I’m going to talk about: the three main types of Javascript files a library might provide (ES Modules, the “classic” global variable kind, and CommonJS) how to figure out which types of files a Javascript library includes in its build ways to import each type of file in your code the three kinds of Javascript files There are 3 basic types of Javascript files a library can provide: the “classic” type of file that defines a global variable. This is the kind of file that you can just <script src> and it’ll Just Work. Great if you can get it but not always available an ES module (which may or may not depend on other files, we’ll get to that) a “CommonJS” module. This is for Node, you can’t use it in a browser at all without using a build system. I’m not sure if there’s a better name for the “classic” type but I’m just going to call it “classic”. Also there’s a type called “AMD” but I’m not sure how relevant it is in 2024. Now that we know the 3 types of files, let’s talk about how to figure out which of these the library actually provides! where to find the files: the NPM build Every Javascript library has a build which it uploads to NPM. You might be thinking (like I did originally) – Julia! The whole POINT is that we’re not using Node to build our library! Why are we talking about NPM? But if you’re using a link from a CDN like https://cdnjs.cloudflare.com/ajax/libs/Chart.js/4.4.1/chart.umd.min.js , you’re still using the NPM build! All the files on the CDNs originally come from NPM. Because of this, I sometimes like to npm install the library even if I’m not planning to use Node to build my library at all – I’ll just create a new temp folder, npm install there, and then delete it when I’m done. I like being able to poke around in the files in the NPM build on my filesystem, because then I can be 100% sure that I’m seeing everything that the library is making available in its build and that the CDN isn’t hiding something from me. So let’s npm install a few libraries and try to figure out what types of Javascript files they provide in their builds! example library 1: chart.js First let’s look inside Chart.js , a plotting library. $ cd /tmp/whatever $ npm install chart.js $ cd node_modules/chart.js/dist $ ls *.*js chart.cjs chart.js chart.umd.js helpers.cjs helpers.js This library seems to have 3 basic options: option 1: chart.cjs . The .cjs suffix tells me that this is a CommonJS file , for using in Node. This means it’s impossible to use it directly in the browser without some kind of build step. option 2: chart.js . The .js suffix by itself doesn’t tell us what kind of file it is, but if I open it up, I see import '@kurkle/color'; which is an immediate sign that this is an ES module – the import ... syntax is ES module syntax. option 3: chart.umd.js . “UMD” stands for “Universal Module Definition”, which I think means that you can use this file either with a basic <script src> , CommonJS, or some third thing called AMD that I don’t understand. how to use a UMD file When I was using Chart.js I picked Option 3. I just needed to add this to my code: <script src="./chart.umd.js"> </script> and then I could use the library with the global Chart environment variable. Couldn’t be easier. I just copied chart.umd.js into my Git repository so that I didn’t have to worry about using NPM or the CDNs going down or anything. the build files aren’t always in the dist directory A lot of libraries will put their build in the dist directory, but not always! The build files’ location is specified in the library’s package.json . For example here’s an excerpt from Chart.js’s package.json . "jsdelivr": "./dist/chart.umd.js", "unpkg": "./dist/chart.umd.js", "main": "./dist/chart.cjs", "module": "./dist/chart.js", I think this is saying that if you want to use an ES Module ( module ) you should use dist/chart.js , but the jsDelivr and unpkg CDNs should use ./dist/chart.umd.js . I guess main is for Node. chart.js ’s package.json also says "type": "module" , which according to this documentation tells Node to treat files as ES modules by default. I think it doesn’t tell us specifically which files are ES modules and which ones aren’t but it does tell us that something in there is an ES module. example library 2: @atcute/oauth-browser-client @atcute/oauth-browser-client is a library for logging into Bluesky with OAuth in the browser. Let’s see what kinds of Javascript files it provides in its build! $ npm install @atcute/oauth-browser-client $ cd node_modules/@atcute/oauth-browser-client/dist $ ls *js constants.js dpop.js environment.js errors.js index.js resolvers.js It seems like the only plausible root file in here is index.js , which looks something like this: export { configureOAuth } from './environment.js'; export * from './errors.js'; export * from './resolvers.js'; This export syntax means it’s an ES module . That means we can use it in the browser without a build step! Let’s see how to do that. how to use an ES module with importmaps Using an ES module isn’t an easy as just adding a <script src="whatever.js"> . Instead, if the ES module has dependencies (like @atcute/oauth-browser-client does) the steps are: Set up an import map in your HTML Put import statements like import { configureOAuth } from '@atcute/oauth-browser-client'; in your JS code Include your JS code in your HTML like this: <script type="module" src="YOURSCRIPT.js"></script> The reason we need an import map instead of just doing something like import { BrowserOAuthClient } from "./oauth-client-browser.js" is that internally the module has more import statements like import {something} from @atcute/client , and we need to tell the browser where to get the code for @atcute/client and all of its other dependencies. Here’s what the importmap I used looks like for @atcute/oauth-browser-client : <script type="importmap"> { "imports": { "nanoid": "./node_modules/nanoid/bin/dist/index.js", "nanoid/non-secure": "./node_modules/nanoid/non-secure/index.js", "nanoid/url-alphabet": "./node_modules/nanoid/url-alphabet/dist/index.js", "@atcute/oauth-browser-client": "./node_modules/@atcute/oauth-browser-client/dist/index.js", "@atcute/client": "./node_modules/@atcute/client/dist/index.js", "@atcute/client/utils/did": "./node_modules/@atcute/client/dist/utils/did.js" } } </script> Getting these import maps to work is pretty fiddly, I feel like there must be a tool to generate them automatically but I haven’t found one yet. It’s definitely possible to write a script that automatically generates the importmaps using esbuild’s metafile but I haven’t done that and maybe there’s a better way. I decided to set up importmaps yesterday to get github.com/jvns/bsky-oauth-example to work, so there’s some example code in that repo. Also someone pointed me to Simon Willison’s download-esm , which will download an ES module and rewrite the imports to point to the JS files directly so that you don’t need importmaps. I haven’t tried it yet but it seems like a great idea. problems with importmaps: too many files I did run into some problems with using importmaps in the browser though – it needed to download dozens of Javascript files to load my site, and my webserver in development couldn’t keep up for some reason. I kept seeing files fail to load randomly and then had to reload the page and hope that they would succeed this time. It wasn’t an issue anymore when I deployed my site to production, so I guess it was a problem with my local dev environment. Also one slightly annoying thing about ES modules in general is that you need to be running a webserver to use them, I’m sure this is for a good reason but it’s easier when you can just open your index.html file without starting a webserver. Because of the “too many files” thing I think actually using ES modules with importmaps in this way isn’t actually that appealing to me, but it’s good to know it’s possible. how to use an ES module without importmaps If the ES module doesn’t have dependencies then it’s even easier – you don’t need the importmaps! You can just: put <script type="module" src="YOURCODE.js"></script> in your HTML. The type="module" is important. put import {whatever} from "https://example.com/whatever.js" in YOURCODE.js alternative: use esbuild If you don’t want to use importmaps, you can also use a build system like esbuild . I talked about how to do that in Some notes on using esbuild , but this blog post is about ways to avoid build systems completely so I’m not going to talk about that option here. I do still like esbuild though and I think it’s a good option in this case. what’s the browser support for importmaps? CanIUse says that importmaps are in “Baseline 2023: newly available across major browsers” so my sense is that in 2024 that’s still maybe a little bit too new? I think I would use importmaps for some fun experimental code that I only wanted like myself and 12 people to use, but if I wanted my code to be more widely usable I’d use esbuild instead. example library 3: @atproto/oauth-client-browser Let’s look at one final example library! This is a different Bluesky auth library than @atcute/oauth-browser-client . $ npm install @atproto/oauth-client-browser $ cd node_modules/@atproto/oauth-client-browser/dist $ ls *js browser-oauth-client.js browser-oauth-database.js browser-runtime-implementation.js errors.js index.js indexed-db-store.js util.js Again, it seems like only real candidate file here is index.js . But this is a different situation from the previous example library! Let’s take a look at index.js : There’s a bunch of stuff like this in index.js : __exportStar(require("@atproto/oauth-client"), exports); __exportStar(require("./browser-oauth-client.js"), exports); __exportStar(require("./errors.js"), exports); var util_js_1 = require("./util.js"); This require() syntax is CommonJS syntax, which means that we can’t use this file in the browser at all, we need to use some kind of build step, and ESBuild won’t work either. Also in this library’s package.json it says "type": "commonjs" which is another way to tell it’s CommonJS. how to use a CommonJS module with esm.sh Originally I thought it was impossible to use CommonJS modules without learning a build system, but then someone Bluesky told me about esm.sh ! It’s a CDN that will translate anything into an ES Module. skypack.dev does something similar, I’m not sure what the difference is but one person mentioned that if one doesn’t work sometimes they’ll try the other one. For @atproto/oauth-client-browser using it seems pretty simple, I just need to put this in my HTML: <script type="module" src="script.js"> </script> and then put this in script.js . import { BrowserOAuthClient } from "https://esm.sh/@atproto/oauth-client-browser@0.3.0" It seems to Just Work, which is cool! Of course this is still sort of using a build system – it’s just that esm.sh is running the build instead of me. My main concerns with this approach are: I don’t really trust CDNs to keep working forever – usually I like to copy dependencies into my repository so that they don’t go away for some reason in the future. I’ve heard of some issues with CDNs having security compromises which scares me. I don’t really understand what esm.sh is doing. esbuild can also convert CommonJS modules into ES modules I also learned that you can also use esbuild to convert a CommonJS module into an ES module, though there are some limitations – the import { BrowserOAuthClient } from syntax doesn’t work. Here’s a github issue about that . I think the esbuild approach is probably more appealing to me than the esm.sh approach because it’s a tool that I already have on my computer so I trust it more. I haven’t experimented with this much yet though. summary of the three types of files Here’s a summary of the three types of JS files you might encounter, options for how to use them, and how to identify them. Unhelpfully a .js or .min.js file extension could be any of these 3 options, so if the file is something.js you need to do more detective work to figure out what you’re dealing with. “classic” JS files How to use it: : <script src="whatever.js"></script> Ways to identify it: The website has a big friendly banner in its setup instructions saying “Use this with a CDN!” or something A .umd.js extension Just try to put it in a <script src=... tag and see if it works ES Modules Ways to use it: If there are no dependencies, just import {whatever} from "./my-module.js" directly in your code If there are dependencies, create an importmap and import {whatever} from "my-module" or use download-esm to remove the need for an importmap Use esbuild or any ES Module bundler Ways to identify it: Look for an import or export statement. (not module.exports = ... , that’s CommonJS) An .mjs extension maybe "type": "module" in package.json (though it’s not clear to me which file exactly this refers to) CommonJS Modules Ways to use it: Use https://esm.sh to convert it into an ES module, like https://esm.sh/@atproto/oauth-client-browser@0.3.0 Use a build somehow (??) Ways to identify it: Look for require() or module.exports = ... in the code A .cjs extension maybe "type": "commonjs" in package.json (though it’s not clear to me which file exactly this refers to) it’s really nice to have ES modules standardized The main difference between CommonJS modules and ES modules from my perspective is that ES modules are actually a standard. This makes me feel a lot more confident using them, because browsers commit to backwards compatibility for web standards forever – if I write some code using ES modules today, I can feel sure that it’ll still work the same way in 15 years. It also makes me feel better about using tooling like esbuild because even if the esbuild project dies, because it’s implementing a standard it feels likely that there will be another similar tool in the future that I can replace it with. the JS community has built a lot of very cool tools A lot of the time when I talk about this stuff I get responses like “I hate javascript!!! it’s the worst!!!”. But my experience is that there are a lot of great tools for Javascript (I just learned about https://esm.sh yesterday which seems great! I love esbuild!), and that if I take the time to learn how things works I can take advantage of some of those tools and make my life a lot easier. So the goal of this post is definitely not to complain about Javascript, it’s to understand the landscape so I can use the tooling in a way that feels good to me. questions I still have Here are some questions I still have, I’ll add the answers into the post if I learn the answer. Is there a tool that automatically generates importmaps for an ES Module that I have set up locally? (apparently yes: jspm ) How can I convert a CommonJS module into an ES module on my computer, the way https://esm.sh does? (apparently esbuild can sort of do this, though named exports don’t work ) When people normally build CommonJS modules into regular JS code, what’s code is doing that? Obviously there are tools like webpack, rollup, esbuild, etc, but do those tools all implement their own JS parsers/static analysis? How many JS parsers are there out there? Is there any way to bundle an ES module into a single file (like atcute-client.js ), but so that in the browser I can still import multiple different paths from that file (like both @atcute/client/lexicons and @atcute/client )? all the tools Here’s a list of every tool we talked about in this post: Simon Willison’s download-esm which will download an ES module and convert the imports to point at JS files so you don’t need an importmap https://esm.sh/ and skypack.dev esbuild JSPM can generate importmaps Writing this post has made me think that even though I usually don’t want to have a build that I run every time I update the project, I might be willing to have a build step (using download-esm or something) that I run only once when setting up the project and never run again except maybe if I’m updating my dependency versions. that’s all! Thanks to Marco Rogers who taught me a lot of the things in this post. I’ve probably made some mistakes in this post and I’d love to know what they are – let me know on Bluesky or Mastodon!

Antirez

• Human coders are still better than LLMs
  May 29, 2025

  This is a short story of how humans are still so much more capable of LLMs. Note that I'm not anti-AI or alike, you know it if you know me / follow me somewhere. I use LLMs routinely, like I did today, when I want to test my ideas, for code reviews, to understand if there are better approaches than what I had in mind, to explore stuff at the limit of my expertise, and so forth (I wrote a blog post about coding with LLMs almost two years, when it was not exactly cool: I was already using LLMs for coding and never stopped, I'll have to write an update, but that's not the topic of this post). But, still: the current level of AI is useful, great too, but so incredibly behind human intelligence, and I want to remark this as lately it is impossible to have balanced conversations. So, today I was working to Vector Sets for Redis, to fix a complicated bug: during the time I stopped working at Redis my colleagues introduced resistance against corruption RDB and RESTORE payloads, even when the checksum of the data passes. This feature is disabled by default, but provides an enhanced layer of safety for people wanting it. But… there is a but as big as an elephant: In order to make HNSWs fast to save into Redis RDBs and to load back, I serialized the *graph* representation, and not the element-vector pairs, otherwise I would have to re-insert back data into HNSWs, and that would be, like, 100 times slower (!). So I store all the links the nodes have with other nodes, as integers, and then I resolve them into pointers, it’s a nice trick and works great. But if you mix this and random corruptions of the representation, and the fact that my own twist on HNSWs enforce reciprocal links between nodes (I wrote my own implementation of HNSWs with many useful features, but reciprocal links are needed to enable many of them) then this could happen: 1. We load corrupted data that says A links to B, but B no longer links to A (corrupted node IDs). 2. We delete node B: since the reciprocity is violated, we don’t clear the link from A to B. 3. Then we scan the graph and once we are at B we access A: use-after-free :-D :-) :-| So after loading data, I need to check that every link is reciprocal, and in the vanilla case this is going to be O(N^2), for each node we need to scan all the levels, for each level all the neighbors of the node, and check that it also links to this node by scanning its links at that level. Not good. # Human vs LLM To start, I implemented the vanilla approach, to see if the fuzzer could no longer find the bug, and it worked indeed, but loading times for a big vector set with 20 million vectors went from 45 seconds to 90 or something. WTF. So I opened a Gemini 2.5 PRO chat and told the LLM, hey, what we can do here? Is there a super fast way to do so? The best solution that Gemini could find was to say: order the pointers of the neighbors links, so you can use binary search. Oh, well, sure, I know this, I’m not really sure if in arrays of 16/32 pointers this is going to be faster or slower. So I asked, anything else? Nope, no better solution. So I told it: look, what about when we see A linking B at level X we store in a hash table A:B:X (but we sort A and B always so that A>B, and links are the same whatever the direction), and when we see the link again we clean it, this time we just scan the whole thing as we are already doing when resolving IDs to pointers in the links, and if at the end the hash table is not empty, we know there is some link that must be non-reciprocal? Gemini told me it was a nice idea, but there was the snprintf() to create the key and the hashing time and so forth, but yep, it was better than what my original approach (even sorting pointers). I made it notice that snprintf() was not needed. We could just memcpy() pointers in a fixed sized key. It recognized that it was possible to do so, then I realized something… Hey, I told Gemini, what about using a fixed accumulator for A:B:X? No hash table at all. Each time we see a link (A:B:X, so 8+8+4 bytes) we xor it in the current accumulator of 12 bytes. If we store it twice, it cancels out, so at the end if the register is non-zero, we know something is odd! However I anticipated Gemini that this system was potentially subject to collisions, and to evaluate them. Even if this feature is normally turned off in Redis, when users enable such extra checks they also often expect some more protection against an attacker deliberately crafting bad payloads. Gemini was quite impressed about the idea, but still told that pointers are… you know, similar in structure, change of a few bits, so if there were three spurious links L1, L2, L3 it could happen that the xor between L1 and L2 was the same as the L3 bits, and we could have a false negative (zero register). I also noticed that allocators tend to be very predictable and externally guessable. I asked Gemini for ways to improve upon this: it got no great ideas. Then I thought, wait, we can actually hash this with a good enough hash function that is still fast, murmur-128 or alike (we don’t need it to have cryptographic properties for this task), and proposed the following schema to Gemini: 1. Take the link A:B:X, but use a seed obtained via /dev/urandom to prefix all the keys with it, so we actually have S:A:B:X. 2. We just xor the output of murmur-128(S:A:B:X) into the 128 bit register. 3. At the end, we check if the register is 0 (all links reciprocal). I asked Gemini to do an analysis of that, and it was finally happy, saying that this makes it a lot harder both to casually find orphaned links that happen to xor to 0 together, and even that an external attacker could ever use this in a useful way, since “S” is not known, there is to control the pointers too, and all that it is really hard to put together. Also, this feature is a best effort extra protection that you need to enable, it is normally off and to be practical it should not pose a too big performance penalty. Well, all this to say: I just finished the analysis and stopped to write this blog post, I’m not sure if I’m going to use this system (but likely yes), but, the creativity of humans still have an edge, we are capable of really thinking out of the box, envisioning strange and imprecise solutions that can work better than others. This is something that is extremely hard for LLMs. Still, to verify all my ideas, Gemini was very useful, and maybe I started to think at the problem in such terms because I had a “smart duck” to talk with. Comments

• What I learned during the license switch
  May 02, 2025

  Yesterday, it was a very intense day. In Italy it was 1st of May, the workers holiday, so in the morning I went for a 4h walk in the Etna with friends Then at 6PM I was at home to release my blog post about the AGPL license switch, and I started following the comments, feedbacks, private messages, and I learned a few things in the process. 1. Regardless of the different few clauses, that IMHO make a difference, the AGPL vs SSPL main difference is that AGPL is "understood". In general, yesterday for the first time I realized that in licensing there is not just what you can do and can't do, but the degree a given license is understood, tested, adopted, ... 2. I was very touched by the words of Simon Willison on the matter (https://simonwillison.net/2025/May/1/redis-is-open-source-again/) because it is very peculiar that different persons, living in different parts of the world, but with a similar age and background in software, feel *so similar* about things. I, too, when was writing Vector Sets, was thinking: I would never use it if it wasn't going to be released under the AGPL (or other open source license I understand). This sentiment, multiplied by a non trivial fraction of the community, makes open source eventually win even in the complex software landscape that there is today. 3. People still care a lot about software distributions. Not that I didn't care, but in the past I burned my fingers with it. I was a very initial Linux user, with SlackWare 3.1 or something like that. During the years I wrote my device drivers, contributed a few patches to the kernel, during the years Debian had maybe ~10 packages of stuff that I wrote, from hping, to the Visitors web log analyzer, dump1090, Redis, and a few more. But, eventually, I started to see all the fragmentation, the rigidity of certain processes (binary compatibility of modules for the Linux kernel), the lack of a consistent design, the lack of a binary format for software distribution with all the libs inside, and so forth. I switched to MacOS on the desktop and continued using Linux on the server in a very pragmatic way, often times more happy to "tar xvzf software.tgz; make" than relying on what distributions offered. And, maybe, my obsession with shipping software with zero dependencies has something to do with it. But people still care a lot, and probably it is important to have Redis as distribution packages in many situations where you want to make things as automatic and reproducible as possible? Well, now there are many folks asking if Redis will re-enter the distributions. My take on that is simple: Redis and ValKey diverged already in a significant way, and will diverge a lot more in the future. I believe distributions should have both, so that users can have a choice, and sometimes this choice is forced by the features difference. Trivially: if you need to do vector similarity searches, you need to use Redis; if instead your company has a no-AGPL policy, you need to use ValKey, and so forth. 4. People are kind to me. In the comments around there were a few harsh takes, and this is normal and even healthy (after all it is part of the reason many companies believe more and more they can't use SSPL or other licenses but an OSI approved one). Yet, when addressing me personally, I see a lot of good words. I just want to say: thank you for all that. 5. We kinda live in a bubble. In one of the forums out there at some point somebody said: "But did you ever switched from Redis to one of the forks?", and there was a chain of comments: "never", "who cares if I can use it" and so forth. And this is true for ValKey too, that if people write apt-get install redis and ValKey is installed instead, and they use SET, GET, DEL, a few more, they don't care. What I mean is that software is no longer the one in 1998 (to use a very crucial and symbolic date for open source, the Internet, and myself) where we were all open source software license experts. Most people, especially the newer generations, have a different and more practical take. So all this is very important (vital, to me), but there is to understand that not every sensibility is alike. In the end, what is the most important aspect of all, is trying to ship good software. Comments

• Redis is open source again
  May 01, 2025

  Five months ago, I rejoined Redis and quickly started to talk with my colleagues about a possible switch to the AGPL license, only to discover that there was already an ongoing discussion, a very old one, too. Many people, within the company, had the feeling that the AGPL was a better pick than SSPL, and while eventually Redis switched to the SSPL license, the internal discussion continued. I tried to give more strength to the ongoing pro-AGPL license side. My feeling was that the SSPL, in practical terms, failed to be accepted by the community. The OSI wouldn’t accept it, nor would the software community regard the SSPL as an open license. In little time, I saw the hypothesis getting more and more traction, at all levels within the company hierarchy. I’ll be honest: I truly wanted the code I wrote for the new Vector Sets data type to be released under an open source license. Writing open source software is too rooted in me: I rarely wrote anything else in my career. I’m too old to start now. This may be childish, but I wrote Vector Sets with a huge amount of enthusiasm exactly because I knew Redis (and my new work) was going to be open source again. I understand that the core of our work is to improve Redis, to continue building a good system, useful, simple, able to change with the requirements of the software stack. Yet, returning back to an open source license is the basis for such efforts to be coherent with the Redis project, to be accepted by the user base, and to contribute to a human collective effort that is larger than any single company. So, honestly, while I can’t take credit for the license switch, I hope I contributed a little bit to it, because today I’m happy. I’m happy that Redis is open source software again, under the terms of the AGPLv3 license. Now, time to go back to the terminal, to show Redis users some respect by writing the best code I’m able to write, and make Vector Sets more useful and practical: I have a few more ideas for improvements, and I hope that more will be stimulated by your feedback (it is already happening). Good hacking! P.S. Redis 8, the first version of Redis with the new license, is also GA today, with a many new features and speed improvements of the core: https://redis.io/blog/redis-8-ga/ You can also find the Redis CEO blog post here: https://redis.io/blog/agplv3/ Comments

• Reproducing Hacker News writing style fingerprinting
  Apr 16, 2025

  About three years ago I saw a quite curious and interesting post on Hacker News. A student, Christopher Tarry, was able to use cosine similarity against a vector of top words frequencies in comments, in order to detect similar HN accounts — and, sometimes, even accounts actually controlled by the same user, that is, fake accounts used to uncover the identity of the writer. This is the original post: https://news.ycombinator.com/item?id=33755016 I was not aware, back then, of Burrows-Delta method for style detection: it seemed kinda magical that you just needed to normalize a frequency vector of top words to reach such quite remarkable results. I read a few wikipedia pages and took mental note of it. Then, as I was working with Vectors for Redis I remembered about this post, searched the web only to discover that the original page was gone and that the author, in the original post and website, didn’t really explained very well how the data was processed, the top words extracted (and, especially, how many were used) and so forth. I thought I could reproduce the work with Vector Sets, once I was done with the main work. Now the new data type is in the release candidate, and I found some time to work on the problem. This is a report of what I did, but before to continue, the mandatory demo site: you can play with it at the following link: https://antirez.com/hnstyle?username=pg&threshold=20&action=search NOTE: since the dataset takes 700MB of RAM, in my tiny server, in the next months I may take this down. However, later in this post you will find the link and the Github repository with the code to reproduce everything from scratch. NOTE2: I hope the web site will survive, it's a very crude Python script. I benchmarked the VSIM command in such a small server and yet it can deliver 80k VSIM per second! The wonders of int8 quantization, together with a few more optimizations. But the Python script is terrible, creates a new Redis connection each time and so forth. Fingers crossed. # Raw data download and processing Well, the first problem I had, in order to do something like that, was to find an archive with Hacker News comments. Luckily there was one with apparently everything posted on HN from the start to 2023, for a huge 10GB of total data. You can find it here: https://huggingface.co/datasets/OpenPipe/hacker-news and, honestly, I’m not really sure how this was obtained, if using scarping or if HN makes this data public in some way. Since I’m not a big fan of binary files, in the specific case of public datasets at least, I used two Python scripts in order to convert the Parquet files into something smaller and simpler to handle. The first script, gen-top-words.py, takes the binary files and generates a txt file with the list of the top N words used in the dataset. It generates 10k words by default, but for the statistical analysis a lot less are needed (or, actually: if you use too many words you no longer capture the style, but the kind of content a user is talking about!). Then, another Python script, accumulates all the comments for each single user and generates a very big JSONL file where there are just two keys: the user name and the frequency table of all the words used by a given user in all the history from HN starts to 2023. Each entry is like that: {"by": "rtghrhtr", "freqtab": {"everyone": 1, "hates": 1, "nvidia": 1, "but": 1, "treats": 1, "ati": 1, "as": 1, "an": 1, "afterthought": 1, "another": 1, "completely": 1, "useless": 1, "tool": 1, "to": 1, "throw": 1, "on": 1, "the": 1, "pile": 1}} At this point, the final script, insert.py, could do all the real work: to apply the Borrows method for each user, create the user style vector, and insert it into Redis. The advantage of pre-processing the files (a slow operation) is that the insertion script could be called more easily with different parameters (especially the number of top words to use) in order to see the different results more promptly, without the need to re-process the Parquet files each time. # How the Burrow method works? In the original post, Christopher wrote that you just need to normalize the frequency of the words usage and apply cosine similarity. Actually the process is a bit more involved. First, let’s ask ourselves, how this method actually works, in its essence? Well, it wants to capture words that each specific user over-uses or under-uses compared to the expected “average” language. To do so, we actually use the following steps (from the Python code). That’s what we do for each of the top words: # Convert to relative frequency rel_freq = frequency / total_words # Standardize using z-score: z = (freq - mean) / stddev mean = word_means.get(word, 0.0) stddev = word_stddevs.get(word, 1.0) # Default to 1.0 to avoid division by zero z_score = (rel_freq - mean) / stddev # Set the z-score directly in the vector at the word's index vector[word_to_index[word]] = z_score So we start by “centering” the frequency the user used a given word, by subtracting the *global* usage frequency for that word. This way, we have a number that describes how much the user under (negative) or over (positive) used such word. But, if you think at it, words that have a much higher variance among usage of different writers are less important, when they change. We want to amplify the signal of words that are under of over used by this user in a much greater way compared to the normal variance of the word. This is why we divide the centered frequency by the global standard deviation of the word. Now we have what is called the “z score”, an adjusted measure of how much a given word is an outlier in one or the other direction. Now, we are ready to insert the word into a Redis vector set, with just: VADD key FP32 [blob with 350 floats] username (I’ll not cover the details of vector sets here since you can find the doc here -> https://github.com/redis/redis/blob/unstable/modules/vector-sets/README.md) Note that Redis performs L2 normalization of the inserted vectors, but remembers the L2 value in order to return back the values when VEMB is used to retrieve the associated vector, so the z_score was set as it is. Finally, with VSIM, we can get similar users: 127.0.0.1:6379> vsim hn_fingerprint ele pg 1) "pg" 2) "karaterobot" 3) "Natsu" 4) "mattmaroon" 5) "chc" 6) "montrose" 7) "jfengel" 8) "emodendroket" 9) "vintermann" 10) "c3534l" All the code (but the webapp itself) can be found here: https://github.com/antirez/hnstyle The README file explains how to reproduce every part. # Why 350 words? One of the things missing in the original post that stimulated this blog post, is how many top words one should use. If you use too many words, you’ll see many comments of mine about Redis, since Redis is one of the top 10k words used. Guess what? I did exactly this error, initially, and VSIM continued to report users that talked about similar topics than myself, not with similar *style*. But fortunately the Internet Archive cached the Christopher results for the “pg” account, here: https://web.archive.org/web/20221126235433/https://stylometry.net/user?username=pg So now I could tune my top-k words to get similar results. Also, reading the original papers, I discovered that, with my surprise, for the analysis to work well you need even as little as 150 words. And in general the range from 150 to 500 is considered to be optimal. Warning: don’t believe that when you search for a user you’ll find mostly fake accounts. For many fake accounts there is too little data, as often people create throw away accounts, write a few comments, and that’s it. So most of the accounts associated with a given user style will be just other people that have a similar writing style. This method I believe is quite powerful in distinguishing who is a native speaker and who is not. This is especially clear from the vectors visualization below. # Validate and visualize… Another thing that I reproduced (also an idea from OP) was to try inserting the same users in two variants, like antirez_A and antirez_B, using two different set of comments. Then check if asking for similar users to antirez_A would report B. Indeed, for *most* of the users I tested this against, it worked very well, and often times it was the top result. So we know that actually our method works. But since from the vectors it is so easy to “see” a style, what about our naked eyes? Recently I switched to Ghostty as my terminal, and it supports the Kitty graphics protocol, so you can display bitmaps directly in the terminal window. It is quite some time I want to play with it. Finally I had a good reason to test this feature. img://antirez.com/misc/hnstyle_1.jpg What’s happening above is that we call the VEMB command, that returns just a list of floats (the vector). Then the vshow utility, also part of the repository, will care to find the smallest square that can contain the vector and show positive values in red, negative in green. As you can see, as a non native speaker I over-use very simple words and under-use more sophisticated words. Other authors stress certain specific words, others are much more “plain”, showing less artifacts. At some point I was curious about what was really happening there: what words I would use too much and too little? So in the demo website you can also press the button to analyze a given user, and see the top 10 words over-used and under-used. Well, a few of mine are definitely due to my issues with English grammar :D Ok, enough with this investigation! Vector sets are now in Redis 8 RC1 and I have more work to do, but this was fun, and I believe it shows that vectors were definitely cool even before AI. Thanks for reading such a long post. EDIT: I forgot to say that the insert.py script also inserts the JSON metadata with the total words written by the user. So you can use FILTER in order to only show matches with a given number of words. This can be useful to detect duplicated accounts since often they are used only seldom, when the identity must be covered: 127.0.0.1:6379> vsim hn_fingerprint ele pg FILTER ".wordcount 1) "montrose" 2) "kar5pt" 3) "ryusage" 4) "corwinstephen" 5) "ElfinTrousers" 6) "beaned" 7) "MichaelDickens" 8) "bananaface" 9) "area51org" 10) "william42" EDIT2: In case the matches look suspicious to you (meaningless), like tptacek noted in a comment in the HN submission of this blog post, here is a "visual" match that shows how, for instance, montrose and pg are really similar in the words usage patterns: img://antirez.com/misc/hnstyle_2.jpg Comments

• Vector Sets are part of Redis
  Apr 03, 2025

  Yesterday we finally merged vector sets into Redis, here you can find the README that explains in detail what you get: https://github.com/redis/redis/blob/unstable/modules/vector-sets/README.md The goal of the new data structure is, in short, to create a new “Set alike” data type, similar to Sorted Sets, where instead of having a scalar as a score, you have a vector, and you can add and remove elements the Redis way, without caring about anything except the properties of the abstract data structure Redis implements, ask for elements similar to a given query vector (or a vector associated to some element already in the set), and so forth. But more about that later, a bit of background, first: From the path of the README itself, you can see the implementation is into “modules”, but actually, Vector Sets are not a module, it’s a part of the Redis core, the thing is that I started developing them as a module, and later I suggested that the implementation should still use the modules API, in order to promote modularity of the internals of Redis, in order to have both the advantages: every Redis instance starting from Redis 8 will have Vector Sets as a native data type, and there are clear boundaries between the core and the implementation ## The first new main data type of Redis after… some time I think that the latest big data structure of Redis were Streams, also developed by me. I resigned, returned, forks happened in the meantime, and it still it looks like the burden to introduce a new data type in Redis is mine :D I must say: I’m ok with that, because as much as I like programming, I also like design, a lot, and I had a feeling, that vectors, and vector similarity, are conceptually very simple, so they deserved a very simple API. And that was what I tried to do. Vector Sets are still a beta feature but I can tell you something, I can guarantee you can learn the API in 3 minutes. I decided that a fundamental requirement for implementing vector similarity was to also reimplement from scratch HNSWs (you can see my implementation in hnsw.c), because that was going to be my core data structure, and I didn’t want to grab some random code from GitHub and be happy with it. However, as I started reading the papers, I started to understand that a few pieces were missing. So, as I did in the past with HyperLogLog, where I had to fill a few gaps (here: https://antirez.com/news/75), there was already some new algorithmic challenges. Especially I wanted two things: 1. To have true deletions of nodes. In Vector Sets you can add new elements with VADD, and you can remove elements with VREM. And I wanted the memory to be reclaimed ASAP. 2. I wanted to be sure that as you delete elements, the connectivity properties of the HNSW graph were retained. So this lead me to a few differences compared to other implementations of HNSW. I don’t use tombstone deletions, but effectively unlink the node in the moment it gets deleted, relinking it back with other potential good neighbors. To do so, in turn, my implementation is designed to enforce that links must be reciprocal, it’s no longer a best effort property: this in turn changes quite a bit what you need to do during insertions. Another modification I did in the HNSW was to support the ability to scan the graph with a predicate function, so that you can ask for nodes matching a given expression. This requires to modify the greedy graph scanning algorithm in some way: to collect potential nodes to visit, and to collect the result set, but also to have some early stop condition in case there is too much selectivity in the query. We don’t want trigger a full graph scan, of course. Other than the HNSW modifications, I wanted a few more things that are more pragmatic and obvious: 1. Threading of all the vector similarity requests. Yeah, that’s new in Redis land, but as much as I believe single thread and shared nothing was a good design in general, I think that vectors are special. They are slow, a lot slower than other data structures modeled by Redis. As a bonus point, as I was implementing threaded VSIM (the command that performs vector similarity queries) I also discovered that with a few tricks you can split the read half and the write half of writes in two parts, so that neighbors candidate collections happen in the background, and the actual insertion is performed in the foreground. This splitting however is not the default, and you need to force it with the CAS option of VADD. 2. I wanted to support quantization and even make it the default. So Vector Sets ship with both 8 bit quantization and binary quantization. There is also support for random projection for dimensionality reduction. However as much as I like RP and having bin quants, the reality is that the “killer” for me is int8 quants. They are super fast, take 25% of the memory of FP32, and the results are nearly identical to those from full vectors for most vectors generated via embedding AI models. Btw the end result is, I believe, a very fast implementation. For instance in my machine, with a 3 million items vector set of 300 components each, I get 50/60k VSIM (top 10 items) per second on my laptop. But I encourage you to do your benchmarks. Also note that Vector Sets are serialized on disk as a graph, so when they are loaded back in memory, after a Redis restart, you don’t pay back the insertion time: loading every million of elements take a few seconds and not the minutes needed otherwise to add back into the in-memory HNSW. ## Data structures, not indexes What I’ve said so far is all about the low level stuff. But for me the most interesting part of Vector Sets is the data model and the API supporting it. Many databases propose vector similarity as a kind of index, but that’s Redis, and things in Redis are data structures: no exception this time. You add stuff like that: > VADD mykey FP32 …blob of data… item1 And so forth. So you can have many small vector sets if you want, one per key. And an important thing here is that if you split your vectors into N different keys (hashing the item you are inserting or alike to select which key to pick), then you can merge different VSIM calls against different keys into a single reply: > VSIM word_embeddings_int8 ele "banana" WITHSCORES COUNT 4 1) "banana" 2) "0.9997616112232208" 3) "bananas" 4) "0.8758847117424011" 5) "pineapple" 6) "0.8288004100322723" 7) "mango" 8) "0.8179697692394257" If I get a few of those results from different keys and instances, I can sort by the score (where 1 means identical, 0 opposite vector) and that’s it. So, my feeling is that Vector Sets can be composed into different patterns to handle having many vectors (they consume quite a bit of RAM) into different instances and so forth. Also it is interesting that splitting linearly scale writes, since each subset will hit a given key, and multiple insertions are possible in parallel. As usually, the Redis community will likely figure many usage patterns that are now not obvious. ## How filtering works About filtering, if threading was not common in Redis, let’s imagine JSON! But for the first time, I found a good reason to expose JSON directly in the Redis API, user facing: > VGETATTR word_embeddings_int8 banana {"len": 6} So basically with VSETATTR / VGETATTR (and equivalent option to set the JSON attribute directly when adding the item in VADD) you can associate a string to the items you want. Then you can do things like that: > VSIM word_embeddings_int8 ele "banana" FILTER ".len == 3" 1) "yam" 2) "pea" 3) "fig" 4) "rum" 5) "ube" 6) "oat" 7) "nut" 8) "gum" 9) "soy" 10) "pua" The filter expression is not a programming language, is what you could write inside the if() statement of high level programming languages, with &&, ||, all the obvious operators and so forth (but I bet we will add a few more). Well, the details are into the doc, there are also memory usage examples, in depth discussions about specific features, and so forth. I will extend the documentation soon, I hope. For now I really (really!) hope you’ll enjoy Vector Sets. Please, ping me if you find bugs :) Comments

• AI is useless, but it is our best bet for the future
  Mar 23, 2025

  I used AI with success 5 minutes ago. Just five minutes ago, I was writing a piece of software and relied on AI for assistance. Yet, here I am, starting this blog post by telling you that artificial intelligence, so far, has proven somewhat useless. How can I make such a statement if AI was just so helpful a moment ago? Actually, there's no contradiction here if we clarify exactly what we mean. Here’s the thing: at this very moment, artificial intelligence can support me significantly. If I'm struggling with complicated code or need to understand an advanced scientific paper on math, I can turn to AI for clarity. It can help me generate an image for a project, make a translation, clean my YouTube transcript. Clearly, it’s practical and beneficial in these everyday tasks. However, except for rare, groundbreaking examples like AlphaFold — Google's AI that significantly advanced our understanding of protein folding — AI has yet to genuinely push forward human knowledge in a fundamental way. Aside from these few exceptional results, AI hasn’t (obviously) yet matched the capabilities of the very best human minds. If an AI system were at the same level as the brightest humans (and not better than that: it's not needed for a first humanity jump) we could deploy millions of such systems to accelerate research dramatically, transforming progress expected to take centuries into developments happening within decades, or decades into years. Yet, if artificial intelligence remains stuck at its current level of development indefinitely (even if with small incremental improvements, enough to fire many translators, programmers, drivers, actors, ...), perhaps it might have been better not to have it at all. I mentioned this during a conference here in Sicily. The thought hadn't crossed my mind until I was asked on stage. While I was formulating my reply I asked myself: if we knew AI would only yield minor incremental improvements, would it be worth enduring the social upheaval caused by job losses and other stresses? Possibly not. Technologies should serve humanity by enabling greater cultural development, reducing suffering, and allowing us to achieve what otherwise would be impossible. The current level of AI, while helpful, doesn't fully achieve that. That's why investing in AI is like making a bet. I advocate for further investment and continued progress — not necessarily because of what AI can currently do, but because of what it might become in the future. The advancements we see today often exceed our expectations, hinting at even greater unforeseen breakthroughs tomorrow. For us proponents of AI, the argument shouldn't hinge solely on AI’s current abilities but rather on its potential. Five years from now, AI could offer revolutionary advancements in medicine, saving countless lives. In 10 to 20 years, it might significantly contribute to environmental solutions, clarifying complex issues or providing effective methods to mitigate climate change. Even a modest breakthrough might clarify that climate change dynamics, while serious, can be controlled more effectively than we currently believe, or don't need control as they are not as dramatic as we may think, or that decisive action can resolve the problem within a manageable timeframe. The real stakes are in the future, not the present. To focus exclusively on today's AI capabilities limits our perspective and makes it challenging to convince skeptics. But if future developments meet or exceed expectations, the temporary social problems arising today would be a small price to pay for the immense benefits. However, we must be cautious. Existential risks (AI potentially becoming a catastrophic threat) are real, though minimal. We must remain vigilant and prepared. Social challenges, too, require thoughtful attention from governments and societies. Ultimately, I believe we must take the risk, persist, and explore what's beyond the frontier, because if AI fulfills even some of its enormous potential, it could redefine our capabilities, reshape society, and completely transform humanity’s future: for the better. Now, let me go back to my LLM for free and fast code review. Comments

• Big LLMs weights are a piece of history
  Mar 16, 2025

  By multiple accounts, the web is losing pieces: every year a fraction of old web pages disappear, lost forever. We should regard the Internet Archive as one of the most valuable pieces of modern history; instead, many companies and entities make the chances of the Archive to survive, and accumulate what otherwise will be lost, harder and harder. I understand that the Archive headquarters are located in what used to be a church: well, there is no better way to think of it than as a sacred place. Imagine the long hours spent by old programmers hacking with the Z80 assembly on their Spectrums. All the discussions about the first generation of the Internet. The subcultures that appeared during the 90s. All things that are getting lost, piece by piece. And what about the personal blogs? Pieces of life of single individuals that dumped part of their consciousness on the Internet. Scientific papers and processes that are lost forever as publishers fail, their websites shut down. Early digital art, video games, climate data once published on the Internet and now lost, and many sources of news, as well. This is a known issue and I believe that the obvious approach of trying to preserve everything is going to fail, for practical reasons: a lot of efforts for zero economic gains: the current version of the world is not exactly the best place to make efforts that cost a lot of money and don't pay money. This is why I believe that the LLMs' ability to compress information, even if imprecise, hallucinated, lacking, is better than nothing. DeepSeek V3 is already an available, public lossy compressed view of the Internet, as other very large state of-art models are. This will not bring back all the things we are losing, and we should try hard supporting The Internet Archive and other similar institutions and efforts. But, at the same time, we should focus on a much simpler effort: to make sure that the weights of LLMs publicly released do not get lost, and also to make sure that the Archive is part of the pre-training set as well. Comments

• Reasoning models are just LLMs
  Feb 09, 2025

  It’s not new, but it’s accelerating. People that used to say that LLMs were a fundamentally flawed way to reach any useful reasoning and, in general, to develop any useful tool with some degree of generality, are starting to shuffle the deck, in the hope to look less wrong. They say: “the progresses we are seeing are due to the fact that models like OpenAI o1 or DeepSeek R1 are not just LLMs”. This is false, and it is important to show their mystification as soon as possible. First, DeepSeek R1 (don’t want to talk about o1 / o3, since it’s a private thing we don’t have access to, but it’s very likely the same) is a pure decoder only autoregressive model. It’s the same next token prediction that was so strongly criticized. There isn’t, in any place of the model, any explicit symbolic reasoning or representation. Moreover, R1 Zero has similar reasoning capabilities of R1 without requiring *any* supervised fine tuning, just generating chain of thoughts, and improving it with a reward function, using reinforcement learning, was enough to learn a stronger form of reasoning. Interestingly enough, part of these capabilities were easily distilled into smaller models via SFT, which brings me to the next point. The other fundamental observation is that the S1 paper shows that you need very few examples (as little as 1000) in order for the model to start being able to build complex reasoning steps and solve non trivial mathematical problems. S1, and R1 Zero, hint that in some way in the pre-training step the models already learned the representations needed in order to perform reasoning, just with the unsupervised next word prediction training target. So it’s not just that R1 is a vanilla LLM in its fundamental structure, but also the unsupervised pre-training creates enough representations and potential that, powerful enough LLMs, with RL (and/or some minor SFT), learn to reply to complex questions the users pose (I'm referring to instruct models, an old but yet impressive capability) and to use chain of thoughts to reason about things and provide better answers. Reasoning models are just LLMs, and who said LLMs were a dead end was just wrong. Now, to be wrong happens (even if the version of being wrong, in this instance, was particularly aggressive, particularly in denial of evidences). However, trying to change the history and the terminology in order to be in the right side is, for me, unacceptable. Comments

• We are destroying software
  Feb 08, 2025

  We are destroying software by no longer taking complexity into account when adding features or optimizing some dimension. We are destroying software with complex build systems. We are destroying software with an absurd chain of dependencies, making everything bloated and fragile. We are destroying software telling new programmers: “Don’t reinvent the wheel!”. But, reinventing the wheel is how you learn how things work, and is the first step to make new, different wheels. We are destroying software by no longer caring about backward APIs compatibility. We are destroying software pushing for rewrites of things that work. We are destroying software by jumping on every new language, paradigm, and framework. We are destroying software by always underestimating how hard it is to work with existing complex libraries VS creating our stuff. We are destroying software by always thinking that the de-facto standard for XYZ is better than what we can do, tailored specifically for our use case. We are destroying software claiming that code comments are useless. We are destroying software mistaking it for a purely engineering discipline. We are destroying software by making systems that no longer scale down: simple things should be simple to accomplish, in any system. We are destroying software trying to produce code as fast as possible, not as well designed as possible. We are destroying software, and what will be left will no longer give us the joy of hacking. Comments

• From where I left
  Dec 10, 2024

  I’m not the kind of person that develops a strong attachment to their own work. When I decided to leave Redis, about 1620 days ago (~ 4.44 years), I never looked at the source code, commit messages, or anything related to Redis again. From time to time, when I needed Redis, I just downloaded it and compiled it. I just typed “make” and I was very happy to see that, after many years, building Redis was still so simple. My detachment was not the result of me hating my past work. While in the long run my creative work was less and less important and the “handling the project” activities became more and more substantial — a shift that many programmers are able to do, but that’s not my bread and butter — well, I still enjoyed doing Redis stuff when I left. However, I don’t share the vision that most people at my age (I’m 47 now) have: that they are still young. I wanted to do new stuff, especially writing. I wanted to stay more with my family and help my relatives. I definitely needed a break. However, during the “writing years” (I’m still writing, by the way), I often returned to coding, as a way to take breaks from intense writing sessions (writing is the only mental activity I found to be a great deal more taxing than coding): I did a few embedded projects; played more with neural networks; built Telegram bots: a bit of everything. Hacking randomly was cool but, in the long run, my feeling was that I was lacking a real purpose, and every day I started to feel a bigger urgency to be part of the tech world again. At the same time, I saw the Redis community fragmenting, something that was a bit concerning to me, even as an outsider. So I started to think that maybe, after all, I could have a role back in the Redis ecosystem. Perhaps I could be able to reshape the company's attitude towards the community. Maybe I could even help to take back the role of the Redis core as the primary focus of new developments. Basically I could be some kind of “evangelist” (I don’t love the name of this role, but… well, you get it), that is, on one side, a bridge between the company and the community, but also somebody that could produce programming demos, invent and describe new patterns, write documentation, videos and blog posts about new and old stuff. And, what about the design of new stuff? I could learn from the work of people in the wild, from their difficulties, distill it, and report back design ideas, in order for Redis to evolve. # Time in NY At some point my daughter, who is now 12, and is a crucial person in my life, enlightening my days with her intelligence, creativity and love, wanted to visit NYC for her birthday. We decided that yes, this was a good idea after all, we had a couple very difficult years recently, so, why not? My daughter is now more of a girl than a child. So, in NYC, I thought: maybe this is the right time, I can do a part time job. I met the new Redis CEO, Rowan Trollope, very recently, in a video call. I had the feeling that I could work with him to tune the future of the company’s relationships with the community and the codebase direction. So I wrote him an email saying: do you think I could be back in some kind of capacity? Rowan showed interest in my proposal, and quickly we found some agreement. # About the license switch People will ask questions about why I *actually* did this, whether there is some back story other than what I just wrote above, if there is some agreement involved, or a big amount of money; something odd or unclear. But sometimes things are very boring: 1. I contacted the company, not the reverse. 2. I’m not getting crazy money to re-enter, it’s not about exploiting some situation — normal salary (but, disclaimer: yes, I have Redis stock options like I had before, no less, no more). 3. I don’t have huge issues with Redis changing its license; Specifically I don’t think the fracture with the community is *really* about this. But since people will ask me about that very important matter, it is better to tell you all the truth immediately. # The licensing dilemma I wrote open source software for almost my whole life. Yet, as I’m an atheist and still I’m happy when I see other people believing in God, if this helps them to survive life’s hardships, I also don’t believe that open source is the only way to write software. When I started to develop Redis in the context of a company where I was one of the two founders and where we kept the software code closed (Redis was opened as it was considered not part of the core product). We didn't want our services to be copied by others, as simple as that. So I’m not an extremist in this regard – I’m an extremist only about software design. Moreover, I don’t believe that openness and licensing are only what the OSI tells us they are. I see licensing as a spectrum of things you can and can’t do. At the same time, I’m truly concerned that big cloud providers have changed the incentives in the system software arena. Redis was not the only project to change license, it was actually the last one… of a big pile. And I have the feeling that in recent years many projects didn’t even start because of a lack of a clear potential business model. So, the Redis license switch was not my decision and perhaps I would have chosen a different license? I’m not sure, it’s too easy to relitigate right now, far from the scene for many years and without business pressures. But in general, I can understand the choice. Moreover, if you read the new Redis license, sure, it’s not BSD, but basically as long as you don’t sell Redis as a service, you can use it in very similar ways and with similar freedoms as before (what I mean is that you can still modify Redis, redistribute it, use Redis commercially, in your for-profit company, for free, and so forth). You can *even* still sell Redis as a service if you want, as long as you release all the orchestration systems under the same license (something that nobody would likely do, but this shows the copyleft approach of the license). The license language is almost the same as the AGPL, with changes regarding the SAAS stuff. So, not OSI approved? Yes, but I have issues calling the SSPL a closed license. You will say (I can hear you): the real problem is that there are companies controlling OSS projects’ direction! So eventually the interests get more aligned with companies and less with the user base. I’m grateful there are many projects out there with zero direct companies involvement (if not external sponsorship), but well, you know what? Involvement of companies, in many large projects, actually slows down this process of skewing the right path. This surely happened in the case of Redis. # The Robin Hood of software Let's jump back into the past, to the first days of Redis. When Redis started to become popular, I wanted to find a way to continue working on it. This was before VMware offered me to sponsor my work. I started to play with the idea of a business model, and guess what? It was in the form of closed source products that would kinda help people running Redis, in some way or the other. (Amazingly, one of the repositories associated with this idea is still online, showing commits of *15* years ago: https://github.com/antirez/redis-tools) I was about to try some kind of open core approach; I also remember I was thinking about delaying the BSD license for new code for six months, in order to create some kind of advantage for paying users. Now I don’t believe I would be a jerk and do strange games with my users, but I would not be what I was able to be thanks to VMware and, later, more extensively thanks to Redis Labs later: a freaking Robin Hood of open source software, where I was well compensated by a company and no, not to do the interests of the company itself, but only to make the best interests of the Redis community. This is a better setup than having your own company, I’m sure about that. VMWare, and later Redis Labs, didn’t pay just for me. If you give a quick look at the repository contributions history, you’ll see the second all time contributor to be Oran Agra (Redis) then there is Pieter Noordhuis (VMWare) and so forth. So basically I think that 12 years of BSD code written just focusing on the user base is a good deal, and it’s something to be happy about. And right now for me the most important part is that the fracture with the community is not about licensing, or at least it’s not mainly about licensing. Actually the new license can solve some part of it: now there is no longer an incentive to just leave the core in maintenance mode and put the new developments into modules. With the new license, cloud providers can’t just cut and paste the Redis code base and sell it without any revenue sharing (was this really asking for too much? This could have prevented all the license switches you saw lately, not just Redis). With the new license, the spotlight can be back on the Redis core, with new, exciting features in the hands of the developers around the world. With tens of people well compensated for their work pushing useful, well documented changes in the GitHub repository. This is also one of the things I would like to help the company with, and I’ll try hard. We need to make the license switch having good effects on the user base and features: that’s my idea. # About AI, LLMs and vector indexing But there is more: Redis is getting interested in developing vector capabilities, and in general to support the kind of programming you can do with AI. Now, every day, I read Hacker News, and I see a huge amount of technical people who dislike AI and the new developments. I also see a lot of people who don't even care to really try the latest models available in depth (hint: Claude AI is in its own league) and still dismiss them as kinda useless. For me, it’s different. I always loved neural networks. I wrote my first NN library in 2003 and was totally shocked by how powerful and cool the whole concept was. And now, at the end of 2024, I’m finally seeing incredible results in the field, things that looked like sci-fi a few years ago are now possible: Claude AI is my reasoning / editor / coding partner lately. I’m able to accomplish a lot more than I was able to do in the past. I often do *more work* because of AI, but I do better work. Recently I wrote a sci-fi short story for an Italian publisher, and thanks to Claude criticizing parts of it I rewrote the ending and produced a much better work (I didn’t let Claude write a single line of the story or the plot: great use of AI is not making machines do what you can do better). Yesterday I needed to evaluate how much faster dot product could be computed with 8 bit quantization of my vectors; I told Claude I needed a benchmark designed in a specific way, and two minutes later I could test it, modify it, and understand whether it was worthwhile or not. Basically, AI didn’t replace me, AI accelerated me or improved me with feedback about my work. And I believe that (regardless of the popularity of RAG, which is not necessarily the main application, nor the most future proof or useful, as models contexts are becoming larger and larger, and soon popular models attention may have linear complexity), sorry for the digression, I was saying that I believe that learned embeddings are here to stay, and vector search is something that belongs to Redis for several reasons: first because vector indexes are data structures, particularly slow data structures, and such data structures can work very well in memory. Also, because I think I found the perfect API to expose them. During my work in designing Redis, I always showed some contradictory tendencies. I was always ready to say “no” to certain things that looked like perfect fits for the project (named Lua scripts, hash fields expires, that are both part of Redis now, btw) but at the same time I added Lua scripting capabilities — when it looked like nuts, an interpreter inside Redis?! —, the Pub/Sub capability, that seemed out of context, then streams, and even synthetic data structures that don’t exist in computer science books, like sorted sets. Because, for me, the fitness of new features into Redis was about two things: use cases and internal design fit. Redis, for me, is lego for programmers, not a “product”. # Vector sets So recently I started to think that sorted sets can inspire a new data type, where the score is actually a vector. And while I was in talks with Rowan, I started to write a design document, then I started to implement a proof of concept of the new data structure, reimplementing HNSWs from scratch (instead of using one of the available libraries, since I wanted to tune every little bit), the Redis way, and well, I’m not sure how this will end, I’m still in the early stages of coding, but perhaps I may end up contributing code again, if this proposal gets accepted. The module I implemented (that would be later merged into the core – for now it’s a module just for the sake of simplicity) implements new commands that manipulate embeddings directly. I’ll show you just that as a hint: VSIM top_1000_movies_imdb ELE "The Matrix" WITHSCORES 1) "The Matrix" 2) "0.9999999403953552" 3) "Ex Machina" 4) "0.8680362105369568" 5) "Akira" 6) "0.8635441958904266" 7) "District 9" 8) "0.8631418347358704" 9) "The Martian" 10) "0.8608670234680176" 11) "The Bourne Ultimatum" 12) "0.8599717319011688" 13) "The Bourne Supremacy" 14) "0.8591427505016327" 15) "Blade Runner" 16) "0.8585404753684998" 17) "Metropolis" 18) "0.8572960793972015" 19) "Inception" 20) "0.8521313071250916" So you have VSIM, VADD, VCARD, all the obvious stuff. It’s exactly the idea of sorted sets, but with multi-dimensional scores (embeddings!) and K-NN matches. What do you think? And, of course, on top of that there are many implementation tricks to make stuff more efficient. But for now it’s proof of concept code, let me work a bit more on it. I’m implementing threading, dimensionality reduction, quantization, and many more things. Quite fun, to be honest. As you can see, there is no mention of hybrid search, the recent buzzword about vector stores. Again, this is the Redis way: to let the developer have a role and decide on their tradeoffs: they know what they are modeling, after all. You have a vector index per key, and like what programmers were able to do with sorted sets, they will invent interesting splitting strategies, new schemas, Lua scripts, patterns and all that is required in order to model their use cases. Still, while normally the associated item will likely be a small string or a document ID, nothing prevents it from being something more complex, with metadata that can be filtered later (but I’ll resist). I just have the feeling that many use cases don’t really need complex server-side filtering, and can be modeled by pre-partitioning data. What I see with great interest is the addition of a potential STORE option, to store the result into a sorted set instead of returning it to the user, where the score is the similarity, of course. All this also has complex and interesting effects on efficiency, scalability, ability to use scripting, and so forth: I hope I’ll have the opportunity to talk more about it in the next weeks and months. Ok, ok: back to the point of this blog post. But perhaps the above is the *real* point, having new ideas that can be exciting. # So, I’m back 🙂 All this to say that, I’m back. I think it’s the right moment for a big thank you to all the Redis community, for what it has done over the years. See you around, I hope there is something more to add to this journey. P.S. I’m active on BlueSky, if you want to follow the developments of all this. https://bsky.app/profile/antirez.bsky.social Comments

• Playing audio files in a Pi Pico without a DAC
  Mar 06, 2024

  The Raspberry Pico is suddenly becoming my preferred chip for embedded development. It is well made, durable hardware, with a ton of features that appear designed with smartness and passion (the state machines driving the GPIOs are a killer feature!). Its main weakness, the lack of connectivity, is now resolved by the W variant. The data sheet is excellent and documents every aspect of the chip. Moreover, it is well supported by MicroPython (which I’m using a lot), and the C SDK environment is decent, even if full of useless complexities like today fashion demands: a cmake build system that in turn generates a Makefile, files to define this and that (used libraries, debug outputs, …), and in general a huge overkill for the goal of compiling tiny programs for tiny devices. No, it’s worse than that: all this complexity to generate programs for a FIXED hardware with a fixed set of features (if not for the W / non-W variant). Enough with the rant about how much today software sucks, but it must be remembered. One of the cool things one wants to do with an MCU like that, is generating some sound. The most obvious way to do this is using the built-in PWM feature of the chip. The GPIOs can be configured to just alterante between zero and one at the desired frequency, like that: from machine import Pin, PWM pwm = PWM(Pin(1)) pwm.freq(400) pwm.duty_u16(1000) Assuming you connected a piezo to GND and pin 1 of your Pico, you will hear a square wave sound at 400hz of frequency. Now, there are little sounds as terrible to hear as square waves. Maybe we can do better. I’ll skip all the intermediate steps here, like producing a sin wave, and directly jump to playing a wav file. Once you see how to do that, you can easily generate your own other waves (sin, noise, envelops for such waveforms and so forth). Now you are likely asking yourself: how can I generate the complex wave forms to play a wav file, if the Pico can only switch the pin high or low? A proper non square waveform is composed of different levels, so I would need a DAC! Fortunately we can do all this without a DAC at all, just a single pin of our Pico. ### How complex sound generation works I don’t want to cover too much background here. But all you need to know is that, if you don’t want to generate a trivial square wave, that just alternates between a minimum and maximum level of output, you will need to have intermediate steps, like that: S0: # S1: #### S2: ###### S3: ####### S4: ######## And so forth, where S0 is the first sample, S1, the second sample, … Each sample duration depends on the sampling frequency, that is how many times every second we change (when playing) or sample (when recording) the audio wave. This means that to play a complex sound, we need the ability of our Pico pin to output different voltages. There is a trick to do this with the Pico just using PWM, that is to use a square wave with a very high frequency, but with a different duty cycle for the different voltages we want to generate. So we set a very very high frequency output: pwm.freq(100000) Then, if we want to produce the S0 sample, we set the duty cycle (whose value is between 0 and 65535) to a small value. If we want to produce the S1 sample, we use a higher value, and so forth. In sequence we may want to do something like that: pwm.duty_u16(3000) # S0 pwm.duty_u16(12000) # S1 pwm.duty_u16(18000) # S2 pwm.duty_u16(21000) # S3 pwm.duty_u16(24000) # S4 The duty cycle is how much time the pin is set to 1 versus how much time the pin is set to 0. A duty cycle of 65535 means 100% of time pin high. 0% means all the time low. All this, while preserving the set alternating frequency. So if we zoom like if we have an oscilloscope, we can see what happens during S2 and S3 sample generation: S2: ###################### # # # # ###################### # # # # While S3 will be like: ###################### ###################### # # # ###################### ###################### # # # The pin goes up and down with the same frequency, but in the case of S3 it stays up more. This will produce a higher average voltage. This allows us to approximate our wave. ### Convert and play a WAV file In order to play a wav file, we have to convert it into a raw format that is easy to read using MicroPython. I downloaded a wav file saying “Oh no!” from SoundCloud. So my conversion will look like this: ffmpeg -i ohno.wav -ar 24000 -acodec pcm_u8 -f u8 output.raw Note that we converted the file to 8 bit audio (256 different output levels per sample). Anyway our PWM trick is not going to approximate the different levels so well, and we are resource constrained. You can try with 16 bit as well, but I got decent results like this. Then, upload the output.raw file on the device via mpremote: mpremote cp output.raw : Now write a file called “play.py” or as you wish, with this content: from machine import Pin, PWM pwm = PWM(Pin(1)) pwm.freq(100000) f = open("output.raw","rb") buf = bytearray(4096) while f.readinto(buf) > 0: for sample in buf: pwm.duty_u16(sample x=1 x=1 x=1 x=1 x=1 f.close() What we are doing here is just getting the file, 4096 samples per iteration, then “playing” it by setting different PWM duty cycles one after the other, according to the samples values. The problem is, in our PCM file we have 24000 samples per second (see ffmpeg command line). How can be sure that it matches the MicroPython speed? well, indeed it is not a perfect match, so I added “x=1” statements to delay it a bit to kinda match the pitch that looked correct. Oh, and if you are wondering what the sample The downside of all this is that it will take your program busy while playing. I didn’t test it yet, but MicroPython supports threading, so to have a thread playing the audio could be the way to go. ### Bonus point: sin wave sound generation # Sin wave wave=[] wave_samples = 40 pwm.freq(100000) for i in range(wave_samples): x = i/wave_samples*3.14*2 dc = int((1+math.sin(x))*65000) wave.append(dc) print(wave) for i in range(1000): for dc in wave: pwm.duty_u16(dc) Comments

• First Token Cutoff LLM sampling
  Jan 12, 2024

  From a theoretical standpoint, the best reply provided by an LLM is obtained by always picking the token associated with the highest probability. This approach makes the LLM output deterministic, which is not a good property for a number of applications. For this reason, in order to balance LLMs creativity while preserving adherence to the context, different sampling algorithms have been proposed in recent years. Today one of the most used ones, more or less the default, is called top-p: it is a form of nucleus sampling where top-scoring tokens are collected up to a total probability sum of “p”, then random weighted sampling is performed. In this blog post I’ll examine why I believe nucleus sampling may not be the best approach, and will show a simple and understandable alternative in order to avoid the issues of nucleus sampling. The algorithm is yet a work in progress, but by publishing it now I hope to stimulate some discussion / hacking. ## There is some gold in the logits Despite the fact that LLM logits are one of the few completely understandable parts of the LLM inner working, I generally see very little interest in studying their features, investigating more advanced sampling methods, detecting and signaling users uncertainty and likely hallucination. Visualizing the probabilities distribution for successive tokens is a simple and practical exercise in order to gain some insights: !~! In the image we can see the top 32 candidate tokens colored by probability (white = 0, blue = 1), the selected token and the rank of the selected token (highest probability = 0, the previous one = 1, and so forth). In the above example, the Mistral base model knows the birth and death dates of Umberto Eco, so it confidently signals the most likely token with most of the total probability. Other times the model is more perplexed because either there are multiple ways to express the continuation of the text, or because it is not certain about certain facts. Asking the date of a name which birthday was not learned during training produces a different distribution of token probabilities. However in general what we want to avoid is to select suboptimal tokens, putting the LLM generation outside the path of minimum perplexity and hallucination. Nucleus sampling fails at doing this because accumulating tokens up to “p”, depending on the distribution may include tokens that are extremely weaker than the first choice. Consider, for instance, the “writer” token in the image above. Umberto Eco was a writer, indeed. The token associated with writer, while it is not scoring with a very high value, it is still a lot more above the second choice. Yet a p of 0.3 may accumulate the second token with a low value like 0.01, and yield it with some single-digit probability, with the risk of putting the generation in the wrong path. Instead, in the case of the following token, “who”, there are multiple choices with a relatively similar score, that could be used for alternative generations. ## Making use of the avalanche effect A key observation here is that to select too weak tokens for the sake of variability is not a good deal: even if we only exploit generations moments where there are a few good candidates in order to diversify the output, the avalanche effect will help us: the input context will change, thus the output of the LLM will be perturbed, with the effect of making it more likely to produce some alternative version of the text. ## First Token Cutoff (FTC) algorithm DISCLAIMER: The algorithm described here hasn’t undergone any scientific scrutiny. I experimented a couple of days with different sampling algorithms having “bound worst token” properties, and this one looks like the best balance between applicability, results and understandability. The algorithm described here can be informally stated as follows: - When the LLM is strongly biased towards a given candidate, select it. - When there are multiple viable candidates, produce alternatives. - The selection of the worst possible token should be bounded to a given amount. Past work, like Tail Free Sampling, also noted that a selection should be made across a small set of high-quality tokens emitted by the LLM. However in TFS such set is identified by performing the derivative to select a cluster corresponding to the tokens that don’t see a steep curve after which the token quality decreases strongly. In the algorithm proposed here, instead, we want the selection to follow a more bounded and understandable cut-off relative to the highest scoring token T0, attributing to T0 a special meaning compared to all the other tokens: the level of certainty the LLM has during the emission of such token (a proxy of perplexity, basically). Thus the algorithm refuses every token that is worse than a given percentage if compared to T0. The cut off percentage, that can have a value from 0 to 1, is called “co”. An example and viable “co” could be 0.5. This is how the algorithm works: 1. Compute softmax() of logits. 2. Sort tokens by probability. 3. Given T0, the probability of the best token, compute the ratio of all the other tokens as: r = 1 - (T[i] / T0) 4. Select only tokens for which r 5. Perform weighted random pick among the selected tokens. Note that in this way, regardless of the fact that tokens may have a smooth monotonically decreasing value, there is a hard limit to the tokens we can include in the set of possibilities. Instead with other methods that try to identify high-score clusters, this is not the case. ## Practical examples One reason why nucleus top-p sampling does not look to fail catastrophically in the practice, is that often times the first token probability is very high, thus when the perplexity is low, and also all the times casually we don’t collect and then pick low-quality tokens, the generation continues along a sensible path. Things are more problematic when there are successive tokens with probabilities like: 0.25, 0.14, 0.01 With a p=0.4, we could collect the third low quality token and yield it ~3% probability. Now consider First Token Cutoff with a co value of 0.5 (token can be up to 50% worse than first one): The second token r value is: r[t1] = 1-(0.14/0.25) = 0.44 # 0.44 r[t2] = 1-(0.01/0.25) = 0.96 # 0.96 > 0.5, this token is refused ## Example output Outputs of Mistral base model (no instruct) with co=0.7 for the prompt “Sorted sets are”. The outputs are three successful outputs not cherry picked for quality. 1. Sorted sets are a powerful data structure in Redis. They can be used to store sorted lists, to store unique values, to store scores for ranking, and to store a sorted list of sorted sets. 2. Sorted sets are a very powerful data structure. They allow you to store data in a way that makes it easy to find the highest or lowest values in the set, and they also allow you to sort the data. This can be useful for many different tasks, such as ranking users by their score, or finding the most popular items in a database. 3. Sorted sets are a very powerful data structure that can be used to solve many different problems. The most common use case is to store a list of unique elements, each of which has an associated value. For example, you could use a sorted set to store the names of all the people in your family, with their ages as the associated values. ## Why understandability matters Sampling parameters are among the few things that the end user of LLMs, or the API user, must tune. Trial and error is often needed, however to have a single tunable parameter for which there is an immediate real-world description and intuition helps a lot. Moreover "co" is a linear parameter, so it's particularly simple to reason about VS parameters like temperature, or even "p" of top_p that while linear strongly depends on the distribution shape of the logits. ## Future work I’m at the start of my investigations, so I’ll study and evaluate better this algorithm. More than anything else, I would love to see more interest in sampling algorithms and more interest in moving forward from top-p alike approaches. There are perhaps interesting information to collect from the logits distribution. For example it is likely that a linear probe could be able to learn when the hidden layers of an LLM are dealing with some factual information. This, with the token perplexity, could be used in order to show the user of LLMs that some part of the output is likely wrong. In general visualizing tokens probabilities distribution is very informative and in some way allows to touch with bare hands how LLMs work and what are the candidates at each step. ## Reference implementation logits = mx.softmax(logits) np_logits = np.array(logits) # MX -> NumPy np_logits = np_logits.flatten() sorted_indices = np.argsort(np_logits) sorted_indices = sorted_indices[::-1] co = 0.7 j = 1 t0 = np_logits[sorted_indices[0]] while 1 - (np_logits[sorted_indices[j]] / t0) j += 1 accepted_logits = [] for i in range(0,j): accepted_logits.append(float(np_logits[sorted_indices[j]])) accepted_logits = mx.array(accepted_logits) idx = mx.random.categorical(accepted_logits) idx = int(np.array(idx)) # Convert zero-dim array to scalar token_id = sorted_indices[idx] ## Credits These experiments were really simple to perform thanks to the MLX library from Apple, and the cool and smart developers working incessantily at it. MLX is extremely accessible, like it should be: after all LLMs are imprescrutable, but the inference itself is a simple process. Comments

• Translating blog posts with GPT-4, or: on hope and fear
  Jan 09, 2024

  My usual process for writing blog posts is more or less in two steps: 1. Think about what I want to say for weeks or months. No, I don’t spend weeks focusing on a blog post, the process is exactly reversed: I write blog posts about things that are so important to me to be in my mind for weeks. 2. Then, once enough ideas collapsed together in a decent form, I write the blog post in 30 minutes, often without caring much about the form, and I hit “publish”. This process usually works writing the titles of the sections as I initially just got the big picture of what I want to say, and then filling the empty paragraphs with text. Why I take step 2 so lightly? Because I got other stuff to do, and if blogging would take more than 30/60 minutes I would rather not blog at all, or blog less, or suffer doing it: all things I want to avoid at all costs. Blogging is too important to let it go. It’s better, for me, to give up on the form. At the same time, this is why many of my blog posts, regardless of the content that may be more or less informative, more or less useful, are generally badly written. I hope that the fact I can write well enough in my mother language in some way it is still visibile in my English posts, but I have the feeling that the extremely limited vocabulary I possess, the grammar errors, the sentence construction that oftentimes I just take from Italian and turn into English, all those limits irremediably damage the reading experience. This is why, for the first time, to write my blog post about LLMs and programming I tried a different approach: I wrote the post in Italian and I translated it to English using GPT-4. The result is a much better blog post than usually, I believe, and the total time to write it was comparable to writing it in English, because writing in Italian in a bit faster, and this compensated the time needed to cut & paste the sections in GPT-4, wait for the output, check that it matched the Italian meaning, and doing a few corrections and rewriting when needed. It shocks me that I can hear my voice when reading the translation. It does not sound written by somebody else. And, interestingly, the tools to spot GPT generated texts tell me that the post was written “100% by human”. At the same time the process may look a bit synthetic: even worse I may lose confidence writing English if I continue along this path, so I’m still not sure how this blog will be written in the future. One thing is sure: the post you are reading is not just written by myself, but as the tradition in this blog demands, not even re-read or corrected if not for a quick second pass. This way you can see what my written English really is, and if you are curious, compare it with the post about LLMs. The difference is not less than huge. Another point of view on the matter could be that my true voice is the one of the translated blog post, so writing in English is the real bluff here. Because the translated post is more representative of my lexical ability in my mother tongue, and not of the reduced one I can feature when writing in English. Maybe it captures more shades of what I really want to say. But then one could go deeper in arguments about what style really is. Is it more about sentence construction, and the way you put down your ideas, or is it a lot more about the vocabulary used, the exact words and adjectives selected to provide a given image and meaning? Probably both, and the two things are quite an inseparable whole. Anyway the simple fact that now, in 2024, I finally have this choice, fulls me of hope and fear. Hope for the possibilities the humanity will have, with machines that can talk. And fear about the potential AI has to make everybody lazy, no longer willing to do things as hard as learning a new language. Comments

• LLMs and Programming in the first days of 2024
  Jan 02, 2024

  I'll start by saying that this article is not meant to be a retrospective on LLMs. It's clear that 2023 was a special year for artificial intelligence: to reiterate that seems rather pointless. Instead, this post aims to be a testimony from an individual programmer. Since the advent of ChatGPT, and later by using LLMs that operate locally, I have made extensive use of this new technology. The goal is to accelerate my ability to write code, but that's not the only purpose. There's also the intent to not waste mental energy on aspects of programming that are not worth the effort. Countless hours spent searching for documentation on peculiar, intellectually uninteresting aspects; the efforts to learn an overly complicated API, often without good reason; writing immediately usable programs that I would discard after a few hours. These are all things I do not want to do, especially now, with Google having become a sea of spam in which to hunt for a few useful things. Meanwhile, I am certainly not a novice in programming. I am capable of writing code without any aid, and indeed, I do so quite often. Over time, I have increasingly used LLMs to write high-level code, especially in Python, and much less so in C. What strikes me about my personal experience with LLMs is that I have learned precisely when to use them and when their use would only slow me down. I have also learned that LLMs are a bit like Wikipedia and all the video courses scattered on YouTube: they help those with the will, ability, and discipline, but they are of marginal benefit to those who have fallen behind. I fear that at least initially, they will only benefit those who already have an advantage. But let's take it step by step. # Omniscient or Parrots? One of the most concerning phenomena of this new wave of novelty and progress in machine learning is the limited ability of AI experts to accept their limited knowledge. Homo sapiens invented neural networks, and then, even more crucially, an algorithm to automatically optimize the parameters of a neural network. Hardware has become capable of training increasingly larger models, and using statistical knowledge about the data to be processed (the priors) and through a lot of trial and error for successive approximations, architectures have been discovered that work better than others. But all in all, neural networks remain quite opaque. In the face of this inability to explain certain emerging capabilities of LLMs, one would have expected more caution from scientists. Instead, many have deeply underestimated LLMs, saying that after all they were nothing more than somewhat advanced Markov chains, capable, at most, of regurgitating extremely limited variations of what they had seen in the training set. Then this notion of the parrot, in the face of evidence, was almost universally retracted. At the same time, much of the enthusiastic masses attributed to LLMs supernatural powers that do not exist in reality. Unfortunately, LLMs can, at most, interpolate in the space represented by the data they have seen during training: and this would already be a lot. In reality, their ability to interpolate is limited (but still astonishing, and also unexpected). Oh, if only the largest LLMs of today could interpolate continuously in the space bounded by all the code they have seen! Even if they would not be able to produce true novelties, they would be able to replace 99% of programmers. The reality is more modest, as it almost always is. An LLM is certainly capable of writing programs that it has not seen in that exact form, showing a certain ability to blend different ideas that appeared in the training set with a certain frequency. It is also clear that this ability has, at the moment, deep limits, and whenever subtle reasoning is required, LLMs fail disastrously. Yet they represent the greatest achievement of AI, from its dawn to today. This seems undeniable. # Stupid but All-Knowing It's true: LLMs are capable, at most, of rudimentary reasoning, often inaccurate, many times peppered with hallucinations about non-existent facts. But they have a vast knowledge. In the field of programming, as well as in other fields for which quality data are available, LLMs are like stupid savants who know a lot of things. It would be terrible to do pair programming with such a partner (for me, pair programming is terrible even in the most general terms): they would have nonsensical ideas and we would have to continuously fight to impose our own. But if this erudite fool is at our disposal and answers all the questions asked of them, things change. Current LLMs will not take us beyond the paths of knowledge, but if we want to tackle a topic we do not know well, they can often lift us from our absolute ignorance to the point where we know enough to move forward on our own. In the field of programming, perhaps their ability would have been of very little interest up to twenty or thirty years ago. Back then you had to know a couple of programming languages, the classic algorithms, and those ten fundamental libraries. The rest you had to add yourself, your own intelligence, expertise, design skills. If you had these ingredients you were an expert programmer, able to do more or less everything. Over time, we have witnessed an explosion of frameworks, programming languages, libraries of all kinds. An explosion of complexity often completely unnecessary and unjustified, but the truth is that things are what they are. And in such a context, an idiot who knows everything is a precious ally. Let me give you an example: my experiments on machine learning were carried forward for at least a year using Keras. Then for various reasons, I switched to PyTorch. I already knew what an embedding or a residual network was, but I didn't feel like studying PyTorch's documentation step by step (as I had done with Keras, which I learned when ChatGPT did not yet exist). With LLMs, it was very easy to write Python code that used Torch. I just needed to have clear ideas about the model I wanted to put together and ask the right questions. # Time for Examples I'm not talking about easy things like: "Hey, what's the method of class X to do Y"? If it were just for that, one might be tempted to agree with those who are skeptical about LLMs. What the more complex models are capable of is much more elaborate. Until a few years ago, it would have been pure magic. I can tell GPT4: look, this is the neural network model I have implemented in PyTorch. These are my batches. I would like to resize the tensors so that the function that emits the batches is compatible with the input of the neural network, and I would like to represent things in this particular way. Can you show me the code needed to do the reshaping? GPT4 writes the code, and all I had to do was test in the Python CLI if the tensors really have the dimensions that are useful to me and if the data layout is correct. Here's another example. Some time ago I had to implement a BLE client for certain ESP32-based devices. After some research, I realized that multi-platform Bluetooth programming bindings are more or less all unusable. The solution was simple, write the code in Objective C using macOS's native API. So, I found myself having to deal with two problems at the same time: learning the cumbersome BLE API of Objective C, full of patterns that I consider nonsensical (I'm a minimalist, that kind of API is at the opposite end of the spectrum of what I consider "good design") and at the same time remembering how to program in Objective C. The last time I had written a program in Objective C was ten years ago: I didn't remember the details of the event loop, memory management, and much more. The final result is this code here, not exactly beautiful, but it does what it has to do. I wrote it in an extremely short time. It would have been impossible otherwise. https://github.com/antirez/freakwan/blob/main/osx-bte-cli/SerialBTE.m The code was written mostly by doing cut & paste on ChatGPT of the things I wanted to do and didn't quite know how to do, so they didn't work properly. Having the LLM explain to me what the problem was and how to solve it. It's true that the LLM didn't write much of that code, but it's also true that it significantly accelerated the writing. Would I have been able to do it without ChatGPT? Certainly yes, but the most interesting thing is not the fact that it would have taken me longer: the truth is that I wouldn't even have tried, because it wouldn't have been worth it. This fact is crucial. The ratio between the effort and the benefit of writing such a program, secondary to my project, would have been inconvenient. Moreover, this had a much more useful secondary collateral effect than the program itself: for that project I modified linenoise (one of my libraries for line editing) so that it works in multiplexing. Another example, this time less about code writing and more about data interpretation. I wanted to set up a Python script using a convolutional neural network I found online, but it was quite lacking in documentation. The network had the advantage of being in ONNX format, so I could easily extract a list of inputs and outputs, and their assigned names. I only knew one thing about this convnet: it detected certain features within an image. I didn't know the input image format and size, and especially, the network's output was far more complicated than I imagined (I thought it was a binary classifier: is the observed image okay or does it have problems? Two outputs, but there were hundreds). I began by copy-pasting the ONNX network metadata output into ChatGPT. I explain to the assistant what little I know about the network. ChatGPT hypothesizes how the inputs are organized, and that the outputs are probably normalized boxes indicating parts of the images corresponding to potential defects, and other outputs indicating the likelihood of these defects. After a few minutes of back-and-forth, I had a Python script capable of network inference, plus the necessary code to transform the starting image into the tensor suitable for input, and so on. What struck me about that session was ChatGPT finally “understood” how the network functioned once it observed the raw output values (the logits, basically) on a test image: a series of floating-point numbers provided the context to identify the exact output details, the normalization, if the boxes where centred or if the left-top corner was specified, and so forth. # Disposable Programs I could document dozens of such cases I've narrated above. It would be pointless, as it's the same story repeating itself in more or less the same way. I have a problem, I need to quickly know something that *I can verify* if the LLM is feeding me nonsense. Well, in such cases, I use the LLM to speed up my need for knowledge. However, there are different cases where I let the LLM write all the code. For example, whenever I need to write a more or less disposable program. Like this one: https://github.com/antirez/simple-language-model/blob/main/plot.py I needed to visualize the loss curve during the learning of a small neural network. I showed GPT4 the format of the CSV file produced by the PyTorch program during learning, and then I requested that if I specified multiple CSV files on the command line, I didn’t want the training and validation loss curves of the same experiment anymore, but a comparison of the validation loss curves of different experiments. The above is the result, as generated by GPT4. Thirty seconds in total. Similarly, I needed a program that read the AirBnB CSV report and grouped my apartments by month and year. Then, considering the cleaning costs, and the number of nights per booking, it would do statistics on the average rental price for different months of the year. This program is extremely useful for me. At the same time, writing it is deadly boring: there's nothing interesting. So I took a nice piece of the CSV file and did copy-paste on GPT4. I wrote to the LLM what the problem was to be solved: the program worked on the first try. I show it to you in full below. ```python import pandas as pd pd.set_option('display.max_rows', None) df = pd.read_csv('listings.csv') reservations = df[df['Type'] == 'Reservation'] reservations['Start Date'] = pd.to_datetime(reservations['Start Date']) reservations['Year'] = reservations['Start Date'].dt.year reservations['Month'] = reservations['Start Date'].dt.month reservations['Nightly Rate'] = (reservations['Amount'] - reservations['Cleaning Fee']) / reservations['Nights'] all_listings = reservations['Listing'].unique() all_years = reservations['Year'].unique() all_months = range(1, 13) index = pd.MultiIndex.from_product([all_listings, all_years, all_months], names=['Listing', 'Year', 'Month']) all_data = pd.DataFrame(index=index).reset_index() merged_data = pd.merge(all_data, reservations, on=['Listing', 'Year', 'Month'], how='left') average_nightly_rates = merged_data.groupby(['Listing', 'Year', 'Month'])['Nightly Rate'].mean().fillna(0) print(average_nightly_rates) ``` To understand exactly how to group the data, which were otherwise quite scattered and disordered, requires some simple reasoning: it’s not a problem that an LLM could solve simply by regurgitating something that approximates the language it has seen. GPT4 has certainly observed similar programs during its training, but almost certainly has not seen a program that had to group things in the exact way I asked for and in a CSV file with that specific format. This shows (in my opinion) that LLMs can interpolate to some extent within the space outlined by the different programs in the training set. Writing this type of program would be, in my opinion, an imprudent use of my time. The fact that an LLM can do it for me certainly speeds up my ability to produce code in a rather direct way: I can focus on the things that matter. # When things don't go as they should: System Programming Despite my success with LLMs and programming, when I write a program in C, I almost always end up using LLMs only as a more convenient form of documentation. I am an expert in system programming, and in that context, LLMs would be useful to me only if they were capable of complex reasoning: in this scenario, they almost always fail. Let's start with the prompt of this experiment: Outline the requirements of a nice, small but working C implementation of a bloom filter. Focus on the important aspects of the hashing function. Then write it, in good quality C. Also consider that your example implementation should be sized to store 100,000 elements, with a false positive probability of at most 5%. The elements added are null-terminated strings. Here’s the output: ```c #include #include #include #include #define N 100000 // Number of elements #define P 0.05 // False positive probability #define M (int)(-(N * log(P)) / (log(2) * log(2))) // Size of bit array #define K (int)(M / N * log(2)) // Number of hash functions unsigned char bitArray[M / 8 + 1]; void setBit(int n) { bitArray[n / 8] |= 1 } int getBit(int n) { return bitArray[n / 8] & (1 } // Basic hash functions (for demonstration purposes) int hash1(char *str) { int hash = 5381; int c; while ((c = *str++)) hash = ((hash return abs(hash % M); } int hash2(char *str) { unsigned long hash = 0; int c; while ((c = *str++)) hash = c + (hash return abs(hash % M); } void add(char *str) { setBit(hash1(str)); setBit(hash2(str)); // Add more hash functions if needed } int is_in(char *str) { return getBit(hash1(str)) && getBit(hash2(str)); // Check more hash functions if used } int main() { char *testStr = "testString"; add(testStr); if (is_in(testStr)) { printf("%s is probably in the set.\n", testStr); } else { printf("%s is definitely not in the set.\n", testStr); } return 0; } ``` The implementation produced by GPT4 is not exactly brilliant. Bloom filters are quite widespread. It’s not such an exotic data structure. However, it’s clear that writing a decent bloom filter requires a greater capacity for abstraction: for example, there’s the problem of finding an effective way to hash the same string N times, and being certain that the hashes are sufficiently decorrelated. Yet if you explicitly ask GPT4 to modify the hashing function so that it produces N decorrelated outputs, its proposed solution is significantly more acceptable: ```c unsigned int hash(const char* str, unsigned int hash_id) { unsigned int hash = 5381; int c; while ((c = *str++)) { hash = ((hash } return hash % BF_SIZE; } ``` If it had come up with this idea on its own, it would have written the bloom filter differently, setting K bits at a time using a single hashing function. GPT4 was able to perform, in isolation, the task of writing an appropriate and more generic hashing function. But in the larger project of writing a bloom filter, it failed to show sufficient reasoning ability, and instead spat out two different but similar hashing functions: this is what one would expect from a poorly written article example on bloom filters. The current weak reasoning abilities of Large Language Models, combined with a possible scarcity of sources on the subject, and even worse, potentially many low-quality sources, collectively result in this meager outcome. Yet, this is not an isolated case. The few times I attempt to use LLMs in the realm of algorithmic or system programming, the results are extremely poor, even when the expected reasoning abilities are not greater than what LLMs seem to possess in contexts like Python programming. At the same time, GPT-4 is capable of decompiling the function it produced (in a separate session), and also precisely understand its purpose: https://chat.openai.com/share/f54b64d7-d19a-4be4-8f10-08553aa812f1 Therefore, stating that there are no uses in system programming contexts is certainly inaccurate. One interesting and pretty much expected thing is that, in these cases, the difference between smaller and larger models is indeed significant. Let’s see how two local models fare. For instance, Mixtral, the MoE of Mistral, responded to the same question as follows: ```c unsigned int hash(const char* str, unsigned int hash_id) { unsigned int hash = 5381; int c; while ((c = *str++)) { hash = ((hash } return hash + hash_id; } ``` Clearly, adding `hash_id` only at the end is an extremely poor solution. Mixtral is an excellent model for many uses, but as for the already weak reasoning capabilities of LLMs, the rule so far is certainly: larger is better. This is a chat with the local LLM deepseek-coder, quantized to 4 bits as I don’t have enough RAM to use the model at higher precision. Yet, being 34 billion parameters large, its reasoning capabilities on this matter seem quite advanced. Here’s the log of the chat (conducted with llama.cpp): > I have a hash function that should output N distinct hashes for the same data. However, I'm seeing a very poor distribution of the produced hashes as I change the value of hash_id: unsigned int hash(const char* str, unsigned int hash_id) { unsigned int hash = 5381; int c; while ((c = *str++)) { hash = ((hash } return hash + hash_id; } Maybe the salting strategy I'm using is poor? How could I change it? [Deep-seek coder reply] A possible issue with your implementation is that you're simply adding `hash_id` to the end of the hash. This can potentially lead to a poor distribution, especially if `hash_id` is too large or is not evenly distributed. Consider using bitwise operations for mixing the `hash_id` in. Here is an example with XOR: ```c unsigned int hash(const char* str, unsigned int hash_id) { unsigned int hash = 5381; int c; while ((c = *str++)) { hash = ((hash } return (hash ^ hash_id); // Mixing the hash_id one more time at the end } ``` This will spread `hash_id` throughout your hashes better than simply adding it, which can lead to less uniform distribution. [Snipped… the actual reply contained more informative stuff] Not bad for a model running on my MacBook M1 Max. It also mixed the sums with XORs. In this case, the model was certainly helped by the fact that I provided clues about the problem to solve, but it was the model that identified what the real source of the trouble was, and the proposed alternative is valid. Something like the above is not within the reach of any book, documentation, or Google search. And as primitive, a result of interpolation, or however you want to perceive it, the model has performed some form of reasoning, if by reasoning, in this specific case, we accept the identification of the origin of a problem and its potential solution. However one wants to think about LLMs, stating that they are not helpful for programmers is extremely rash. At the same time, however, my experience over the past few months suggests that for system programming, LLMs almost never provide acceptable solutions if you are already an experienced programmer. Let me show you another real world example. My current project, ggufflib, involves writing a library that reads and writes GGUF format files, which is the format in which llama.cpp loads quantized models. Initially, to understand how the quantization encodings worked (for speed reasons the bits of each quant are stored in fancy ways), I tried using ChatGPT, but then I resolved to reverse engineer llama.cpp's code: it was much faster. An LLM that can decently assist a system programmer, if it sees the data encoding “struct” declaration and the decoding function, should be able to reconstruct the data format documentation. The functions of llama.cpp were small enough to fit entirely in the context of GPT4, yet the output was completely useless. In these cases, things are done as in the past: paper and pen, reading the code, and seeing where the bits that the decoder extracts are registered. Let me explain better the above use case so that you can try it yourself, if you wish. We have this structure from llama.cpp implementation. // 6-bit quantization // weight is represented as x = a * q // 16 blocks of 16 elements each // Effectively 6.5625 bits per weight typedef struct { uint8_t ql[QK_K/2]; // quants, lower 4 bits uint8_t qh[QK_K/4]; // quants, upper 2 bits int8_t scales[QK_K/16]; // scales, quantized with 8 bits ggml_fp16_t d; // super-block scale } block_q6_K; Then there is this function that is used to perform the dequantization: void dequantize_row_q6_K(const block_q6_K * restrict x, float * restrict y, int k) { assert(k % QK_K == 0); const int nb = k / QK_K; for (int i = 0; i const float d = GGML_FP16_TO_FP32(x[i].d); const uint8_t * restrict ql = x[i].ql; const uint8_t * restrict qh = x[i].qh; const int8_t * restrict sc = x[i].scales; for (int n = 0; n for (int l = 0; l int is = l/16; const int8_t q1 = (int8_t)((ql[l + 0] & 0xF) | (((qh[l] >> 0) & 3) const int8_t q2 = (int8_t)((ql[l + 32] & 0xF) | (((qh[l] >> 2) & 3) const int8_t q3 = (int8_t)((ql[l + 0] >> 4) | (((qh[l] >> 4) & 3) const int8_t q4 = (int8_t)((ql[l + 32] >> 4) | (((qh[l] >> 6) & 3) y[l + 0] = d * sc[is + 0] * q1; y[l + 32] = d * sc[is + 2] * q2; y[l + 64] = d * sc[is + 4] * q3; y[l + 96] = d * sc[is + 6] * q4; } y += 128; ql += 64; qh += 32; sc += 8; } } } If I ask GPT4 to write an outline of the format used, it struggles to provide a clear explanation of how the blocks are stored on the lower / upper 4 bits of “ql” depending on the weight position. For this blog post, I also tried asking it to write a simpler function that shows how data is stored (maybe it can’t explain it with words, but can with code). The produced function is broken in many ways, the indexes are wrong, the 6-bit -> 8-bit sign extension is wrong (it just casts to uint8_t), and so forth. Btw, this is the code that I ended writing myself: } else if (tensor->type == GGUF_TYPE_Q6_K) { uint8_t *block = (uint8_t*)tensor->weights_data; uint64_t i = 0; // i-th weight to dequantize. while(i num_weights) { float super_scale = from_half(*((uint16_t*)(block+128+64+16))); uint8_t *L = block; uint8_t *H = block+128; int8_t *scales = (int8_t*)block+128+64; for (int cluster = 0; cluster for (uint64_t j = 0; j f[i] = (super_scale * scales[j/16]) * ((int8_t) ((((L[j%64] >> (j/64*4)) & 0xF) | (((H[j%32] >> (j/32*2)) & 3) i++; if (i == tensor->num_weights) return f; } L += 64; H += 32; scales += 8; } block += 128+64+16+2; // Go to the next block. } } From the function above, I removed what was the actual contribution of this code: the long comments documenting the exact format used by llama.cpp Q6_K encoding. Now, it would be immensely useful if GPT could do this for me, and I bet it’s just a matter of months, because these kind of tasks are within what can be reached without any breakthrough, just with a bit of scaling. # Putting Things in Perspective I regret to say it, but it's true: most of today's programming consists of regurgitating the same things in slightly different forms. High levels of reasoning are not required. LLMs are quite good at doing this, although they remain strongly limited by the maximum size of their context. This should really make programmers think. Is it worth writing programs of this kind? Sure, you get paid, and quite handsomely, but if an LLM can do part of it, maybe it's not the best place to be in five or ten years. And then, do LLMs have some reasoning abilities, or is it all a bluff? Perhaps at times, they seem to reason only because, as semioticians would say, the "signifier" gives the impression of a meaning that actually does not exist. Those who have worked enough with LLMs, while accepting their limits, know for sure that it cannot be so: their ability to blend what they have seen before goes well beyond randomly regurgitating words. As much as their training was mostly carried out during pre-training, in predicting the next token, this goal forces the model to create some form of abstract model. This model is weak, patchy, and imperfect, but it must exist if we observe what we observe. If our mathematical certainties are doubtful and the greatest experts are often on opposing positions, believing what one sees with their own eyes seems a wise approach. Finally, what sense does it make today not to use LLMs for programming? Asking LLMs the right questions is a fundamental skill. The less it is practiced, the less one will be able to improve their work thanks to AI. And then, developing a descriptive ability of problems is also useful when talking to other human beings. LLMs are not the only ones who sometimes don't understand what we want to say. Communicating poorly is a great limitation, and many programmers communicate very poorly despite being very capable in their specific field. And now Google is unusable: using LLMs even just as a compressed form of documentation is a good idea. For my part, I will continue to make extensive use of them. I have never loved learning the details of an obscure communication protocol or the convoluted methods of a library written by someone who wants to show how good they are. It seems like "junk knowledge" to me. LLMs save me from all this more and more every day. Comments

• The origins of the Idle Scan
  Oct 19, 2023

  The Idle scan was conceived at the end of 1998, evidenced by emails. I had moved to Milan a few months prior, having been there since September if I recall correctly, brimming with new ideas, unaware that my stay in that city would be brief. I spent the summer on the beaches of Sicily, mainly occupied with reading many books recommended by the folks at Seclab (mostly by David). However, those readings needed a catalyst: the Idle scan was an attack born from theoretical rumination, but the stream of thoughts originated from a rather practical circumstance. I had recently created Hping, a tool whose logo was borrowed from that of Nutella. I mention this to emphasize the seriousness that governed my efforts at that time — after all, I was only twenty-one and already in Northern Italy with a full-time job on my shoulders; some understanding was warranted. Hping was a Swiss Army knife for the TCP/IP protocol. Its initial use was mostly exploratory, for research. With Hping, you could assemble TCP, UDP, and ICMP packets in the most bizarre manner, and encapsulate them in equally eccentric IP packets, fragmented, with fields set to anomalous values. These packets were sent around to observe the network stack response of different operating systems. This is where Idle scan originates: playing with Hping for just a few minutes revealed a well-known yet (to me) surprising fact. The response packets had an ID field that continuously incremented by some measure. At that time, given that the attacks I would later disclose were not yet known, this ID field behavior aroused no concern. Every time an operating system emitted an IP packet, it first incremented a counter (which reset to zero once it reached the maximum value of two to the sixteenth power minus one), then the packet was sent with the ID set to the counter's value. The counter was universal for all outgoing packets. This allowed, for starters, to estimate the outgoing traffic of any networked computer. This information leak struck me; I saw it as a concerning anomaly. I wrote an initial post on BUGTRAQ, highlighting the issue. Among the responses was one from an Internet luminary, someone who had drafted central RFCs for the TCP/IP protocol. He said that, yes, they were aware; it was a well-known fact. In short: although this characteristic could be used for traffic estimation, most saw no risk. And resolving that issue, deemed negligible, would require a significant overhaul of operating systems. Not worth it. But I was losing sleep over it, hardly a secondary issue to me. I was convinced that the information provided by the incrementing ID field could be combined with other elements (which I had not fully grasped yet) to mount a far more serious attack. I discussed it with Lorenzo Cavallaro. Lorenzo had introduced me to raw socket techniques months earlier, which I had used to write Hping. He had become my go-to conversational partner, as well as a dear friend; if I had to ponder on TCP/IP-related matters, I would discuss them with him. When I told him about these new ideas, I had to be vague, not by choice but because I hadn't yet pieced together the final attack. Despite my vagueness, he appeared fairly interested. A couple of days later, I spoke to him again. Finally, I managed to give him a more comprehensive description of the Idle scan (not yet named as such). I had reasoned it out on paper, as there was no way to test it practically. If I remember correctly, Hping was missing some of the necessary functions, although I wouldn't bet on this part: it's been twenty-five years. My memory clears as I write. I believe you couldn't choose the exact flags of the outgoing TCP packet. Anyway, from what I remember, the modifications to Hping took little time, so we were able to test my theory in practice as well. The Idle scan really worked, and proving it in real conditions was very exciting. I don't remember who made this parallel, but essentially, it was like observing the movement of an invisible celestial body indirectly, measuring only its gravitational effects on another, more easily observable body, much like the discovery of Pluto. All that was left was to make the attack public. Unfortunately, the thought of writing a lengthy email in English terrified me. I was aware that the Idle scan was an interesting idea, and precisely because of that, I was afraid of botching its announcement due to poor presentation. After all, the attack was cumbersome; it could even end up being described so poorly as to make it incomprehensible. Lorenzo and I wrote that email late at night, at my place, certainly drunk. Better than writing it alone; Lorenzo's English, even if slightly, was better than mine. It happened that a colleague from Seclab (also one of the founders), whom I consider a true genius in information security, came from a family of restaurateurs from Emilia. One evening he shows up at my place with a bag. Inside the bag is a fine piece of grey mullet roe, which I had never seen, let alone tasted (it would become one of my favorite foods). The cook-hacker prepares pasta for all of us, and together we eat and drink sufficiently. At the end of the evening, the others leave (there was someone else, besides the cook-hacker), but Lorenzo stays a bit longer. That evening, Lorenzo and I, fairly tipsy, recklessly thought the time had come to write the Idle scan announcement email (still preserved in the BUGTRAQ archives, and which today appears genuinely delirious [1]). We sit down at my PC to work. The email opens like this: I have uncovered a new tcp port scan method. Instead all others it allows you to scan using spoofed packets, so scanned hosts can't see your real address. Not just the grammar, even the peremptory tone of the email is utterly inappropriate. To top it off, we propose calling the attack dumb host scan, but fortunately, someone quickly finds a better name, which became the official one: Idle scan. Long story short, despite its flawed origins, the Idle scan has become a classic attack, and it's safe to say, as is often the case, that the rest is history. [1] https://seclists.org/bugtraq/1998/Dec/79 Comments

• In defense of linked lists
  Nov 04, 2022

  A few days ago, on Twitter (oh, dear Twitter: whatever happens I’ll be there as long as possible – if you care about people that put a lot of energy in creating it, think twice before leaving the platform). So, on Twitter, I was talking about a very bad implementation of linked lists written in Rust. From the tone of certain replies, I got the feeling that many people think linked lists are like a joke. A trivial data structure that is only good for coding interviews, otherwise totally useless. In a word: the bubble sort of data structures. I disagree, so I thought of writing this blog post full of all the things I love about linked lists. So, get ready to read a sentimental post about a data structure, and don't tell I didn't warn you. Linked lists are educational. When your teacher, or the page of a book, or anything that exposes you for the first time to linked lists shows you this little circle with an arrow pointing to another circle, something immense happens in your mind. Similar to what happens when you understand recursion for the first time. You get what data structures made of links truly are: the triviality of a single node that becomes a lot more powerful and complex once it references another one. Linked lists show the new programmer fundamental things about space and time in computation: how it is possible to add elements in a constant time, and how order is fundamentally costly, because if you want to insert an element “in place” you have to go from one node to the other. You immediately start thinking of ways to speed up the process (preparing you for the next things), and at the same time you understand, deeply, what O(1) and O(N) really mean. Linked lists are augmentable. Add a pointer to the previous element, and now it is possible to go both sides. Add “far” pointers from time to time, and you have a skip list with completely different properties. Change every node to hold multiple items and your linked list becomes unrolled, providing very different cache obviousness properties. Linked lists can be embedded, too. The Linux kernel, for instance, has macros to add a field to any structures in order to link them together. There is more: linked lists are composable. This is a bold property: you can split a linked list into two in O(1), and you can glue two linked lists in O(1) as well. If you make judicious use of this property, interesting things are possible. For instance, in Redis modules implementing threaded operations, the thread processing the slow request dealt with a fake client structure (this way there was no locking, no contention). When the threaded command finally ended its execution, the output buffer of the client could be glued together to the actual buffer of the real client. This was easy because the output buffer was represented with a linked list. Linked lists are useful: Redis can be wrong, but both Redis and the Linux kernel can’t. They are useful because they resemble certain natural processes: adding things in the order they arrive, or in the reverse order, is natural even in the physical world. Pulling items incrementally is useful too, as it is moving such items from head to tail, or moving them a position after the current one. Linked lists are simple. It is one of those rare data structures, together with binary trees and hash tables and a few more, that you can implement just from memory without likely stepping into big errors. Linked lists are conceptual. A node pointing to itself is the most self centered thing I can imagine in computing: an ideal representation of the more vulgar infinite loop. A node pointing to NULL is a metaphor of loneliness. A linked list with tail and head connected, a powerful symbol of a closed cycle. For all those reasons, I love linked lists, and I hope that you will, at least, start smiling at them. Comments

• Scrivendo Wohpe
  Jul 17, 2022

  (English translation of this post: http://antirez.com/news/136) Dopo due anni di lavoro, finalmente, Wohpe, il mio primo libro di fantascienza, ma anche il mio primo scritto di prosa di questa lunghezza, è uscito nelle librerie fisiche italiane, su Amazon, e negli altri store digitali. Lo trovate qui: https://www.amazon.it/Wohpe-Salvatore-Sanfilippo/dp/B09XT6J3WX Dicevo: il primo scritto di questa lunghezza. Ma posso considerarmi del tutto nuovo alla scrittura? Ho scritto per vent’anni in questo blog e in quelli passati che ho tenuto nel corso del tempo, e molto spesso ho usato Facebook per scrivere brevi racconti, frutto di fantasie o basati su fatti reali. Oltre a ciò, ho scritto di cose tecniche, specialmente riguardo la programmazione, per un tempo altrettanto lungo, e sono stato un lettore di racconti e di romanzi per tutto il corso della mia vita. E allora perché scrivere Wohpe è stato anche imparare a scrivere da zero? Nei primi mesi di scrittura del romanzo, ma anche prima, nei mesi precedenti, quando mi preparavo scrivendo lunghi racconti che poi avrei cestinato, mi è successo ciò che accade spesso a coloro che imparano a giocare a scacchi. Tanti seguono questo percorso: imparano le regole, e va bene, lo sappiamo che con quelle si fa poco; le regole permettono solo di muovere i pezzi in maniera legale. Ma poi, subito dopo, imparano dei rudimenti di tecnica e di strategia, magari studiando duramente per qualche settimana. Però quando sono alla scacchiera, se una mossa non è brutalmente peggiore o migliore di un’altra, tutte le mosse sembrano equivalenti. Il giocatore di scacchi poco abile, poco esperto, non ha un vero gusto per le mosse; non riesce a valutarle non solo per ciò che sono in termini assoluti, ma neanche secondo una sua propria idea. Il risultato è un gioco casuale. Solo più avanti, dopo molte ore di gioco, ella finalmente riuscirà a esprimere delle scelte che, a prescindere dal fatto siano esse giuste o sbagliate, hanno quantomeno una coerenza, sono state davvero pensate: voglio muovere il cavallo qui, per queste precise ragioni, e per tali ragioni preferisco questa mossa a tutte le altre possibili. Così chi scrive ed è agli inizi, se una frase è buona, con certezza non lo sa (e per continuare col paragone di sopra, così come il gioco dello scacchista sarà casuale, la sua scrittura sarà casuale). Sposta una virgola, cambia una parola. Suona bene o male? Ha delle idee che si è formato scrivendo a scuola e poi scrivendo da adulto, ma queste idee lo assistono poco quando l’ambizione è quella di scrivere una prosa di livello letterario. L’autore alle prime armi non ha un suo stile, perché prima di non saper scrivere non sa ancora leggere: quando legge un libro che adora, raramente capisce esattamente *cosa* accade nella pagina di così convincente, e così anche quando rilegge se stesso, non sa se ha scritto bene o male. Leggi, se vuoi imparare a scrivere! Dicono tutti. Peccato non sia vero: bisogna prima di tutto scrivere per imparare a scrivere, così come bisogna fare dei cortometraggi per imparare a fare il regista, e guardare gli altri film non sarà sufficiente (anche se sarà certamente utile). E per imparare a scrivere, il tipo di lettura che serve davvero è la rilettura di alcuni libri che abbiamo scelti come modelli; quello sì che è utile: la rilettura per comprenderne le forme, fino in fondo. Ora questo semplice fatto, di aver capito quale sia il mio stile, e di saper finalmente leggere e avere un giudizio che emerge immediatamente quando ho tra le mani un’opera, è già un risarcimento più che sufficiente dei due anni di sforzi di scrittura nei quali mi sono profuso. Ed è una vera fortuna che l’esperienza in sé sia stata di così grande valore, perché quello che molti nuovi autori non sanno è quanto violento possa essere il mercato editoriale mondiale, e quello italiano in particolare. In Italia un libro di fantascienza che ha un buon successo, edito da un piccolo o medio editore, vende 500 copie. Deve andare bene, per avere questi numeri. La gran parte dei libri vende meno di 100 copie. A noi informatici queste cifre fanno rabbrividire. Il più stupido programma che ho scritto ha avuto dieci volte più utenti. Mi spingerei a dire che il più stupido programma che ho scritto e che ho pubblicato con un minimo di energie, ha avuto dieci volte questi *lettori*, gente che ne hanno letto il codice sorgente per capire come funzionasse. Io sono stato, e per questo ringrazio non so bene chi, un programmatore di una certa notorietà, direte voi. Ciò che voglio dire va molto oltre la mia esperienza personale. Anche coloro che non sono conosciuti da nessuno e provano a fare, nella programmazione, una cosa mediamente interessante, appena ben descritta e documentata, e la mostrano un po’ in giro, ricevono un interesse enorme rispetto a quello che spetta agli autori di fiction. I motivi sono tanti e abbastanza ovvi, non vale neppure la pena di soffermarsi su di essi, dunque perché vi racconto queste cose? Per questo motivo: Sono poche le attività, oltre alla letteratura, dove c’è lo stesso mostruoso scompenso tra le forze necessarie per produrre un’opera e la scarsa risposta del pubblico. Chi decide di dedicare molto tempo alla scrittura, deve conoscere questo fatto da subito. Io per fortuna sapevo già tutto, grazie ai miei amici scrittori; però lo stesso certe sfumature di questa irrilevanza finiscono per essere difficili da accettare. E allora perché tutti scrivono? Sono straripanti le file di quelli che tentano la fortuna editoriale. Credo sia un meccanismo simile a quello che accade nell’IT, con tanti che provano a creare un nuovo linguaggio di programmazione: il fallimento è quasi certo, ma il tentativo stesso è una delle imprese più soddisfacenti alle quali dedicare le proprie migliori energie. Ora sono a un bivio: potrei scrivere altra prosa, rimettermi a scrivere codice, o provare a tenere vive le due attività allo stesso tempo. Cosa farò non lo so ancora. Per ora, vediamo cosa succede con Wohpe, sia con la versione italiana che con la traduzione in inglese, a cui in questo momento una capace traduttrice sta lavorando. E su questa cosa delle traduzioni, eseguite col supporto dell’autore, e di quale esperienza significativa sia dal punto di vista filologico, magari vi parlerò qualche altra volta (io e Bridget parliamo l’inglese e l’italiano, ma siamo madrelingua lei di una e io dell’altra lingua, e ciò è molto interessante quando si collabora tra traduttore e autore). Chiudo il post dicendo a chi mi legge: se ne avete voglia, scrivete prosa! Io ora lo so per certo: non è un caso che la scrittura sia stata per centinaia di anni considerata l’arte più alta nella quale cimentarsi. Scrivendo si cercano delle cose, e se si insiste abbastanza si finisce per trovarle davvero. Comments

• Writing Wohpe
  Jul 17, 2022

  (Traduzione italiana di questo post: http://antirez.com/news/137) [Sorry for the form of this post. For the first time I wrote a post in two languages: Italian and English. So I went for the unusual path of writing it in Italian to start, translating it with Google Translate, and later I just scanned it to fix the biggest issues. At this point GT is so good you can get away with this process.] After two years of work, finally, Wohpe, my first science fiction book, but also my first prose writing of this length, has been released in Italian physical bookstores, on Amazon, and in other digital stores. You can find it here: https://www.amazon.it/Wohpe-Salvatore-Sanfilippo/dp/B09XT6J3WX I was saying: the first writing of this length. But can I consider myself entirely new to writing? I have written for twenty years in this blog and in the past ones that I have kept over time, and very often I have used Facebook to write short stories, the result of fantasies or based on real facts. On top of that, I've been writing about technical stuff, especially programming, for an equally long time, and I've been a short story and novel reader my entire life. So why was writing Wohpe also learning how to write from scratch? In the first months of writing the novel, but also earlier, in the previous months, when I was preparing myself by writing long stories that I would then throw away, what happened to me often happens to those who learn to play chess. Many follow this path: they learn the rules, and that's okay, we know that you don't do much with them; the rules only allow you to move pieces legally. But then, soon after, they learn some rudiments of tactics and strategy, perhaps studying hard for a few weeks. But when they are on the board, if one move is not brutally worse or better than another, all the moves seem equivalent. The unskilled, inexperienced chess player has no real taste for moves; she fails to evaluate them not only for what they are in absolute terms, but not even according to his own idea. The result is a casual game. Only later, after many hours of play, will she finally be able to express choices that, regardless of whether they are right or wrong, have at least a coherence, they were really thought: I want to move the horse here, for these precise reasons, and for these reasons I prefer this move to all other possible ones. So whoever writes and is at the beginning, if a sentence is good, he certainly does not know it (and to continue with the above comparison, just as the chess player's game will be casual, his writing will be casual). Move a comma, change a word. Does it sound good or bad? He has ideas that he formed by writing at school and then writing as an adult, but these ideas help him little when the ambition is to write a literary level prose. The novice author does not have his own style, because before not knowing how to write he does not yet know how to read: when he reads a book he adores, he rarely understands exactly * what * happens on the page that is so convincing, and so also when he re-reads himself, he does not know if he wrote well or badly. Read, if you want to learn to write! They all say. Too bad it's not true: first of all you have to write, to learn how to write, just as you have to make short films to learn to be a director, and watching other films will not be enough (although it will certainly be useful). And to learn to write, the kind of reading that is really needed is the rereading of some books that we have chosen as models; what is useful: the rereading to fully understand its forms. Now this simple fact, of having understood what my style is, and of finally knowing how to read and have a judgment that immediately emerges when I have a work in my hands, is already more than enough compensation for the two years of writing efforts in which I have lavished. And it is fortunate that the experience itself was of such great value, because what many new authors do not know is how violent the world publishing market can be, and the Italian one in particular. In Italy a science fiction book that has a good success, published by a small or medium publisher, sells 500 copies. It has to be fine to have these numbers. Most books sell less than 100 copies. To us computer scientists these figures make us shiver. The stupidest program I wrote had ten times as many users. I would go as far as to say that the stupidest program I've written and published with a minimum of energy has had these * readers * ten times, people who have read its source code to understand how it works. I have been, and for this I thank I don't know who, a programmer of a certain notoriety, you might say. What I mean is far beyond my personal experience. Even those who are not known by anyone and try to do something on average interesting, just well described and documented, in programming, and show it around a bit, receive enormous interest compared to what belongs to the authors of fiction. The reasons are many and quite obvious, it is not even worth dwelling on them, so why am I telling you these things? For this reason: There are few activities, besides the literature, where there is the same monstrous imbalance between the forces needed to produce a work and the poor response of the public. Anyone who decides to devote a lot of time to writing must know this fact right away. Fortunately, I already knew everything, thanks to my writing friends; however, certain nuances of this irrelevance end up being difficult to accept. So why is everyone writing? The ranks of those who try their editorial fortune are overflowing. I think it's a similar mechanism to what happens in IT, with many trying to create a new programming language: failure is almost certain, but doing so is one of the most satisfying companies to devote your best energy to. Now I'm at a crossroads: I could write more prose, get back to coding, or try to keep the two activities alive at the same time. What I'll do I don't know yet. For now, let's see what happens with Wohpe, both with the Italian version and with the English translation, which a capable translator is working on right now. And on this matter of the translations, carried out with the support of the author, and what a significant experience it is from the philological point of view, maybe I will talk to you some other time (Bridget and I speak English and Italian, but we are native speakers of one and I of the other language, and this is very interesting when collaborating between translator and author). I close the post by saying to those who read me: if you feel like it, write prose! I now know for sure: it is no coincidence that for hundreds of years writing has been considered the highest art in which to try one's hand. By writing you look for things, and if you insist enough you end up really finding them. Comments

• Programming and Writing
  May 14, 2021

  One year ago I paused my programming life and started writing a novel, with the illusion that my new activity was deeply different than the previous one. A river of words later, written but more often rewritten, I’m pretty sure of the contrary: programming big systems and writing novels have many common traits and similar processes. The most obvious parallel between the two activities is that in both of them you write something. Code is not prose written in a natural language, yet it has a set of fixed rules (a grammar), certain forms that most programmers will understand as natural and others that, while formally correct, will sound hard to grasp. There is, however, a much deeper connection between the two activities: a good program and a good novel are both the sum of local and global elements that work well. Good code must be composed of well written and readable single statements, but overall the different parts of the program must be orthogonal, designed in a coherent way, and have clean interactions. A good novel must also succeed in the same two scales of the micro and the macro. Sentences must be well written, but the overall structure and relationship between the parts is also crucial. A less structural link between programming and writing is in the drive you need when approaching one or the other: to succeed you need to make progresses, and to make progresses you have to be consistent. There is extensive agreement on the fact that programs and novels don’t write themselves, yet. Twenty years of writing code helped me immensely with this aspect; I knew that things happen only if you sit every day and write: one day one hundred words, the other day two thousands, but rare is the day I don’t put words on the page. And if you have written code that is not just a “filler” for a bigger system, but a creation of your own, you know that writer block also happens in programming. The only difference is that for most people you are an engineer, hence, if you don’t work, you are lazy. The same laziness, in the case of an artist, will assume the shape of a fascinating part of the creative process. The differences. I believe the most sharp difference between writing and programming is that, once written, edited and finalized, a novel remains immutable, mostly. There are several cases of writers returning on their novels after several years, publishing a bug fixed version of it, but this is rare and, even when happens, a one-shot process. Code evolves over time, is targeted by an endless stream of changes, often performed by multiple people. This simple fact has profound effects on the two processes: programmers often believe that the first version of a system can be quite imperfect; after all there will be time to make improvements. On the other hand writers know they have a single bullet for every novel, to the point that writing prose is mostly the act of rewriting. Rewriting sentences, whole chapters, dialogues that sound fake, sometimes two, three, or even ten times. I believe programming, in this regard, can learn something from writing: when writing the first core of a new system, when the original creator is still alone, isolated, able to do anything, she should pretend that this first core is her only bullet. During the genesis of the system she should rewrite this primitive kernel again and again, in order to find the best possible design. My hypothesis is that this initial design will greatly inform what will happen later: growing organically something that has a good initial structure will result in a better system, even after years of distance from the original creation, and even if the original core was just a tiny faction of the future mass the system would eventually assume. In case you are interested, a quick update about my sci-fi novel. After many self-reviews I sent the manuscript to my editor, Giulio Mozzi. He will send me the change proposals in a few weeks. I’ll start a new review process informed by his notes, and hopefully finalize the novel in one or two months. Then, finally, I’ll be ready to publish the Italian version. A the same time the finalized novel will be sent to my translator, in the US, and when she ends the translation the English version will be published as well. It’s a long journey, but one that I deeply enjoyed taking. Comments

Dave Cheney

• Microblog: TestMain can cause one to question reality
  Feb 21, 2024

  This morning a one line change had several of us tearing up the fabric of reality trying to understand why a failing test wasn’t failing, or, in fact, being run at all. Increasingly frantic efforts to upgrade/downgrade Go, run the tests on another machine, run the tests in CI, all served to only unnerve us […]

• A few bytes here, a few there, pretty soon you’re talking real memory
  Jan 05, 2021

  Today’s post comes from a recent Go pop quiz. Consider this benchmark fragment. A convenience wrapper around sort.Sort(sort.StringSlice(s)), sort.Strings sorts the input in place, so it isn’t expected to allocate (or at least that’s what 43% of the tweeps who responded thought). However it turns out that, at least in recent versions of Go, each […]