Memory inside Linux containers

Or why don’t free and top work in a Linux container?

Lately at Heroku, we have been trying to find the best way to expose memory usage and limits inside Linux containers. It would be easy to do it in a vendor-specific way: most container specific metrics are available at the cgroup filesystem via /path/to/cgroup/memory.stat, /path/to/cgroup/memory.usage_in_bytes, /path/to/cgroup/memory.limit_in_bytes and others.

An implementation of Linux containers could easily inject one or more of those files inside containers. Here is an hypothetical example of what Heroku, Docker and others could do:

# create a new dyno (container):
$ heroku run bash

# then, inside the dyno:
(dyno) $ cat /sys/fs/cgroup/memory/memory.stat
cache 15582273536
rss 2308546560
mapped_file 275681280
swap 94928896
pgpgin 30203686979
pgpgout 30199319103
# ...

/sys/fs/cgroup/ is the recommended location for cgroup hierarchies, but it is not a standard. If a tool or library is trying to read from it, and be portable across multiple container implementations, it would need to discover the location first by parsing /proc/self/cgroup and /proc/self/mountinfo. Further, /sys/fs/cgroup is just an umbrella for all cgroup hierarchies, there is no recommendation or standard for my own cgroup location. Thinking about it, /sys/fs/cgroup/self would not be a bad idea.

If we decide to go down that path, I would personally prefer to work with the rest of the Linux containers community first and come up with a standard.

I wish it were that simple.

The problem

Most of the Linux tools providing system resource metrics were created before cgroups even existed (e.g.: free and top, both from procps). They usually read memory metrics from the proc filesystem: /proc/meminfo, /proc/vmstat, /proc/PID/smaps and others.

Unfortunately /proc/meminfo, /proc/vmstat and friends are not containerized. Meaning that they are not cgroup-aware. They will always display memory numbers from the host system (physical or virtual machine) as a whole, which is useless for modern Linux containers (Heroku, Docker, etc.). Processes inside a container can not rely on free, top and others to determine how much memory they have to work with; they are subject to limits imposed by their cgroups and can’t use all the memory available in the host system.

This causes a lot of confusion for users of Linux containers. Why does free say there is 32GB free memory, when the container only allows 512MB to be used?

With the popularization of linux container technologies – Heroku, Docker, LXC (version 1.0 was recently released), CoreOS, lmctfy, systemd and friends – more and more people will face the same problem. It is time to start fixing it.

Why is this important?

Visibility into memory usage is very important. It allows people running applications inside containers to optimize their code and troubleshoot problems: memory leaks, swap space usage, etc.

Some time ago, we shipped log-runtime-metrics at Heroku, as an experimental labs feature. It is not a portable solution though, and does not expose the information inside containers, so that monitoring agents could read it. To make things worse, most of the monitoring agents out there (e.g.: NewRelic?1) rely on information provided by free, /proc/meminfo, etc. That is plain broken inside Linux containers.

On top of that, more and more people have been trying to maximize resource usage inside containers, usually by auto-scaling the number of workers, processes or threads running inside them. This is usually a function of how much memory is available (and/or free) inside the container, and for that do be done programmatically, the information needs to be accessible from inside the container.

More about /proc

In case you wondered, none of the files provided by the cgroup filesystem (/sys/fs/cgroup/memory/memory.*) can be used as a drop-in replacement (i.e.: bind mounted on top of) for /proc/meminfo, or /proc/vmstat. They have different formats and use slightly different names for each metric. Why memory.stat and friends decided to use a format different from what was already being used at /proc/meminfo is beyond my comprehension.

Some of the contents of a /proc filesystem are properly containerized, like the /proc/PID/* and /proc/net/* namespaces, but not all of them. Unfortunately, /proc in general is considered to be a mess. From the excellent “Creating Linux virtual filesystems” article on LWN:

Linus and numerous other kernel developers dislike the ioctl() system call, seeing it as an uncontrolled way of adding new system calls to the kernel. Putting new files into /proc is also discouraged, since that area is seen as being a bit of a mess. Developers who populate their code with ioctl() implementations or /proc files are often encouraged to create a standalone virtual filesystem instead.

I went ahead and started experimenting with that: procg is an alternative proc filesystem that can be mounted inside linux containers. It replaces /proc/meminfo with a version that reads cgroup specific information. My goal was for it to be a drop-in replacement for proc, without requiring any patches to the Linux kernel. Unfortunately, I later found that it was not possible, because none of the functions to read memory statistics from a cgroup (linux/memcontrol.h and mm/memcontrol.c) are public in the kernel. I hope to continue this discussion on LKML soon.

Others have tried similar things modifying the proc filesystem directly, but that is unlikely to be merged to the mainstream kernel if it affects all users of the proc filesystem. It would either need to be a custom filesystem (like procg) or a custom mount option to proc. E.g.:

mount -t proc -o meminfo-from-cgroup none /path/to/container/proc

FUSE

There is also a group of kernel developers advocating that this would be better served by something outside of the kernel, in userspace, making /proc/meminfo be a virtual file that collects information elsewhere and formats it appropriately.

FUSE can be used to implement a filesystem in userspace to do just that. Libvirt went down that path with its libvirt-lxc driver. There were attempts to integrate a FUSE version of /proc/meminfo into LXC too.

Even though there is a very nice implementation of FUSE in pure Go, and that I am excited with the idea to contribute a plugin/patch to Docker using it, at Heroku we (myself included) have a lot of resistance against using FUSE in production.

This is mainly due to bad past experiences with FUSE filesystems (sshfs, s3fs) and the increased surface area for attacks. My research so far has revealed that the situation may be much better nowadays, and I would even be willing to give it a try if there were not other problems with using fuse to replace the proc filesystem.

I am also not comfortable with making my containers dependent on an userspace daemon that serves FUSE requests. What happens when that daemon crashes? All containers in the box are probably left without access to their /proc/meminfo. Either that, or having to run a different daemon per container. Hundreds of containers in a box would require hundreds of such daemons. Ugh.

/proc is not the only issue: sysinfo

Even if we could find a solution to containerize /proc/meminfo with which everyone is happy, it would not be enough.

Linux also provides the sysinfo(2) syscall, which returns information about system resources (e.g. memory). As with /proc/meminfo, it is not containerized: it always returns metrics for the box as a whole.

I was surprised while testing my proc replacement (procg) that it did not work with Busybox. Later, I discovered that the Busybox’s implementation of free does not use /proc/meminfo. Guess what? It uses sysinfo(2). What else out there could also be using sysinfo(2) and be broken inside containers?

ulimit, setrlimit

On top of cgroup limits, Linux processes are also subject to resource limits applied to them individually, via setrlimit(2).

Both cgroup limits and rlimit apply when memory is being allocated by a process.

systemd

Soon, cgroups are going to be managed by systemd. All operations on cgroups are going to be done through API calls to systemd, over DBUS (or a shared library).

That makes me think that systemd could also expose a consistent API for processes to query their available memory.

But until then…

Solution?

Some kernel developers (and I am starting to agree with them) believe that the best option is an userspace library that processes can use to query their memory usage and available memory.

libmymem would do all the hard work of figuring out where to pull numbers from (/proc vs. cgroup vs. getrlimit(2) vs. systemd, etc.). I am considering starting one. New code could easily benefit from it, but it is unlikely that all existing tools (free, top, etc.) will just switch to it. For now, we might need to encourage people to stop using those tools inside containers.

I hope my unfortunate realization – figuring out how much memory you can use inside a container is harder than it should be – helps people better understand the problem. Please leave a comment below and let me know what you think.


  1. To be fair, I don’t really know what NewRelic is doing these days, I am just using them as an example. They may be reading memory metrics in a different way (maybe aggregating information from proc/*/smaps). Pardon my ignorance. 

Software Development Guidelines

From Merrian-Webster:

guide·line (noun): an indication or outline of policy or conduct

U.S. Dept. of Veterans Affairs:

A guideline is a statement by which to determine a course of action. A guideline aims to streamline particular processes according to a set routine or sound practice. By definition, following a guideline is never mandatory. Guidelines are not binding and are not enforced.

These definitions are very important. What I am listing here are my guidelines, a set of items that usually drive my particular style of coding and engineering. They are not rules. As someone once wisely pointed out (no references, sorry):

“guidelines account for judgement, rules don’t”

They are also my guidelines. It is ok if they do not fit in your case, I am not trying to describe the definitive way of writing, maintaining and operating software. Most of the items here should be obvious for many people, but I hope this helps you think about your own guidelines and understand more others.

After a quick disclaimer, let us go to the list. There is a lot more I would like to publish someday, but I will try to not make this too long and painful.

Extremism

There is a big chance that ying-yang is in my blood; my father is Chinese. One of the most important lessons I learned so far is that extremism is bad, everything needs balance.

This is at the top of my list, mainly because it nicely applies to the other items as well. I consider them to be good ideas, but I will not just blindly apply them to every situation. I usually do, unless I can come up with a very good justification not to.

The recursiveness of this item is also beautiful. It means that you can even be extreme (passionate?) about something, as long as you can give a damn good reason for it. You should not be extreme about not being extreme.

Take for example my deep hate for inheritance (in the OO context): there are good reasons for it. But it is not true that I would never use inheritance: the fact that it should be avoided is something to keep in mind, not something to block you from delivering software.

While we are on the topic, I really like how Go approaches inheritance.

Enabling vs. Directing

I took this very important lesson from some good discussions I had with my friend Tiago many years ago, back when we were coworkers in Germany.

Enabling means that something can be used in many different ways. Even in ways that we have not even considered yet. Humans are creative and will come up with different ways of using and applying an enabling idea. Directing however, means something that is designed to be used in a particular way, or an specific action.

A good example is how to design Java annotations. Here is how we could annotate a class to define that its objects need to be saved to a database (i.e. they are persistent):

@Save
public class Person {
  // ...
}

This is an example of a directing implementation, it specifically tells that objects of this class need to be saved. Using that information for anything else would be awkward.

Here is how the Java Persistence API (thanks to Hibernate) defines it:

@Entity
public class Person {
  // ...
}

This information could be used by many different components on a system. For instance, a logging component could read and use it to determine that some of its fields need to be filtered from logs (such as passwords), just because persistent entities usually contain sensitive data. A formatting component could use different colors for entity objects. There are all sorts of different uses for that information and creative people will come up with even more.

It does not mean that all of those uses will be correct, but that enables more from your code and makes it potentially more flexible.

This concept applies to almost any decision we need to do related to software development. Think about shipping a new feature, designing a component, planning an api, coding styleguides, people and project management styles, technology (framework?) choices, etc. There is probably a more enabling than directing way of doing them all in each context, which would empower people more.

In practice, it can be very hard. I have experienced many cases where it was really hard to tell if we were directing or enabling people. Keeping this in mind already helps a lot though.

Premature Optimization

This has been discussed a lot already, no need to talk too much about it. We all know that “Premature optimization is the root of all evil”.

However, quoting Albert Einstein:

“Everything should be made as simple as possible, but not simpler.”

Do not use premature optimization as an excuse to be lazy, or irresponsible about what you ship. Keep it in mind, but balance it up with your previous experience and feedback from others. There are times when you just know it is going to bite you soon.

Which leads us to the next item…

Productionization

I have learnt a lot of this over the past years running large production services (both at Locaweb and now at Heroku).

From the beginning, think about how your code is going to run in production. Do you understand the platform (runtime) it is going to run on? Are you confortable in troubleshooting hard problems? While we are on the topic, how are you planning to debug those problems?

Are you collecting metrics of how it is being used and how it services its requests? Even if you do not officially publish a SLA, understanding your service times is invaluable to dig into issues that will come with production load. Remember to track not only mean values, mean alone is useless. Always mix it up with variance, or rather with percentiles. Try to target for high percentiles – 95th and 99th are good targets, depending on what your scale is.

Last but not least: are you confident that you can notice (and be notified of) problems before your customers start complaining about your service on twitter? Metrics and automated monitoring are very important to run production services.

Testing

It does not really matter if automated or not. Yes, I said it. I have seen successful software (for whatever that means) both ways. Testing what you ship is just being responsible about it. And that includes having the proper infrastructure to do it: staging environments, gradual rollouts, feature flagging, etc.

Do not get me wrong. Automated testing should be usually preferred, but it is not the ultimate goal. I have seen many “evangelists” speaking hours about how automated testing is important, while the kernel of the operating system they use prefers a more traditional approach with lots of manual (or semi-automated) tests, Q/A testing teams and not many (or even zero) automated tests in its codebase.

Quick note about test (or behavior) driven development: it has more to do with software design methods than with the tests themselves. IMO it is a good practice, which works for me sometimes, but not always in every project, everyday.

Dependency Inversion

I am really proud of what some of my heavy Java development days have taught me.

Designing loosely coupled components is a big one. Dependency Injection, Dependency Inversion Principle and Inversion of Control are all related topics that to me mean a simple thing: design for single (or few) responsibilities.

It means that when you are writing that piece of code (be it a method, function, class, module, script or whatever), focus it on a single responsibility, and keep it small. If it gets to big, break it. If it needs resources to do its job, do not go after the resources it needs there, receive (inject) them instead.

In summary: instead of going after your dependencies, inject them. When a component needs to go after a resource it needs, it usually means that:

  1. The component is now responsible for its dependency lifecycle: this adds more responsibility. Now that it opens that damn connection to your DB (or your queue service), it needs to decide when to close, and needs to know how to do it. Also, what happens when you need to share the same connection in other parts of the system?
  2. The resource needs to be globally accessible: if the component did not create the resource it needs, or if that resource is shared in many parts of the codebase, it needs to be globally available. Hiding the resource creation behind factories does not help when the factory objects themselves need to be globally reachable. I hope I do not need to talk much about how globally accessible things are bad, but it is usually hard to test components using them, and it leads to tighter coupling. Changing that global resource gets harder with more stuff using it.

When you invert control and inject dependencies instead, you have the opportunity to centralize resource management in a single place. That single place centralizes the (single) responsibility of deciding where to inject that resource and how to share it. In more fancy environments, something like this can be called Dependency Injection Framework, but it does not need to be a big bloated framework once you understand the mechanics. In fact, this does not require a framework at all, it is just plain old method/function/constructor invocation with proper parameters.

All in all, let us please not forget about balance. This is not a rule, there are times when what you need is just a goddamn simple function.

Do Not Block The Event Loop

Evented programming (or event-driven programming) is very popular these days and one of the side effects is that some projects will want to be evented without careful consideration.

Going evented or not is one of the big decisions involved in writing code, with big implications. Things blocking the event loop are usually very hard to debug. When you go that route, all code called (including external libraries) needs to be aware of the event loop and not to block it.

There are many advantages though, notably it leads to much more lightweight servers which support much higher concurrency levels. When I am deciding if I should do event-driven code or not, here is what I consider (feel free to add a comment below with your own thoughts):

  • Is it an event-driven runtime/platform? Nodejs, for example, has been designed from the ground up to be evented. Meaning that all libraries and code written to run on it are already aware of the event loop. If the platform was not designed to be evented (like Ruby with EventMachine) much more care must be taken to not call code which will block the event loop. It is hard to control all the code in libraries included in your project. Take that into consideration.
  • Consider evented if the piece of code you are writing is mostly a data multiplexer, meaning that it just takes data from one side and sends it to another, acting like a pipe, distributor, load balancer, or router. This type of component is usually I/O bound.
  • Avoid evented if the piece of code you are writing is CPU bound and does a lot of processing. The chance it will block the event loop is much higher. I have seen projects having to resort to threads (or external processes) to move that part out of the event loop, often leading to spaghetti concurrency very hard to follow. Part evented, part threaded.

There Is Always Something To Learn

As I learn more, I expect this list to change, but I would say that it currently contains the factors influencing my development style the most. It was a very healthy exercise to think about what defines me as a programmer. I hope it is for you too, try it out!

Running Java Web Applications on Heroku Cedar Stack

Update: Heroku now does support Java.

Heroku does not officially support Java applications yet (yes, it does). However, the most recently launched stack comes with support for Clojure. Well, if Heroku does run Clojure code, it is certainly running a JVM. Then why can we not deploy regular Java code on top of it?

I was playing with it today and found a way to do that. I admit, so far it is just a hack, but it is working fine. The best (?) part is that it should allow any maven-based Java Web Application to be deployed fairly easy. So, if your project can be built with maven, i.e. mvn package generates a WAR package for you, then you are ready to run on the fantastic Heroku Platform as a Service.

Wait. There is more. In these polyglot times, we all know that being able to run Java code means that we are able to run basically anything on top of the JVM. I’ve got a simple Scala (Lift) Web Application running on Heroku, for example.

There are also examples of simple spring-roo and VRaptor web applications. I had a lot of fun coding on that stuff, which finally gave me an opportunity to put my hands on Clojure. There are even two new Leiningen plugins: lein-mvn and lein-herokujetty. :-)

VRaptor on Heroku

Let me show you what I did with an example. Here is a step-by-step to deploy a VRaptor application on Heroku Celadon Cedar:

  1. Go to Heroku and create an account if you still do not have.
  2. We are going to need some tools. First Heroku gems:
    $ gem install heroku
    $ gem install foreman
    
  3. Then Leiningen, the clojure project automation tool. On my Mac, I installed it with brew:
    $ brew update
    $ brew install leiningen
    
  4. The vraptor-scaffold gem, to help us bootstrap a new project:
    $ gem install vraptor
    

That is it for the preparation. We may now start dealing with the project.

  1. First, we need to create the project skeleton:
    $ vraptor new <project-name> -b mvn
    $ lein new <project-name>
    $ cd <project-name>
    

    The lein command is not strictly necessary, but it helps with .gitignore and other minor stuff.

  2. Now, the secret sauce. This is the ugly code that actually tries to be smart and tricks Heroku to do additional build/compilation steps:
    $ wget https://gist.github.com/raw/1129069/f7ffcda9c42a3004fa8d3c496d4f13887c11ebf4/heroku_autobuild.clj -P src
    

    Or if you do not have wget installed:

    $ curl -L https://gist.github.com/raw/1129069/f7ffcda9c42a3004fa8d3c496d4f13887c11ebf4/heroku_autobuild.clj > src/heroku_autobuild.clj
    
  3. You also need to tweak the Leiningen project definition – project.clj. The template is here. Please remember to adjust your project name. It must be the same that you are using in the pom.xml. Or you may download it directly if you prefer:
    $ curl -L https://gist.github.com/raw/1129081/5790e173307d28a8df256e0aaa5a9fe7757e922f/project.clj > project.clj
    
  4. Unfortunately, Leiningen comes bundled with an old version of the Maven/Ant integration. The embedded maven is currently at version 2.0.8, which is old. The versions of some plugins configured in the default pom.xml from vraptor-scaffold are incompatible with this old version of maven.

    The best way to solve it for now is to remove all <version> tags from items inside the <plugins> section of your pom.xml. This is specific to VRaptor and other frameworks may need different modifications. See below for spring-roo and Lift instructions.

    If even after removing versions from all plugins, you still get ClassNotFoundException: org.apache.maven.toolchain.ToolchainManager errors, try cleaning your local maven repository (the default location is $HOME/.m2/repository). The pom.xml that I used is here.

  5. Now, try the same command that Heroku uses during its slug compilation phase. It should download all dependencies and package your application in a WAR file inside the target directory.
    $ lein deps
    

    Confirm that the WAR was built into target/. It must have the same name as defined in your project.clj.

  6. Create your Heroku/Foreman Procfile containing the web process definition:
    web: lein herokujetty
    

    And test with:

    $ foreman start
    

    A Jetty server should start and listen on a random port defined by foreman. Heroku does the same.

  7. Time to push your application to Heroku. Start editing your .gitignore: add target/ and tmp/, and please remove pom.xml. Important: The pom.xml must not be ignored.
  8. You may also remove some unused files (created by leiningen):
    $ rm -rf src/<project-name>
    # do not remove src/main src/test and src/heroku_autobuild.clj!
    
  9. Create a git repository:
    $ git init
    $ git add .
    $ git commit -m "initial import"
    
  10. Push everything to Heroku and (hopefully) watch it get built there!
    $ heroku create --stack cedar
    $ git push -u heroku master
    
  11. When it ends, open another terminal on the project directory, to follow logs:
    $ heroku logs -t
    
  12. Open the URL that heroku creates and test the application. You may also spawn more instances and use everything Heroku has to offer:
    $ heroku ps:scale web=2
    $ heroku ps:scale web=4
    $ heroku ps:scale web=0
    
  13. Quick tip: you do not have SSH access directly to Dynos, but this little trick is very useful to troubleshoot issues and to see how slugs were compiled:
    $ heroku run bash
    

spring-roo on Heroku

Once you understand the process for a VRaptor application, it should be easy for others as well. Just follow the simple step-by-step here, to create a simple spring-roo maven application.

Then, before running lein deps or foreman start, you must adjust your pom.xml, to remove plugins incompatible with the older mvn that is bundled in Leiningen. Here is the pom.xml that I am using (with some of the plugins downgraded).

You can see my spring-roo application running on Heroku here.

Scala/Lift on Heroku

The same for Scala/Lift: follow the instructions to bootstrap an application with maven. One simple change to the pom.xml is required. My version is here.

You also need to turn off the AUTO_SERVER feature of H2 DB. It makes H2 automatically starts a server process, which binds on a tcp port. Heroku only allows one port bound per process/dyno, and for the web process, it must be the jetty port.

To turn it off, change the database URL inside src/main/scala/bootstrap/liftweb/Boot.scala. I changed mine to a in-memory database. The URL must be something similar to jdbc:h2:mem:<project>.db.

My Lift application is running on Heroku here.

I hope it helps. Official support for Java must be in Heroku’s plans, but even without it yet, we’ve got a lot of possibilities.

DSLs: one interface, multiple implementations

One interface, multiple implementations is one of these design concepts I like most. It’s basically polymorphism in its pure form!

Coding a little bit a while ago, I realized how DSLs could reinforce this idea. My example here is a simple implementation for queue consumers, as a simple Ruby internal DSL:

class GitRepositoryCloner < Consumer
  queue "RepositoriesToBeCloned"
  exclusive true

  handle do |message|
    # git clone repository!
    # I'm pretty sure github has something alike
  end
end

And to enqueue a message, we could do:

GitRepositoryCloner.publish("rails")

GitRepositoryCloner is a simple consumer of the queue named RepositoriesToBeCloned. One of the times I did something similar to this, I needed support for exclusive consumers. Then, my choices were ActiveMQ as the messaging middleware together with the Stomp protocol.

Using them as an example, let’s take a look on a possible implementation for the Consumer class, using the stomp gem:

module ActiveMQ
  class Consumer

    def self.queue(name)
      @queue_name = name
    end

    def self.exclusive(bool)
      @exclusive = bool
    end

    def self.handle(&blk)
      @callback = blk
    end

    def self.listen
      broker = Stomp::Client.new(Config[:broker])
      broker.subscribe(@queue_name, 
          :'activemq.exclusive' => @exclusive) do |message|
        @callback.call(message)
        broker.acknowledge(message)
      end
    end

    def self.publish(message)
      broker = Stomp::Client.new(Config[:broker])
      broker.publish(@queue_name, message, :persistent => true)
    end

  end
end

Consumer = ActiveMQ::Consumer

The last line is where we choose the ActiveMQ::Consumer as the default implementation.

The beautiful aspect of this little internal DSL, composed only of three methods (queue, exclusive and handle), is that it defines an interface. Here, I have seen a common misconception from many developers coming from Java, C# and similar languages which have the interface construct. An interface in Object Orientation is composed of all accessible methods of an object. In other words, the interface is the object’s face to the rest of the world. We are not necessarily talking about a Java or C# interface construct.

In this sense, these three methods (queue, exclusive and handle) are the interface of the Consumer internal DSL (or class object, as you wish).

Let’s say for some reason, we would like to switch our messaging infrastructure to something else, like Resque, which Github uses and is awesome. Resque’s documentation says that things are a little bit different for Resque consumers. They must define a @queue class attribute and must have a perform class method.

As we would do with regular Java/C# interfaces, let’s make another implementation respecting the previous contract:

module Resque
  class Consumer

    def self.queue(name)
      @queue = name
    end

    def self.exclusive(bool)
      self.extend(Resque::Plugins::LockTimeout) if bool
    end

    def self.handle(&blk)
      self.send(:define_method, :perform, &@blk)
    end

    def self.listen
      raise "Not ready yet" unless self.respond_to?(:perform)
    end

    def self.publish(message)
      Resque.enqueue(self, message)
    end

  end
end

There you can see how the implementations differ. The exclusive consumer feature is provided by the LockTimeout plugin. In this case, instead of passing the activemq.exclusive parameter to the connection, we must use the Resque::Plugins::LockTimeout module, as the documentation says. Another key difference is in the message handling process. Instead of passing a handler block to the subscribe method, Resque consumers are required to define a perform method, which we are dynamically creating with some metaprogramming: Class#define_method(name).

Finally, here is how we switch our messaging backend to Resque, without any changes to the consumer classes (GitRepositoryCloner in this example):

Consumer = Resque::Consumer

That’s it: one interface, two (multiple) implementations.

Ruby and dependency injection in a dynamic world

It’s been many years I’ve been teaching Java and advocating dependency injection. It makes me design more loosely coupled modules/classes and generally leads to more extensible code.

But, while programming in Ruby and other dynamic languages, the different mindset always intrigued me. Why the reasons that made me love dependency injection in Java and C# don’t seem to apply to dynamic languages? Why am I not using DI frameworks (or writing simple wiring code as I did many times before) in my Ruby projects?

I’ve been thinking about it for a long time and I don’t believe I’m alone.

DI frameworks are unnecessary. In more rigid environments, they have value. In agile environments like Ruby, not so much. The patterns themselves may still be applicable, but beware of falling into the trap of thinking you need a special tool for everything. Ruby is Play-Doh, remember! Let’s keep it that way.

– Jamis Buck

Even being a huge fan of DI frameworks in the Java/C# world (particularly the lightweights), I completely agree with Jamis Buck (read his post, it’s very good!). This is a recurrent subject in talks with really smart programmers I know and I’ve tried to explain my point many times. I guess it’s time to write it down.

The whole point about Dependency Injection is that it is one of the best ways to achieve Inversion of Control for object dependencies. Take the Carpenter object: his responsibility is to make and repair wooden structures.

class Carpenter {
    public void repair(WoodenStructure structure) {
        // ...
    }
    public WoodenStructure make(Specification spec) {
        // ...
    }
}

Because of his woodwork, the carpenter often needs a saw (dependency). Everytime the carpenter needs the saw, he may go after it (objects who take care of their dependencies on their own).

class Carpenter {
    public void repair(WoodenStructure structure) {
        PowerSource powerSource = findPowerSourceInsideRoom(this.myRoom);
        Saw saw = new ElectricalSaw(powerSource);

        // ok, now I can *finally* do my real job...
    }
}

The problem is that saws could be complicated to get, to assemble, or even to build. The saw itself may have some dependencies (PowerSource), which in turn may also have some dependencies… Everyone who needs a saw must know where to find it, and potencially how to assemble it. What if saw technology changes and some would rather to use hydraulic saws? Every object who needs saws must then be changed.

When some change requires you to mess with code everywhere, clearly it’s a smell.

Have you also noticed that the carpenter has spent a lot of time doing stuff which isn’t his actual job? Finding power sources and assembling saws aren’t his responsibilities. His job is to repair wooden structures! (Separation of Concerns)

One of the possible solutions is the Inversion of Control principle. Instead of going after their dependencies, objects receive them somehow. Dependency Injection is the most common way:

class Carpenter {
    private Saw saw;
    public Carpenter(Saw saw) {
        this.saw = saw;
    }
    public void repair(WoodenStructure structure) {
        // I can focus on my job!
    }
}

Now, the code is more testable. In unit tests where you don’t care about testing saws, but only about testing the carpenter, a mock of the saw can be provided (injected in) to the carpenter.

In the application code, you can also centralize and isolate code which decides what saw implementation to use. In other words, you now may be able to give the responsibility to do wiring to someone else (and only to him), so that objects can focus on their actual responsibilities (again Separation of Concerns). DI framework configuration is a good example of wiring centralized somewhere. If the saw implementation changes somehow you have just one place to modify (Factories help but don’t actually solve the problem – I’ll leave this discussion for another post).

class GuyWithResponsibilityToDoWiring {
    public Saw wireSaw() {
        PowerSource powerSource = getPowerSource();
        return new Saw(powerSource);
    }
    public PowerSource getPowerSource() { ... }
}

Ok. Plain Old Java Dependency Injection so far. But what about Ruby?

Sure you might do dependency injection in Ruby (and its dynamic friends) exactly the same way as we did in Java. But, it took some time for me to realize that there are other ways of doing Inversion of Control in Ruby. That’s why I never needed a DI framework.

First, let’s use a more realistic example: repositories that need database connections:

class Repository {
    private Connection connection;
    public Repository(Connection connection) {
        this.connection = connection;
    }
    public Something find(Integer id) {
        return this.connection.execute("SELECT ...");
    }
}

Our problem is that many places need a database connection. Then, we inject the database connection as a dependency in everywhere it is needed. This way, we avoid replicating database-connection-retrieval code in such places and we may keep the wiring code centralized somewhere.

In Ruby, there is another way to isolate and centralize common behavior which is needed in many other places. Modules!

module ConnectionProvider
  def connection
    # open a database connection and return it
  end
end

This module can now be mixed, or injected in other classes, providing them its functionality (wiring). The “module injection” process is to some degree similar to what we did with dependency injection, because when a dependency is injected, it is providing some functionality to the class too.

Together with Ruby open classes, we may be able to centralize/isolate the injection of this module (perhaps even in the same file it is defined):

# connection_provider.rb

module ConnectionProvider
  def connection
    # open a database connection and return it
  end
end

# reopening the class to mix the module in
class Repository
  include ConnectionProvider
end

The effect inside the Repository class is very similar to what we did with dependency injection before. Repositories are able to simply use database connections without worrying about how to get or to build them.

Now, the method that provides database connections exists because the ConnectionProvider module was mixed in this class. This is the same as if the dependency was injected (compare with the Java code for this Repository class):

# repository.rb

class Repository
  def find(id)
    connection.execute("SELECT ...")
  end
end

The code is also very testable. This is due to the fact that Ruby is a dynamic language and the connection method of Repository objects can be overridden anywhere. Particularly inside tests or specs, the connection method can be easily overridden to return a mock of the database connection.

I see it as a different way of doing Inversion of Control. Of course it has its pros and cons. I can tell that it’s simpler (modules are a feature of the Ruby language), but it may give you headaches if you have a multithreaded application and must use different implementations of database connections in different places/threads, i.e. to inject different implementations of the dependency, depending on where and when it’s being injected.

Opening classes and including modules inside them is a global operation and isn’t thread safe (think of many threads trying to include different versions of the module inside the class). I’m not seeing this issue out there because most of Ruby applications aren’t multithreaded and run on many processes instead; mainly because of Rails.

Regular dependency injection still might have its place in Ruby for these cases where plain modules aren’t enough.

IMO, it’s just much harder to justify. In my experience modules and metaprogramming are usually just enough.

Ruby indentation for access modifiers and their sections

Ruby programmers are very flexible (and permissive) when talking about Ruby code indentation. Most of rubyists I know prefer to indent with two spaces instead of tabs (soft tabs in some editors). There are even some style guides and code conventions published for the language, but none of them is official and they talk too little about code indentation practices.

Things go even worse when we have to choose how to indent private, protected and public sections of classes. Programmers and projects I’ve seen so far seem to adopt different styles. I’ll try to summarize them here:

1. Indent as a new block

This is the most common pattern I see out there. Some programmers prefer to treat sections created by access modifiers as new blocks and indent them:

class MyClass

  def the_public_method
    # ...
  end

  private

    def first_private_method
      # ...
    end

    def second_private_method
      # ...
    end

  protected

    def the_protected_method
      # ...
    end

end

Pros:

  • Visibility: easy to see if a method has a non-standard access modifier (non-public in most cases).
  • Produces a very readable code.
  • Used in the rails codebase (mainly older code).

Cons:

  • Semantically wrong: access modifiers do not create new scopes. They are simple method calls. Deeper explanation in the next item.
  • Methods inside the same scope with different indentation levels.
  • (opinion) One more level of indentation. When you have classes inside modules it starts being a problem: “indentation hell” :-). Your code ends up using its 80 columns very fast and you start having to break lines more often.

2. No indentation

Ruby access modifiers are simple method calls. The Module class (from which Ruby classes inherit) has the private, protected and public methods, that when called with no arguments, simply change the visibility of subsequent defined methods. You may confirm this by testing that the following code works:

class MyClass

  self.send(:private)

  def the_private_method
    # ...
  end

  def another_private_method
    # ...
  end

end

Try to call any of these methods in an instance of the MyClass class and you will confirm they are private.

Because access modifiers are simple method calls, they don’t create a new scope. Semantically speaking, they shouldn’t create another level of indentation. This is even advocated by one of the most important Ruby style guides.

class MyClass

  def the_public_method
    # ...
  end

  private

  def first_private_method
    # ...
  end

  def second_private_method
    # ...
  end

  protected

  def the_protected_method
    # ...
  end

end

Because of its correctness, this was my preferred style until recently, when I found the next ones.

Pros:

  • It is semantically correct: method calls don’t create new scopes, then method calls shouldn’t increase indentation levels.
  • (opinion) this style makes access modifiers look like python decorators, Java annotations and C# attributes. IMO, it’s one of the nice things about Ruby: its ability to reproduce most of other languages features, without requiring new syntax or fancy constructs. It’s a simple method call.

Cons:

  • Hard to see if a method has any non-standard access modifier. In classes with many methods (code smell!) you must constantly scroll to see what access modifier is applied.
  • Some would argue that this could be solved with proper syntax highlighting or visual method decoration. But it requires editor/IDE support, then I’m considering it as a disadvantage.

3. if-else style

We have a similar case in Ruby: the if keyword creates a new block and has associated else and elsif statements, which are also new blocks. The convention suggests the following indentation (note that if, elsif and else are in the same indentation level):

if something?
  2.times { play }
  jump(2.meters)
elsif other?
  sing and dance
else
  cry
  walk
end

Some of the code recently pushed to Rails 3.0 use the same indentation style for classes with access modifiers. I liked it:

class MyClass

  def the_public_method
    # ...
  end

private

  def first_private_method
    # ...
  end

  def second_private_method
    # ...
  end

protected

  def the_protected_method
    # ...
  end

end

Pros:

  • Easy to see access modifiers inside classes. With a fast look it is easy to identify sections of private, public and protected methods. They look like blocks.
  • Readable code. Similar to other Ruby constructs.
  • Used in the rails codebase (mainly newer code).

Cons:

  • Still semantically wrong. Access modifiers are inside the scope created by the class keyword. They should be indented.
  • if, elsif and else are keywords and have special meaning. Access modifiers aren’t and should follow the same convention of other method calls.

4. Indentation with one space

C++ also splits private, protected and public methods in sections. The construct is very similar to Ruby:

class MyClass {
 public:
  MyClass();
  void ThePublicMethod(int n);
  void Print(ostream &output) const;

 private:
  int *Items;
  bool FirstPrivateMethod();
  int SecondPrivateMethod();
};

While reading Google’s style guide for C++ code, their recommendation for public, private and protected sections indentation caught my attention. They suggest that you indent the private, protected and public keywords with only one space.

It seemed a bit awkward in the beginning. But soon, I started liking it because it makes sections easy to identify, access modifiers remain indented inside classes and methods stay all in the same indentation level, as they are all in the same scope.

class MyClass

  def the_public_method
    # ...
  end

 private

  def first_private_method
    # ...
  end

  def second_private_method
    # ...
  end

 protected

  def the_protected_method
    # ...
  end

end

Pros:

  • Easy to see access modifiers and sections inside classes.
  • Semantically correct.
  • All methods remain in the same indentation level.
  • Google style guide.

Cons:

  • People are always fighting for 2 spaces vs 4 spaces vs tabs indentation. One space? Very uncommon.
  • Special treatment for regular method calls. Statements inside the same scope with different indentation levels

My choice

Currently, I tend to like the 3rd (new Rails style) and 4th (Google) styles more. I somehow feel they give the best deal, considering their advantages and disadvantages. Being able to easily identify sections and access modifiers inside classes is very important to me.

In my current project, the team adopted the Google style. I’m happy with it, but you must be flexible in the beginning to adopt one-space-indentation. I won’t lie, it feels strange until you get used. Another issue I have is that my TextMate indent Ruby code with two spaces and treat them as one “step” when navigating with keyboard cursors. Then, I have to type a few more keystrokes to indent access modifiers the way I want. It’s hard to explain, you will have to try it yourself.

And you, what’s your opinion?
How do you indent access modifiers and their sections inside classes?

Soda is cheaper than Water

(at least here in Brazil)

These prices were taken from http://www.paodeacucar.com.br, Sao Paulo, in the date of this post. Compare them:

Soda (Brazilian Guarana):

Guarana Tai

Versus standard natural Water:

Agua Indaia sem gas

The soda has approximately a R$ 0,92 price/liter ratio, versus R$ 1,25 price/liter of the natural water bottle. Some years ago, I would never imagine soda being so much cheaper than water. This just show us how the world is really turning fast.

Even being more expensive, I have no doubt that water is simpler than soda, and more healthy too. Hey, they use water in soda production process!

My message:

Even when the flavored (more complicated) solution seems to be cheaper and more attractive, many times the simpler solution is more healthy.

Think about it when making your next decision about technology and software design. Please.