Nimble Method

Why You Should Use 'Cache-Control: public' for Caching to Work with SSL

We've been using long Expires and Cache-Control: max-age headers to mark our javascripts and css as browser-cacheable for quite some time. This way the browser only does full GETs on these once, and avoids conditional GETs except on page refresh. This only requires a few lines of configuration in Rails (asset timestamps and asset packaging) and nginx and generally works well.

Except when it does not. We've had some bug reports that in some cases under SSL these files were not cached at all. Meaning, just going to a different page in our application would make browser do an unconditional GET to refetch css and javascript. Naturally, this would hurt the performance.

Now, SSL in itself makes things worse. Firefox only caches SSL content in memory not on disk, meaning if the user closes the browser the cache goes away. But the reported behavior was worse than that -- no caching at all.

It took us a while, but we did manage to chase it down. It seems that Firefox divides SSL content into page marked with Cache-Control: public, which have priority for caching, and the rest. When you open enough tabs, say 50+, with enough rich pages in Firefox, its in-memory cache fills up and refuses to cache non-public SSL content. We haven't had time to fully chase it down in the FF codebase, but what's likely to happen is that FF actually does put in into cache, then realizes that cache is full, looks for pages to evict, and right away evicts the newly added content. Since SSL content is not cached on hard disk, once the memory cache is full, the non-pubic SSL files are not cached at all. (Note that's just a hypothesis, we'd be most curious to know what actually happens and why.)

The server-side workaround is to add Cache-Control: public to all your SSL content you want cached. We do:

# nginx configuration for static content  
location /static {
    expires 1y;
    add_header Cache-Control public;
}

This bug is present in Firefox 2 and 3, but as a user you can configure them to work around the problem by setting browser.cache.memory.capacity to increase the size of memory cache or using browser.cache.disk_cache_ ssl to enable disk cache for SSL pages. The default 24M memory cache size for systems with 2G of RAM is really small, especially with FF3 being able to handle hundreds of tabs well. Note that the disk_cache option works only on Firefox 3 because of some bugs in earlier browser versions.

Small remark for Acunote users - you don't have to do anything about this. We enabled Cache-Control: public and your Firefox will cache javascript and css files no matter how many tabs you have.

The Road to Hell Is Paved With Caching

In Acunote we are heavily using fragment caching. We cache whenever we can and whatever we can. Sounds like a great idea to do? Well, almost... Sometimes caching hurts and here's why.

Prehistory

Acunote is a project management tool and therefore the most prominent page in Acunote is the task list in the current sprint. The page itself shows some information specific to the user who logged in and some information that is shared among the users within the user's organization.

We can cache the shared part of the page in the fragment cache taking date and page filters into account. The code roughly looks like this:

<%- cache(:part => "#{filters}_#{DateUtils::today}") do -%>
    <%= render :partial => '...' -%>
<%- end -%>

Using filesystem caching store we end up having this (simplified) cache directory layout:

 - organization1.acunote.com
   - sprints
     - show.part=filter1.date.cache
     - show.part=filter2.date.cache
     - show.part=filter3.date.cache
 - organization2.acunote.com
   - sprints
     - show.part=filter1.date.cache
     - show.part=filter2.date.cache
     - show.part=filter3.date.cache

Problem

Problems start when we try to expire caches. We can't expire by url or by hash because we want to expire all cache parts (for all possible filters and for all dates). Therefore we use regexp:

expire_fragment(%r{show.part=.*})

That turned into a problem in production. The reason is that expire_fragment in Rails searches the whole cache directory recursively and removes files with names matching the regular expression. On machines with local disks that's usually fast, but on network filesystems or on shared disk cluster filesystems (like GFS in our case) stat'ing lots of files can be really slow. To make things worse, the performance of stat call on GFS really depends on the phase of the moon or something like that ;)

Our logs showed that operations that would usually take less than a second took on occasion took 70 seconds! They were so slow so I could actually log in to our production servers, find the process taking too much time to complete and attach strace to it. Strace blessed me, immediately showing multiple stat's for files in the cache directory.

Solution

Longer term we'll switch to a more scalable fragment caching strategy. There is a good reason why high-performance caches do not support fancy invalidation, and there are known way to code to their more restricted interface. Meanwhile, immediate solution was really simple. We only had to expire fragments in specific directories and to make that easier, I wrote a little extension to Rails' fragment caching code:

module ActionController
  module Caching
    module Fragments

    #dir is the cache path relative to the cache root
    def expire_matched_fragment_in_dir(dir, regexp, options = nil)
      return unless perform_caching
      self.class.benchmark("Expired fragments in dir matching: #{regexp.source}") do
        fragment_cache_store.delete_matched_in_dir(dir, regexp, options)
      end
    end

    class UnthreadedFileStore

      def delete_matched_in_dir(dir, matcher, options = nil)
        path = @cache_path + dir
        return unless File.exist?(path) #it's ok to not have the cache dir
        search_dir(path) do |f|
          if f =~ matcher
          begin
            File.delete(f)
          rescue SystemCallError => e
            # If there's no cache, then there's nothing to complain about
          end
          end
         end
        end
      end

    end

  end
end

#include only if you use cache test plugin
module Cosinux
  module FragmentCacheTest
    class TestStore
      def delete_matched_in_dir(dir, matcher, options = nil)
        @deleted_matchers.push(matcher)
      end
    end
  end
end

Conclusions

Monitoring the performance of production application is sometimes even more important than profiling it on the development machines. We already had monitoring system in place so we could detect when our application got slower, figure out the root cause and and fix the problem.

Another thing I highly recommend is of course strace. You may indeed see that your application is doing something so weird you'd never think of just looking at the code or at the profiler. So, I now declare strace to be officially added into my own arsenal of invaluable profiling tools.