Be careful how you rsyslog

2015-04-28 by

Bandcamp was offline briefly yesterday due to what I like to call an unexpected single point of failure. Good systems design is all about addressing single points of failure, making sure you have redundancies in place, but sometimes you discover single points of failure that you didn’t realise you had.

Yesterday’s problem was caused by maintenance on our central rsyslog server, which we use to collect analytics from our application servers. When that central server went down, it set a chain of events in motion:

  • Remote logging from our app servers blocked, since we have rsyslog configured to use TCP, which attempts to guarantee delivery.
  • Those blocked messages blocked all syslog logging on the app servers, since the default rsyslog configuration puts all logging in to a single delivery queue.
  • Within minutes that delivery queue filled up, causing all subsequent logging requests to block, freezing not just our apps but also system services like sshd. So, no logging in.

In the course of responding to the outage we quickly decided it was prudent to reboot the affected servers before continuing to investigate the root cause. We didn’t know at the time that once we restarted our apps the countdown clock started ticking. Fortunately we got to the bottom of the problem before the servers froze up again.

Lesson learned! Our rsyslog configuration now uses a dedicated queue for remote logging, and that queue spills over to disk if it fills up, preventing rsyslog from blocking logging if the central server goes offline. Here’s the relevant code:

$ActionQueueType LinkedList
$ActionQueueFileName apptimer
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
local0.* @@rsyslogserver:10514


 Leigh Dyer is the Lead Engineer of the Systems Team at Bandcamp, Inc.

Embedded Player Customization

2010-11-24 by

Way back when we released our embedded music player, we made a conscious effort to keep it simple and small so we could get it to you and get it to you quickly.  Customization was pretty much nonexistent, but the player was lean and mean and sharing didn’t involve making any choices.

Then we got feedback that people wanted some options, so we added a few layouts and a cute naming scheme inspired by a sign on the wall at Bandcamp’s Seattle office (also known as a Starbucks), and figured we pretty much had everyone covered for a while.  While making this change, we got more requests for additional layouts, so I did a bit of work to move from the old hardcoded-in-actionscript layout model to a model that was easier to modify (for my own benefit) by having some internal data structures that described each layout by name.  This was pretty simple stuff — just positioning and sizing of the various elements.  I didn’t want to reinvent CSS in actionscript or anything.

This served us well enough for a long time, until we decided to finally create a player with a full track listing on it.  At that point, we were up to 6 layouts, which is about the max that will fit comfortably into our Share dialog. We decided that the next logical step was to expose this “add new layouts” functionality to everyone, so if you have a site where you have the ability to include arbitrary HTML, you can make the player look exactly like you want it. Note that I’m shrewdly excluding sites like MySpace and Facebook and tons of others that are very particular about what you can paste into them and will, in fact, modify what you paste.

So if you’d like to make a player that looks like this:

or like this:

and you’re kind of excited about dorking with JSON and hitting reload a lot, then this blog post is for you.

How it works

Normally, the embed code for a player contains a /size=grande/ portion of the URL which selects the layout for your player.  The dimensions of the <object> tag(s) are also set explicitly in the embed code, and must match the layout specified by the “size” argument, or things will look funny (this is because we can’t assume the Flash widget has script access to the host page’s DOM, so we can’t set that size from within the layer-outer ourselves).  Instead of using this “size” argument, you may provide a “layout” argument which specifies the URL to a layout file.  This layout file is a raw blob of JSON which simply tells us where everything in the player goes, how big it is, and maybe a few pieces of style information (like whether its text should be centered).

The layout specification

Here’s the layout for what I’d argue is the simplest possible functional player you can make:

    "play": { "x": 0, "y": 0, "w": 31, "h": 30, "show": true }

It’s just a play/pause button.  We set the position and the width and height of the overall layout, then we have an element in there that describes the play button and specifies that it should be visible.  Pretty straightforward.

You can also add overrides that only apply to either the track or album versions of the player.  For example:

    "play": { "x": 0, "y": 0, "w": 31, "h": 30, "show": true,
        "album": { "x": 10 }

This tells us that the play button should be at (0,0) unless we’re looking at an album player, in which case its x coordinate should be changed to 10.  You could specify this the other way around, too, by saying that it’s at 10,0 unless we’re looking at a track player, in which case the x should be 0.  Same difference.  If you’re wondering why we have this feature, just take a look at the difference between the track and album versions of any of the standard layouts (except ‘short’) — we move the buttons around and hide or show the next/prev buttons appropriately, and there are a couple other subtle differences as well.


The final piece of information you can specify per-element is “styles”.  Flash programmers will be comfortable with this one.  This is a named set of styles which will get applied to the various mx.* elements directly using setStyle().  Of course, we don’t say much about what types the elements are, so it’s hard for you to know exactly what styles apply, but if you stick to the basic, obvious ones (“color”, “textAlign”, “fontFamily”, “fontSize”, “fontWeight”), in addition to our custom styles documented in the “Special Styles” section below, you should be fine.  Here’s an example of a style blob for the “currenttime” element:

    "currenttime" : { "x": 100, "y": 0, "w":50, "h":16, "show": true,
        "styles": { "color" : "#000000", "textAlign": "right" }

The elements

Ok, so what elements do you have to work with?

element name(s) description
“art” This is the track or album art.  It should be square, and as of this writing, either 100×100 or 150×150 pixels in size.  It will load the correct resolution art for one of these sizes.  In the future, maybe we’ll support arbitrary sizes, but for now it’s just those two.  Use a different size at your own risk.
“maintext” and “subtext” These are the main header and the subheader.  What exactly we display here has shifted slightly, hence the vague names, but at the moment the main header is the item (album or track) title and the subheader is the artist name.  This could conceivably change in the future, but it seems kind of unlikely.  As an example of how this could change, in the past, I believe the subheader was “from <album> by <artist>” for tracks that are part of an album.
“play” The play/pause button.  Should be either 31×30 or 47×45 — we’ll switch the image appropriately.
“next” and “prev” The next/prev buttons.  Each should be 17 pixels wide and 11 pixels tall. Only useful on album players.
“currenttitle” This is the section that shows the title of the currently playing item.  This is only useful in an album player, since it would otherwise be redundant [note: see “Special Styles” below]
“currenttime” The current location in the track, expressed as mm:ss
“totaltime” The total time in the current track, expressed as mm:ss
“linkarea” The spot where we stick the “Buy”, “Download”, and “Share” links as appropriate
“timeline” The progress/seek bar.  You should probably keep this to 16 pixels tall or things might look funny
“tracklist” the clickable and scrollable track list.  Only really appropriate on album players.

Making an Embed Code

To create the custom player embed code, you must provide three additional parameters to the Flash embed: “height”, “width” and “layout.” The height and width are self-explanatory; the browser needs to know how big to make the <object> tag. The “layout” parameter is a specially-encoded version of the URL to your layout’s JSON description. To make creation of the embed code easier, we’ve provided a secret set of options in the Share dialog. Open the Share dialog by clicking “Share” on the track or album page, then to reveal the secret options, just shift-click on the radio button for “Venti” and the standard size options will change into boxes where you can enter the URL and the dimensions.

Special Styles

There are a couple of styles that can be added to items that are not standard Flash element styles:

  • The “currenttitle” item can have a style called “isMenu”, which, if set to the string “true”, causes the item to also act as a popup track list menu.
  • The “tracklist” item can have a style called “rowHeight” which defines the height, in pixels, of each row in the track list.

Other Notes

If you add a “debug=true” argument to your player, it will “cache-bust” the layout URL by appending some random junk to it.  This makes it a lot easier to tweak a layout by reloading it repeatedly, because Flash seems to be fairly aggressive about its caching and everyone probably doesn’t have control over the cache control headers on their web servers. This argument is added automatically in the preview in the “Share” dialog when you’ve selected a custom layout (see “Making an Embed Code” above), so you can just click the “Refresh player” link below the sample player to reload your url ad infinitum.


  1. Every element that is visible must have “show” set to true in your layout spec.  If your element is not showing up, double check that.
  2. Because our Flash is going to grab your layout file from your server, you must have a crossdomain.xml file which allows requests from To save you the trouble of googling the details, here is an example of a crossdomain.xml file that would work.

Happy custom-layouting!

Expression Evaluators

2010-02-07 by

Bugs are fun but this time I want to talk about win. Recently I finished work on a recursive descent expression evaluator in the Bandcamp code. (That’s a mouthful and I enjoy slinging that whenever I can—it’s up there with binary space partitions.) An expression evaluator is something that can take a sequence of characters like

2 + 2

and figure out that it means to add two and two and give up four. Expression evaluators are obscure but oddly enough I’ve written at least five in my professional life.

The 1st brush was when I was 25, new at Adobe and began to seriously dig into the C programming language. In the appendix of K&R I found the grammar for the entire language in a formal notation called “modified BNF”. It described, line by line, every element of the language beginning with the highest thing, the mysteriously named translation-unit, and breaking it down from there:

  translation-unit external-declaration


  declaration-specifiersopt declarator declaration-listopt compound-statement

In this grammar a thing is defined in terms of simpler things. And these simpler things are subsequently defined in terms of even simpler things. And so on, downward, until you get to the atoms of a language, basic elements like integer-constants and character-constants. This is a conventional device in Computing.

Can we take a moment to worship at the altar of K&R? In 189 thin pages and eight quick chapters you have everything there is to know about the C programming language. The tutorial that’s chapter 1 is a model of concise, practical writing. Please for the love of God everyone who wakes up and thinks I’ll write a computer book! internalize and become one with this chapter. I mean the entire book.

So I wrote a program that understood C based on this grammar. It didn’t evaluate the language, but it could parse it. Meaning it was code that understood the rules for the arrangement of the keywords and symbols in the language. Programming is all about reference and so of course self-reference pops up from time to time—I had written a program in C that understood C.

You think C’s an obscure language that should be put to pasture? You owe everything you’re doing on a computer right now to C. Your operating system is written in C. Your warm and fuzzy language that’s not C is written in C (and C is written in C—ha!). Photoshop is written in C (okay C++ but we regret ever touching the ++). Apache is written in C. The syntax of your favorite language is heavily influenced by C. The world runs on C. #thatsawesome #fistpump

The 2nd and 3rd times I wrote an expression evaluator were when I wrote some plug-ins for Photoshop a few years later. Photoshop has an external programming interface that programmers can use to write little (and big) plug-ins that live in the app and do algorithmic things to images. When Photoshop invokes a plug-in the code is given the pixels of the image, the code messes with those pixels and voilá an effect. A butt simple plug-in could zero the green and blue values of every pixel, effectively putting the image through a red filter. Or it could halve every value making the image darker. It could peek at nearby pixels; that’s how blurring is done. Once you realize pixels are just numbers you can go crazy with the effects.

Fellow Adobeite Joe Ternasky had the idea of making a meta plug-in—a plug-in for making plug-ins—where you could enter expressions into boxes:

R = [    ]
G = [    ]
B = [    ]

Then when the plug-in runs it applies those expressions to each pixel in the image.

R = r
G = 0
B = 0

would be the red filter.

R = r/2
G = g/2
B = b/2

would be the darkener. All the normal math operators are available, as are several functions to give the pixel’s original values, coordinates, its distance and angle from the center, etc., and the values of a few user-defined sliders. If you were enough of a math-head you could write your own plug-ins to do the canonical resize, rotate, blur, and then go crazy with whatever other effects you imagined. The plug-in was called Filter Factory.

It had a fan club.

Filter Factory needed an expression evaluator so Joe and I wrote one. Then for fun we rewrote it as a set of expression compilers that turned the expressions into raw machine language. Joe did the 68K compiler and I did the PowerPC. The filters consequently ran just as fast as the ones built into Photoshop. It was cool. This is when I began to understand recursive descent expression evaluators.

Fast forward to Bandcamp. The 4th expression evaluator I wrote was last summer for our HTML templates. A template might have lines like this:

<a href="{{band.stats_url}}">stats</a>

When the page is served the {{curly brace}} things are filled in by the Liquid template engine we use. Liquid can do conditionals—the stats link on a band’s page for example is shown only if you’re logged in:

{% if user.logged_in %}
<a href="{{band.stats_url}}">stats</a>
{% endif %}

The user never sees the curly braces or the conditionals. The HTML is put together on the server and the custom page with the {{blanks}} filled in and the conditionals figured out is sent down to you. Liquid is too limited for some of the really awesome stuff we do in Bandcamp, but because it’s Ruby and open source we’ve been able to hack on it. I added a full-on expression evaluator. Now we can have smarter conditionals and counters.

{% if user.logged_in && user.admin_level >= 10 %}
{% let items_shown = items_shown + 1 %}

This expression evaluator was a breeze to write in Ruby. It’s only a page of code and doesn’t cheat by calling eval().

The 5th expression evaluator. This week I’ve been improving the system we use to distribute files around among the servers that together make Bandcamp. All of the files on Bandcamp (images, audio tracks, etc.) are kept in a big virtual file system called the zstore. It looks like one really big hard drive that’s made from the hard drives of all the machines. The files are spread across all the servers, and no single server has a complete set of all the files. There is redundancy so that if one machine dies we’ve got copies of the files someplace else.

For example, the transcoding servers hold most of the lossless audio files so that they’re immediately available for encoding into whatever format is needed. The downloader machines keep the audio files around for quick-start downloading. The webapp machines covet the public files (cover art mostly, in various sizes and formats). But any file can be anywhere. Our zstore code can act like every file is always available on every machine, and if a file isn’t where it’s needed the zstore copies it first behind the scenes. I wrote the zstore as one of the original integral pieces of Bandcamp, way back in ancient history (2008). I’ve studied distributed file systems and implemented a few before. I was amped to write one for Bandcamp.

Why did I write one instead of using an existing DFS? That’s worth its own post. The bottom line is that distributed file systems are still thumb-sucking toddlers, Computer Science speaking. We’re still figuring out how they should work: dealing with connectivity events, ensuring redundancy, providing the most-recent copy of a file when it’s needed, archiving, versioning, etc. These are hard. By focusing on Bandcamp’s specific needs I was able to write a DFS that does just what we need, and it does it well.

In addition to copying files between machines on demand, there’s code that proactively copies files around to anticipate needs and reduce delays. For example when a user uploads cover art for an album it first goes into the zstore on the webapp machine that handled the upload. The other webapp machines then see it and grab copies for themselves.

I’ll take one.

Me too.

I’m in.

Each machine needs to know what files it should proactively copy. I was reworking this code last week and decided an expression evaluator was the bee’s knees for this. I extended the Liquid evaluator with strings and built-in variables and extended comparisons. Now each zstore machine has an expression like this:

morbo:    tag == 'lossless' && (id % 100) >=  0 && (id % 100) <= 47
linda:    tag == 'lossless' && (id % 100) >= 48 && (id % 100) <= 96
calculon: tag == 'lossless' && (id % 100) >= 97 && (id % 100) <= 99
devil:    public == true
santa:    public == true

Whenever a file is added or modified in the zstore each of the machines looks at its expression and decides if it should make a local copy. What the rules above mean is that morbo, linda and calculon all want the lossless audio files, and each claims some percent of them (because no machine is large enough to hold all of them). The terms tag and id are built-in variables that resolve to attributes of the file. Devil and santa want all the files marked public because they’re webapp machines.

(The part of the expression (id % 100) >= 0 && (id % 100) <= 47 essentially partitions the entire set of files into 100 slots based on the file ids. A file id is a large random integer assigned when a file is created; it’s never changed or re-used. The % operator means modulo. I recently learned about a persistent partitioning scheme that uses a much higher number of slots and then distributes groups of slots to particular machines. This has the advantage of creating less havoc when a machine disappears. TODO.)

Doesn’t that look a bit like Filter Factory?

The code for this latest incarnation of Joe’s Recursive Descent Expression Evaluator is less than 200 lines of Ruby. You can take a look at it here:

bc-expression at google code

Start about half way down at parse_comparison_op. One of the interesting things about a top-down grammar is that it’s easy to implement that way, too. See if you can follow the code from there to the bottom of the source. (By the way, does anyone want to turn this into a gem? TXTMEOK) I find it fascinating when the same Computing concepts—the expression evaluator in this case—come up over and over again in wildly different contexts. I love this about programming.

You may be asking what’s recursive about all of this? Think about parentheses in an expression. If the topmost item in the grammar is an expression, and you descended the grammar through all the valid syntax, at the very bottom you’d come to a term. A term is the simplest unit that has a value. It’s either a constant (42), a variable (id, tag, public), or… well, take a look:


  comparison || comparison
  comparison && comparison

  additive == additive
  additive != additive
  additive < additive
  additive > additive

  multiplicative + multiplicative
  multiplicative - multiplicative

  term * term
  term / term
  term % term

  ( expression )  # Hey!

How Not To Generate A CRC

2010-01-21 by

Yesterday we had a success disaster. An überpopular mashup album that mixed the Wu Tang Clan and The Beatles was released a few days ago and yesterday it got stumbled upon and boing boinged and tweeted to the four winds. Our downloads per minute went through the roof. We’ve dealt with big releases in the past and our download machines typically respond with a big collective yawn. This time around we had problems. Downloads were slow to start and the downloader processes crashed a bunch. We eventually had to make the album streaming-only for a few hours until we figured it out.

I remember first hearing the term “success disaster” from Chuck Geschke, co-founder of Adobe and certified wonderful guy. I’d been there a few years working on Photoshop and Illustrator. This was back when the apps were just a small percent of Adobe’s business. Adobe made most of its money from licensing PostScript and selling laser printer schematics to other companies. The apps were taking off and Adobe was unprepared to deal with a big, sudden increase of flesh-and-blood customers. Phone support was understaffed, production and fulfillment channels were too small. It’s a success, and a disaster. Yay.

Wu & Co’s download success was our download disaster. What’s interesting about what happened yesterday is that the disaster was made a lot worse by a bug in my code, a really stupid, awesome bug. If the bug hadn’t been there the downloaders would’ve hummed through the day with no more than a lazy stretch. The bug’s been in production for at least six months, annoying the downloaders but staying under the radar until now.

“I work for a music publishing company in Australia, we have a number of artists who currently use bandcamp… I’ve been very impressed with the simplicity of design, flawless coding and speed of downloads. I think that you understand the way people consume music, better than most other companies.”

Flawless coding. Yeah.

This is about how not to generate a stable CRC. We have this very cool system in Bandcamp where tracks are turned into the format you’re downloading – mp3, aac, flac, etc. – on demand. If you’re the first person to download that new track in ogg vorbis, the downloader takes a small detour and makes it first. Once made, it caches it so that future ogg vorbis downloads happen immediately. There’s an aging algorithm in the cache that keeps active tracks around and garbage collects the inactive ones. In this way we’re able to run a site with hundreds of thousands of tracks and millions of downloads a week on way less hardware than you’d think.

In addition to generating formats on demand, the downloaders also check to see if the metadata in a track is up-to-date before starting a download. Lots of pieces of info get put into an mp3’s metadata: the track title and artist name, ’course, the track number, release date, album name, album artwork, lyrics (which is totally cool on an iPhone), etc. If any of this info was changed since the last time the track was downloaded, a metadata job is fired off to update the mp3 before the download begins. (we put metadata in all the downloadable formats, but I’ll use mp3s as the example here.)


Here’s the bug. For every file I store a signature of the metadata in our database. The signature’s a simple CRC32 of all the relevant metadata info. The info tends to be a couple hundred bytes so nothing fancier than a 32-bit integer CRC is necessary. Before every download I gather the current metadata info from the database – the track info, the info about the album the track’s on, the band’s info – generate a CRC and compare it to the CRC in the database. If they’re different, it’s time to update the mp3. And then the new CRC gets stored in the database to compare against in future downloads. Storing a CRC is tiny, efficient and much better than storing a snapshot of all the metadata info itself.

But darlin’. The CRC really needs to be stable in order to be useful. One’s CRC algorithm (not mentioning names) is useless if it generates a different number for the same source data. I call the CRC routine today and I get X. Tomorrow I get Y and X != Y even though the metadata hasn’t changed. That’s worse than useless because we’re depending on the CRC to reduce the number of times we have to update the metadata of every track that’s downloaded. If we get this wrong, we may underestimate the number of machines we need in our system dedicated to doing the metadata jobs. They may be underpowered for a large rush of downloads. They may get backed up and downloads may pile up and time out. And then Ethan may have to spend 6 1/2 hours answering support emails. Just sayin’.

Here’s how I did it. I have a routine that collects all the metadata used in an mp3 and puts it into an array of arrays, something like this:

['TIT2', {:encoding => 1, :text => 'Do I Move You?'}],
['TPE1', {:encoding => 1, :text => 'Nina Simone'}],
['USLT', {:encoding => 1, :text => '...', :lang => 'eng'}],

The first element of each array, for example ‘TIT2’, is the official ID3 metadata key (see and the second element is the metadata itself, for example the track’s title. This array of arrays is handed to a routine that does the (ridiculously hard) work of updating an mp3’s innards with the new metadata.

This blob of track data is also given directly to the CRC routine. Makes sense, right? The CRC is dependent on all the metadata that’s going into the mp3, so just use the same data blob that’s actually used to set the metadata. I simply run a CRC32 algorithm right on the array:

return Zlib.crc32(Marshal.dump(metadata_array))

The stoopid stoopid stoopid part that I’m embarrassed I overlooked is that hashes are not guaranteed to be read in the same order that they are created. If you do this:

h = {:a => 1, :b => 2}
puts h.inspect

You might get this:

{:a => 1, :b => 2}

or this:

{:b => 2, :a => 1}

Depending on the process’s mood. It’s not a mistake. Hashes are unordered sets of data. It almost never matters what order the keys are in; if you want the value of :x you’ll always get 1 and with :y you’ll always get 2. If you want an ordered set then you use an array. Duh. In practice the order of items in a hash will actually be the same as long as the program is running. But if you stop the program and run it again, you might get a different order. (I haven’t looked at the implementation but I imagine it has to do with the values of object ids. Doesn’t Ruby 1.9 do something different?)

So in other words, because the metadata blob given to the CRC routine contains hashes, and because the order of items in a hash is not fixed, the generated CRC will be different from time to time. Run the program once you get X. Run it again, the exact same code, you get Y.

Our downloader processes have great uptime. They’re rarely restarted. So for a given downloader the CRC will be stable. What’s humorous about what happened yesterday is that the downloads were being handled simultaneously by the two downloader machines morbo and linda. Linda would get a request for the Wu’s album, calculate CRCs for every track (and there are 27 of them), update the mp3s, cache those files and save the new CRCs in the database. Subsequent download requests to linda would match and the cached files would be used. Meanwhile morbo was doing the same thing, coming up with its own CRCs, updating the mp3s again, re-caching the files and updating the database. Basically morbo and linda were fighting over the “correct” CRCs, disagreeing in their silicon way, and generating tons of metadata update jobs and file I/O. Oh linda, how cute. You got the CRC wrong. I’ll fix it for you. Morbo you zoo creature! Take that! 27 tracks times mp3-320, mp3-v0, mp3-v2, etc. times thousands of downloads per hour. The metadata jobs backed up and the cache disk write I/O went to hell. Downloads failed. Ethan emailed.

Of course when I first saw these thousands of metadata jobs fly by in the logs my honed engineering intuition led me to blame someone else. I figured the artist was sitting at his site madly tweaking the release date in some ambien ADHD fit. We actually sent him an email saying Lay off the edits! In retrospect I had seen clues of this in logs over the months. I remember noticing that one track had been updated more than a thousand times and thinking what a dick.

I had the fix for this in half an hour. Five lines of code that temporarily converted the hashes to arrays and sorted them by key, ensuring that the order was predictable and stable across all processes.

['TIT2', [[:encoding, 1], [:text, 'Do I Move You?']] ],
['TPE1', [[:encoding, 1], [:text, 'Nina Simone']] ],
['USLT', [[:encoding, 1], [:lang, 'eng'], [:text, '...']] ],

(Note how :lang now comes before :text.)

We pushed this fix to our downloaders a few hours after the success disaster hit and later re-enabled downloads for Wu vs. B. Everything’s humming along fine. When we restart a downloader we have to wait until all of its current downloads dry up. In a glorious final middle finger to me for my grievous sins, linda happened to be in the middle of incorrectly calculating the CRCs and updating the metadata for every track of a fucking song a day for an entire year album. Yeah, you want me to restart? FU. (Righteous album, dubs.)