Fundamentalism: TDD and Functional Programming

The big news this week is that @dhh thinks TDD is bad. Oh my gosh let’s all shake our fists in the air, “He’s so wrong! I love me some TDD!” And I do like tdd, I even wrote a gem called tdd.

But I think the point @dhh is trying to make is following a set of principles without thought can lead to bad outcomes. This I agree with. This is fundamentalism. When we examine other’s fundamentalist attitudes they seem silly, this is the bread and butter of the Daily Show. But when the camera is turned on us, it’s a different story. Some will defend their beliefs to the end.

Testing Isn’t Dead

TDD isn’t dead, nor is testing. The allure of “test first development” is that you suss out a great design and end up with tests. But for me in practice it’s just allure. It ends up being really hard and I start drawing out architecture which doesn’t exist instead of writing code that works. My friend who owns a web development agency said that if they did everything test first, he’d be out of business in 10 days. Their main goal is to ship mostly working software so their customer can play with it, make changes, and move forward. At this point they add tests on the stuff they don’t scrap from the prototype.

I think this is important: you can’t quickly get to a prototype when you unit test the shit out of everything because it makes it really hard to change anything without tediously changing your tests.

Pragmatic Not Dogmatic Testing

Not unit testing the crap out of everything is hard, and it takes time to learn how not to do it. I’ve seen junior devs submit code for review and say, “Boom done, tests and everything!” and they have 5 times the amount of test code as they do actual code. When requirements change the hard part is changing the tests, not the actual code. You have to pragmatic about what to test instead of blindly following TDD principles. 

Some questions I ask myself to avoid TDD traps are:

• How do I know when this code is done?
• Can I write a test that tells me the code is ready to ship?
• Can I avoid testing the framework or boilerplate code?
• Will I be able to sleep at night knowing this code is in production?
• What if I had to change the behavior of this class, how hard would my tests be to change?

Curse of the Middle Path

I started reading “The Curse of the Excluded Middle” by Erik Meijer which introduces the idea that sort of doing functional programming is not making your life any better. You must, in his mind, not take a middle path and be a fundamentalist. The world must be black and white. 

The world, as I see it, is mostly grey. There are few circumstances where things are clear cut, where the decision just makes itself apparent. Meijer argues you must be:

(a) accepting that programming is ultimately about mutating state and other effects, but for pragmatic reasons tame effects as much as possible; or (b) abolishing all implicit imperative effects and making them fully explicit in the type system, but for pragmatic reasons allow occasional explicit effects to be suppressed.

As a Rubyist, option B sounds awful. It sounds like math. I like math just fine, but programming for me is about modeling the world around us, it’s about modeling social contracts. I pay you money, you give me things.

Rarely in my experience are things easily reduced to functions where the outcome is repeatedly the same. Even if it was, it’s hard for me to reason about the world that way. I’m a social creature who exists inside a large system (the world) which I cannot model in my head at all times. But I do understand relationships within context. I cannot boil down everything to simple functions. Modeling the entire world with functions is the dream of physicists everywhere “if we just knew the position and velocity of every particle in the universe we could predict everything!”

But we know this is not (yet) possible. Even if we could model the universe this way we couldn’t understand it. The simulation would be as complicated as the real thing. Fiercely simple functions at a large scale can act very strange. This is the basis of Chaos Theory. You’ve probably seen a representation of this in Conway’s Game of Life or Sierpinski’s triangle. To really see how far this rabbit hole goes check this out: http://www.oftenpaper.net/sierpinski.htm   

This is where I think OO has an advantage. It allows us mere humans the ability to reason about a part of a large system by understanding relationships. But, functional programming has a place too. We need to be certain, in some cases, the outcome is always the same and having to pick a side is a waste.

Scala Is a Middle Path

I recently took a class on Scala. We use Scala at work and I made mention I’d like to know how it all works. Holy options. There’s at least three ways to achieve the same thing in Scala both in the object oriented way and in the functional way. Scala is really terrible to read and understand later, but damn, you sure feel clever writing code which is going to take someone, likely yourself, a lot of time to decode in the future. It’s this weird mix of OO and FP, I think they get a lot of things right but I can be really hard to understand the FP side of things.

And I think this where Meijer is drawing a line in the sand. He’s saying, you can only see the world in two ways, OO or PURELY FUCKING FUNCTIONAL. If you try to do both, you’ll end up with a mess. And this is the same as people saying that TDD IS THE ONE TRUE WAY. It’s not. People create kick ass projects writing some tests, or testing after the fact. Some people make kick ass products in Scala (like uh… Twitter). But you, the user of these ideas, techniques, or frameworks are left with the careful task of not DoingBadThings™. There is no one true way which buys you both safety from DoingBadThings™ and a kickass product, but maybe we can learn from these fundamentalist crazy people.

Confronting Fundamentalism

When confronted with fundamentalism you have to ask yourself a few questions:

1. Is this fundamentalist approach going to help me achieve my goals?
2. Is there a middle path that is easier?
3. What can I learn from these far flung crazies?

Often I find that fundamentalists may have some merit in their ideas. They must think their idea is awesome and have had some success with it to shout it from the rooftops (but cognitive bias is a strange animal). It’s up to us choose the right path. This is challenging because it’s easy to say “Well that dude over there told me there were answers so I just started doing what he told me.” It’s a struggle to evaluate the good and bad of a new way of doing something or a new way of thinking about things, but the outcomes are real and tangible. I wouldn’t be where I am today without picking up Ruby and Rails and following some of its dogma. I also wouldn’t be where I am without picking up some dogma about testing. I’m now considering how I can be more successful with some functional programming in my life.

We are adverse to failure, but I urge you to pick up some ideas, try them out, and throw them away when they stop working.

Building an Internal Gem Infrastructure: A Crash Course

I gave this short talk as part of RailsConf 2014. LivingSocial graciously sponsored my trip. We’re hiring awesome engineers if you didn’t already know. Hit me up on twitter @ubermajestix if you’re interested in working with a great team.

Being able to build, release, and host your own gems is really helpful. This post will introduce the basics of doing all three of these things. This is more of a motivation post as there are tutorials out there with more depth and step by step instruction than I will provide here.

Why

Moving code into gems really helps when you’re splitting up monolithic apps or building a service oriented architecture. Sharing code with gems can reduce copy/paste which reduces the mayhem maintenance costs it imposes.

Building a gem infrastructure can help foster an “internal open source” culture where anyone can contribute. Gems get their own
repo and readme. You get to reap all the benefits of pull requests and open contribution within your company.

We’ve heard experts say “go ahead, make a mess” but it’s often hard to figure out where to put your mess in a Rails project. A gem is a great place to make a mess of small classes with single responsibilities. Putting your code into a gem forces you to draw boundaries around your code. You get your own tests! You can separate concerns.

There are a bunch of benefits to isolating your code. First, your code is really easy to delete when it’s no longer needed. You don’t even have to delete your gem, you can just remove the calls to your gem from other gems or apps that use it; its documentation and tests live on. It might even be resurrected in the future. This is so much easier than searching through old commits for “that code that did that one thing”.

Also, you have to think about how your code will interact with Rails. Does it need database access? Where should logs go? Do you even need Rails for this code to do its job? This has helped me constrain my code to its simplest form, use the smallest possible pieces of Rails, mock and stub boundaries in tests, and write good docs.

An Example

I recently was tasked with automating the generation of a daily report which needed to be uploaded to a third party’s FTP. The processing results are then emailed back to us as a plain text file. Sometimes I need to parse the email if anything went wrong. As a coworker said, 1997 called, it wants its architecture back. But those are the constraints we have to work in.

I could have plunked down the CSV and FTP code right in the database model, maybe thrown the email parsing code in a helper,  and called it a day. We all know that’s gross. We’ve all done it and justified it, “Gets the job done right? Fat models bro!”

But you’ve also been on the other side a week later, a month later, or even years later, “Ah man, why is all this code here?” and “Do we even use this anymore?”, “What does this even do?”

So I decided to cut a gem, and put all this domain specific code in one place. It’s a pretty simple gem, and that’s ok. It’s easy to describe and document, but it also let us have the conversation of where the code should be run from, instead of just throwing it in the monolith and moving on.

Building A Gem

Ok this is a crash course inside the crash course on bulding your own gem. This may be review for some of you or it may be new to you. If you haven’t written a gem, I highly recommend trying as it’s a great addition to your Ruby tool belt.

There are a bunch of ways to build a gem, but the easiest one I’ve used is Bundler’s gem command. The best part is it’s probably already installed on your machine. Just run:

bundle gem ls-super_awesomeness

Here’s what it creates for you:

• The most important file is the gemspec there at the bottom. Without that file this isn’t a gem. You can readup on the details online here and here, but the version Bundler generates is pretty straight forward.
• A Gemfile
• A License (MIT)
• A README
• A Rakefile which will help you build, install, and release
• Your code goes in lib
• If your namespacing you’ll have a namespace directory with your files in it
• Finally your version file deep down in there.

Namespacing is something I think is really important and bundler will help you do this. Here’s how it works. You just shorten your organization’s name into two letters. It can be more or less, but I’ve always seen two characters in the wild. It’s important because it tells your reader that this is internal code and not an open source implementation.

Ok, so now a little bit about versioning, again this might be a refresher, but I feel like I explain this more than I have to. We liberally use the semantic versioning spec to control version numbers at LivingSocial. Checkout semver.org for the full spec.

The bugfix number is the number furthest to the left, it should be incremented when bugs are fixed or really small changes happen that don’t affect your public API. The minor version number is the middle number and should be incremented when you’ve added a new feature or additional functionality but your haven’t removed or broken the public API. Consumers of your code will not be negatively affected by upgrading. The major version number should be incremented when a breaking change is made. This usually means you’ve removed some functionality, extracted it to another gem, or have changed the public API. You should have a very good reason for breaking the public API as all consumers are affected and will have to upgrade their code. Think about how hard it is to upgrade from Rails 2 to 3 and from 3 to 4.

If a gem is used in production, is stable and reliable, then it should be at 1.0.0. I’m wary of deploying gems that aren’t 1.0.0, if you’re in production and the API is stable and your version is 0.0.1237, you’re doing it wrong.

Letting The World Behold Your Awesomeness

Ok so you’ve built your gem and got your version number figured out. Now, we need to let the world behold your awesomeness, or at least others in your company.

It should be EASY for anyone to release their code. Having a bunch of commands to release code is messy and error prone, even just three commands.If you use bundler to create your gem, `rake release` will do all this for you but we’ll need to modify its behavior to so you…

But be careful! You can accidentally open source your code. It happens! You might have sensitive information in your gem and there are bots that mirror gems as they hit rubygems.org, so once that code is out there, it’s not coming back.

It should also be easy to add a version tag in git. This helps people figure out what commits went into a release and Github makes version tags really easy to use. Bundler’s `rake release` task will use the version number from your VERSION constant in your version.rb file, so make sure you update your version number before you release.

So how do we get `rake release` to not open source your code?

We monkeypatch subclass bundler’s gem_tasks. If you look in the Rakefile bundler generated for you, you’ll see require ‘bundler/gem_tasks’. This is how you get the build, install, and release rake tasks. At LivingSocial is we built our own gem that all other gems use to get their default rake tasks. If you subclass bundler you can add your own rake tasks and most importantly keep developers from pushing their code to the wrong place and help them push their code to the right place.

Here’s the magic code. Notice how we disable the rubygems push and raise an error just in case. Then, we add a method that pushes to our geminabox server. From here we just replace the bundler/gem_tasks with your customized gem_tasks in your Rakefiles. Boom, releasing and tagging made easy.

Hosting Your Gems

Now you’ll need a place to store all these gems. Here are the options that I would recommend you use.

We use geminabox at LivingSocial. It gets the job done. It hosts our gems and provides a web interface to manage and view gems. It will support authentication through custom middleware (their wiki page has examples). It can be setup to do pull through mirroring of rubygems. Mirroring is useful when you don’t want to depend on rubygems.org when you deploy and you don’t want to vendor all your gems. It also provides a CLI by patching your system gem command with inabox. You can use that command to push gems and configure your machine to talk to your geminabox instance.

I’ve also used stickler at a previous company. It supports hosting and has a nice web interface. It supports auth in a similar way to geminabox. It will mirror gems from rubygems.org but you have to tell it which gems to mirror, but it does support mirroring everything in your Gemfile.lock. It has a really nice CLI and the author wanted me to mention that it will soon support the new bundler API. There’s a great blog post about setting it up over at copiousfreetime’s site.

Both stickler and geminabox are pretty easy to deploy and have lots of documentation online to help you out. In my experience stickler is faster and more reliable than geminabox, but your mileage may vary.

If you want to go the hosted route, Gemfury.com appears to be the only solution out there. It supports auth, but doesn’t appear to support mirroring. It comes with a CLI and also supports nodejs and python packages in addition to ruby gems.

There you have it, a crash course with almost everything you need to setup a gem infrastructure for your company.

Rails 3 Memcached Session Store

Couldn’t be simpler in Rails 3.

Dalli is the new standard gem for dealing with memcached.

Gone are the days of “memcache, memcache-client, memcached” clusterfuck. Just use Dalli.

All theses steps are outlined in the Dalli README.

So if you don’t want to read the README here you go:
1. drop gem "dalli" in your Gemfile
2. In config/application.rb or config/environments/*.rb drop in config.cache_store = :dalli
3. In config/initializers/session_store.rb put the following:
require 'action_dispatch/middleware/session/dalli_store'
Rails.application.config.session_store :dalli_store, :memcache_server => ['host1', 'host2'], :namespace => 'sessions', :key => '_foundation_session', :expire_after => 30.minutes


Show last two directories of pwd in BASH prompt

Problem: Show current directory and one above it in bash prompt.

finished bash prompt

So ok googling for a few hours produced a failed attempt at using PROMPT_DIRTRIM=2 (which only works on bash v.4 which is not installed on OS X, you can get it from macports so i wish you good luck)

Found a post here that looked like it might have worked… nope. Then I looked this post which was sort of unrealated but there was a useful comment from swobodin which i used to get this to work.

The function is:

function last_two_dirs {
pwd |rev| awk -F / '{print $1,$2}' | rev | sed s_\ _/_
}

So I dropped that in my ~/.bash_profile and I call the function from my export PS1 line (with colors and git functions too):

export PS1='\[33[0;32m\]$(date +%H:%M:%S) \[33[0;36m\]$(last_two_dirs)\[33[0m\]$(__git_ps1)\[33[0;33m\]$(git_dirty_flag)\[33[0m\] ➡  '

MacPorts MySQL + MySQL Gem Install Protips

We moved off of MySQL at work and I hosed my machine, so I never reinstalled mysql.

The I started working on the upgrade to Rails 2.3 for Rentmappr.com and I realized I needed mysql.

So you know, port install that shit right?

sudo port install mysql5 mysql5-server mysql5-devel

Ok that took forever boo. Yay for PostgreSQL’s quick port and gem install! I suggest moving big projects to that if you can and small ones to sqlite, ditch mysql people its not really worth it.

So now, we need the mysql gem. Gem install that shit.

sudo gem install mysql


uh fuck...

uh fuck…

So I need to pass it some flags because MacPorts installs everything in /opt

Uh ok… so let’s try –with-opt-dir

almost...

Closer but no cigar.

Now what? Google!

Finally googled for “macports mysql gem split TrueClass” and found the solution:

sudo gem install mysql -- --with-mysql-include=/opt/local/include/mysql5 --with-mysql-lib=/opt/local/lib/mysql5 --with-mysql-config=/opt/local/lib/mysql5/bin/mysql_config

credit: http://codeintensity.blogspot.com/2007/09/installing-ruby-mysql-gem-with-macports.html

Yay!

fucking finally!

Engineering Goals

I got a lot of good stuff out of going to Mountain West Ruby Conference this weekend. Sitting here this morning I’ve come up with some goals I have for myself and my engineering team.

1. Excel at usability
   • This is more a personal goal than anything
   • But its important for us to get right
   • If people can’t fully use our software they find shortcuts and often break things
2. Improve cross team understanding
   • More pair programming – if we can each better understand everyone’s code, we can make sure we are doing things correctly, in a sane way, and we can better maintain each others code
   • I have learned so much from just spending a few hours looking over other team members shoulders. This helps me come up with new ideas and I better understand our complex code base and algorithms
3. Improve testing
   • We have really great coverage on our models, i.e. we’ve unit tested the shit out of our code (mostly). However, these tests are way to close to the metal, we need a higher level set of tests (using something like Cucumber and WebRat). We need to know exactly how new features will affect the entire code base from a very high level.
   • This will help in organizing new features, instead of a bunch of us in a room going, “uh, well, uh I think that shouldn’t break anything”, we can say, first, “why?” and then second “this feature will break these X number of things and will take X number of hours to complete”
4. More Instrumentation
   • how much load can our systems handle?
   • where are our bottlenecks and what pieces of infrastructure need to be retuned/replaced/recoded?
   • who actually uses each product?
   • this will also help us say either “yes, we can add more load, more data, new features, etc…” or “no we really can’t add X feature until we’ve fixed X piece of code/infrastructure”

Helper Methods for render :update do |page|

@aneiro showed me this the other day and I just got it to work. So here’s what’s up.

Instead of having a metric crap-ton of rjs files I just started doing render :update do |page| in my controllers. This lets you do, as some people complain, javascript view logic in your controller. So I ended up with three actions that each call the same set of JS. Hide that thing, replace this div with a partial, show some stuff, fade some things, change the css class of that table row, etc…

So what you can do is this: define a method in your controller and put all the page.replace_html and page.hide calls in it. Do it. Ok so then in your render :update block you just call that method and pass page to it.

One caveat is you have to actually define your new method as a helper method, which is super easy. Just call, in your controller, helper_method .

Check out the gist: http://gist.github.com/61454

Design By Management = Bad Idea

Management didn’t get to where they are by being good designers. They do not know what looks good and what looks bad. They do know, I hope, how to run a business.

You should leave complex things like typesetting, color theory page layout, etc to a designer. Management types should worry about getting their khakis to match their wingtips, and what color seats to put in the new Audi.

Steps for not letting management ruin your company’s image:

Step one: DO NOT LET THEM DESIGN THE WEBSITE

Step two: DO NOT LET THEM DESIGN POWERPOINT TEMPLATES

Step three: DO NOT LET THEM DESIGN LETTERHEAD OR LOGOS

Step four: DO NOT LET THEM DESIGN EMAIL SIGNATURES

Management likes to think they can do anything. Its America, anyone can do anything right? No. You are wrong. Management is good at telling people what to do and doing things like quarterly projections and attending bullshit conferences to speak on bullshit panels to other bullshit management people.

They are not designers. They are rarely creative. They rarely know how to even open Photoshop. They need to be told Microsoft Word is not professional design software. They need to be told to not use shitty fonts like Impact and Trebuchet.

If you can’t afford good design help, then fine, leave your image and brand to the wind. Its not like huge corporations ever got anywhere by thinking a lot about how things look and how other people perceive their company through visual mediums. Nope that never happened.

Don’t leave this page cause I said so dialog

I have a process where I create an object on one screen, and then in the next screen add some information to that object.

Problem is, if you navigate away from that second page I have an object that has no important info attached to it.

My process is a wizard approach to setting up a new account, which only really has a name, and then the account has users and billing info attached to it in separate models/tables. So give it a name and save it, redirect to the next screen and add users and billing.

So I want the “Hey you! Don’t navigate away” dialog to appear on all but a few links and form submits on the add users and billing page. Its actually super easy.

Take a look at this very nicely formated gist I would put this stuff in code tags and display it here but it looks like crap and I can’t embed gists unless I pay wordpress. And wordpress stripped all my erb and html tags so just go look at the gist.

TwitPlot.com my twitter testing ground

Going to spare you the obligatory 5 paragraph ‘I haven’t posted in months’ intro. So, my new project has been figuring out how to condense this business of twitter into something people can understand and find useful.

So I bought http://twitplot.com yesterday, grabbed the pretty well done twitter gem http://github.com/jnunemaker/twitter/ and started playing with the public search api.

What I’ve gotten out of it is a way to see what people are saying about any keyword (or idea really) anywhere in the world (or at least anywhere that I can geocode using a hacked version of GeoKit http://geokit.rubyforge.org/).

Some useful things I’ve done in the app that I’d like to share are how I’ve captured urls and user names in a “tweet” and added the markup so you can click on them. The regex was pretty easy, check it out:


def add_url_markup(text)
urls = []
users =[]
# find urls
text.split.each{|a| a=~/(http\:\/\/\S+)/; urls << $1}
# find twitter usernames (@someusername)
text.split.each{|a| a=~/(\@\S+)/; users << $1}
# add markup
users.compact.each{|user| text.gsub!(user, "#{user}")}
urls.compact.each{|url| text.gsub!(url, "#{url}")}
return text
end

view this code at gist.github.com

So check twitplot.com out play with it let me know what you think.