On testers and testing

Over the years, I’ve come to hold some strong. opinions on testing, testers and the entire business of quality assurance. Inspired by this post on Facebook’s testing, I wanted to write this down so I can point people to it. Some of this is controversial. In fact, even mentioning some of this in conversation has caused people to flip the bozo bit on me.

  1. Most product teams don’t need a separate testing role. And even if they do, the ratio of full time dev:full time test should be in the order of >20:1. For evidence, look no further than some of the most successful engineering teams of all time. Whether it be the Facebook of today or the original NT team from 30 years ago, some great software products have come from teams with no or little testers.

  2. Developers need to test their own code. Period. The actual mechanism is unimportant. It could be unit tests, full fledged automated tests or button mashing manually or some combination thereof. If your developers can’t/won’t or somehow think it’s ‘beneath them’, you need better developers.

  3. Here’s a politically incorrect thing to say. Several large engineering organizations start hiring testers from the pool of folks who couldn’t cut it as developers. I’ve been in/heard of a scary number of dev interviews where somebody has said “(S)he is not good enough to be a dev. Maybe a test role?”. This leads to the widely held but rarely spoken of perception that someone in test is not as smart as someone in dev. And due to the widespread nature of #3, the few insanely smart people who are actually genuinely passionate about quality and testing get a raw deal. I know this since I worked with a few of them.

  4. Tracking code coverage is dangerous. Since it is so easily measurable, it often becomes a proxy for the actual goal - building quality software. If your dev team is spending their cycles writing silly tests to hit every rarely used code path - just because it shows up in some status report - you have a problem. Code coverage is one of many signals that needs to be used but since it is so seductively numeric in nature, it drowns out many others. It falls victim to Goodhart’s law.

  5. I’m yet to see well-written test code in organizations that have testers separate from developers. Sadly, this is accepted as a fact of life when it doesn’t need to be.

  6. Just like metrics like code coverage are overused, other quality signals are underused. Signals like - how many support emails has this generated? Actually using the product all the time yourself and detecting issues. Analyzing logs from production and customer installations. All the other tactics mentioned in the Facebook post at the top.

  7. A common tactic by engineering leaders is to try and reduce the size of their testing orgs by moving to automated testing. This is a huge mistake. If you have a user-facing product, you absolutely need manual eyes on it. Sit someone down from the Windows organization at Microsoft for coffee and they’ll blame the focus on automated testing for a lot of the issues in Windows Vista. The mistake here is assuming that you need a full-tim test person to use the product.

Some disclaimers

# 07 Jan 2012 Comments

Photo Data

I have over 30GB of photos collected over many years. Most of them were taken by me, some by Aarthi, some by other professional photographers (my wedding album). But that’s just the photos I know of. While doing some photo backup, I realized I really didn’t know what I had in my ever growing photo directory. So like any other self-respecting red-blooded geek, I saw only sane option. Write some hacked-together code which will make for a fun blog post and will never be looked at ever again.

Right.

EXIF, stage right

I needed to grab the EXIF data out of these images. A little bit of Googling and StackOverflow-ing later, I had found the right Python Imaging Library incantation to read EXIF data out.

However, it turns out camera manufacturers like to use their creativity in using EXIF data. I came across several weird numeric fields, malformed data, files that made PIL choke, etc. In the end, I came up with a sane list of EXIF tags and then hacked together a script to walk through my tree of photos and dump it into a CSV file.

You can find my script here if you’re interested. And here’s the data from my photo collection - a 3mb CSV.

Next up, time to break out R. Unfortunately, I’m as good at R as I am at Parkour. Basically, I have no idea what I’m doing. However, getting the basics right proved to be trivial. I’m probably getting things wildly wrong below so please do correct me. My secret wish is that someone uses the data to do much more interesting things with it.

The one where I pretend to know statistics

First, I loaded the data into a data frame in R. The latin1 encoding below takes care of all the character mapping weirdness you expect with raw data from the wild.

> data <- read.csv("~/code/photostats/exifdata.csv", fileEncoding = "latin1")

After that, you can use the nifty summary() function to get a good overview of your data. The output of summary(data) is too big to paste here - you can find it here.

Now we can play with the data and try and answer some questions which I’m going to make up for the sake of justifying the work put in so far. For example, what are all the pieces of software that have mucked with my photos over the years? There was a zoo - from well known apps to weirdo firmware versions. You can get them by running the below command.

> unique(data$Software)

A far more interesting question - what camera do I like to take photos with? The answer is overwhelmingly obvious - my iPhones (see bottom right of table below ) with 1882 out of my ~9100 photos. However, I have over 1400 of unknown origin which I need to dig into further. I suspect that these were from my wedding photos.

> sort(summary(data$Model))

Pretty pictures

What I really wanted to dig into were my photo taking habits, about what drives me to take photos. For example, here’s a graph of when these photos have been taken. Surprisingly, I’ve stayed fairly consistent over the years with expected spikes at special events and trips(that spike in September 2010 being a huge collection of photos from my wedding). To make this gorgeous graph below (really, isn’t it breathtaking?), I first converted the dates into something R could understand and then made a histogram, breaking on every month.

dates = strptime(data$DateTime, "%Y:%m:%d %H:%M:%S")
hist(dates, "months", col="lightblue",freq=TRUE)

A similar interesting question is which day of the week do I find myself taking photos?

barplot(table(weekdays(dates)))

As expected, most of my photo taking happens over the weekends. I suspect my wedding photos (which happened on a weekend) skew this a bit and the distribution wouldn’t be so uneven without it.

Now, I’m obviously barely scratching the surface here. I’ve just started getting serious about my DSLR so in some time, I can compile some interesting stats on ISO, aperture, etc. And there’s far more behavioral data I can get out of it (what situations do I prefer my iPhone vs other cameras)? And then I can always throw OpenCV into the mix and figure out who/what is inside the photos I take. Now that should be real fun.

# 04 Dec 2011 Comments

Secure your browsing using a home VPN

This is one of those posts that is meant to save time for myself in the future when I’ll have to figure all of this out from scratch.

I spend a lot of time in coffee shops and public places with unsecure wifi. Unsecure wifi scares the bejeesus out of me so I wanted to figure out a way to secure any traffic going through. It would also be nice to access things on my home network. It turns out there are a million different ways to do this and I found one that worked for me. Here were the constraints that I imposed.

If you don’t have these constraints, there are many different ways to do this. Here are some alternate options

Alternate paths

But if you happen to have these specific set of constraints I do and like DIY-hacks, read on.

DO NOT SKIP - IMPORTANT - Security risks of using PPTP

VPNs can be created using a multitude of protocols and the one we are going to use, PPTP, is the most insecure of the lot. Wait, what? Why are we picking the most insecure one if the whole purpose of the exercise is to make internet usage more secure? Worse, by using something insecure, we could let somebody get into our home network and rampage around. If you’re not going to use a long passphrase/password, you shouldn’t be doing this.

Here’s why I picked PPTP and I believe using it with very long passwords/passphrases is acceptable.

  1. OpenVPN is the most secure solution, arguably but iOS doesn’t support it out of the box. iOS does support L2TP but DD-WRT doesn’t support that. So we’re stuck with PPTP. If you’re willing to run a server at home, you should be using L2TP.
  2. PPTP’s security increases when using long passwords. The security attacks are typically dictionary based. So make sure you use a long password.
  3. And finally, chances are low that an attacker at a public wifi station is going to put in the effort to go after you. If that isn’t true, you’re in trouble.

If you don’t understand what I’m talking about or if you don’t agree, you shouldn’t be doing this.

END SECURITY SECTION

Setting up your DD-WRT wifi router as a VPN server

Hit ‘Apply Settings’.


	#!/bin/sh
	echo "nopcomp" >> /tmp/pptpd/options.pptpd
	echo "noaccomp" >> /tmp/pptpd/options.pptpd
	kill `ps | grep pptp | cut -d ' ' -f 1`
	pptpd -c /tmp/pptpd/pptpd.conf -o /tmp/pptpd/options.pptpd



	sed -i -e 's/mppe .*/mppe required,stateless/' /tmp/pptpd/options.pptpd

Setting up DynDNS

The next step is to get this accessible from anywhere in the world. DD-WRT has built-in support for DynDNS which makes this easy.

Setting up OSX as a VPN Client

At this point, you should have a functional VPN server. Let’s connect to it! I’m going to lay out the instructions for OSX and since Apple uses the same terminology, iOS setup is almost identical from inside the General Settings->Network UI. All other VPN clients should have a similar configuration experience as well.

If this didn’t work

There are several things that could go wrong above. Here are some common debugging tasks

Happy VPNing!

# 07 Aug 2011 Comments

Lessons learnt from shipping APIs for Microsoft's cloud platform

I was talking to someone at work about REST APIs and thought it’d be good to write up some of my experiences from designing and running the team that shipped the Windows Azure Service Management APIs. There are some other great ‘lessons learnt’ posts out there, like this one from Foursquare and this one from Daniel Jacobson of Netflix.

Disclaimer: These are my views alone. I don’t even think the rest of the team agrees and I know for a fact that some of these opinions are quite controversial.

Some background

The Windows Azure Service Management API (or SMAPI) was built in 2008 to essentially let developers control all aspects of their Windows Azure account - from deploying new code, changing configuration, scaling up and down services, managing storage accounts, etc. Here are the docs. I can’t talk about the numbers but it is safe to say that this API is hit an incredibly high number of times each day, from all sorts of clients - IDEs, developer tools, automated monitoring systems, evil bots, you name it. I was lucky to work with an incredible team on this API (hi Zhe! hi Dipa! hi Frank!) and I’m proud of what we managed to build. Onto the lessons learnt.

REST conventions don’t matter…that much

I’m going to incur the wrath of many of my friends for saying this. When we started off designing the API, we spent weeks on perfecting the cleanest URL design possible. All of us had read Fielding’s thesis and it wasn’t uncommon for this book to be waved around in heated design meetings. But at the end, I came to an important realization - we were wasting our time.

The URI+HTTP verb+wire format doesn’t matter.

Here’s why. None of your developers will actually see the raw HTTP request+response. If your API is going to be the least bit successful, it has to have language specific bindings either built by you or the community. Sure, it definitely comes in handy to be able to compose API requests in curl in your sleep and you’ll definitely make your bindings developers’s lives easier. And if you blatantly ignore some basic REST idioms (like making GETs non-idempotent) you will break your API in all sorts of nasty ways. But if you’re spending every other meeting arguing over whether something should be a PUT or a POST for days or weeks on end, you’re wasting your time and the discussion is academic.

Evidence in point - look at the AWS APIs (not S3 but the rest). All of them are the ugliest RPC-style, verbose XML beasts you can find. But you can’t argue with how insanely successful they’ve been, can you?

Start with how the API is going to be used, work backwards

One of the best things we did while designing the API was to start with how the code using the API would look like and then work backwards to the REST definitions. Since the API didn’t exist yet, we just wrote pseudo-code in different languages for various important scenarios. This gave us a great feel and pointed out some obvious flaws which weren’t visible from just reading the API specification . For example, we found out that we were forcing developers to construct too many intermediate objects in-between successive calls to our API. We also found that developers couldn’t send back the same data structures they had retrieved from the API without modifying it. Without the pseudo-code, we probably wouldn’t have found these flaws until deep into our implementation where any change would have incurred a lot more cost. Or worse, we could have shipped the API like this and be stuck with this behavior for a long time.

We made it a rule that no API function could be reviewed unless it had some pseduo-code next to it. This also had the side effect of greatly improving API design meetings when people could see concrete usage patterns instead of a list of objects and functions with parameters.

A terrible way to design an API is to start with your internal systems first, figure out a REST syntax that maps reasonably to it and then throw that over the wall to your users. There are many of these APIs floating around, from Microsoft and other companies, and these almost unerringly cause developers a lot of pain.

Reduce the cognitive load

Be minimalistic in the number of ‘things’ your API exposes. Each concept your API exposes carries a significant cognitive load and dramatically increases the learning curve for your users. We were brutal in trimming the number of concepts we exposed, even when we risked merging slightly different types of ‘things’ into one ‘thing’. We were also brutal in trimming the operations we supported on each ‘thing’. All of this went a long way in making the API simpler than when it started out.

Make sure key actions are simple and fast, prefer chunky over chatty

One of the issues we hit very late in our API design was what is commonly called the ‘N+1’ problem. This is where someone queries a parent object to find a list of children and then issues a separate HTTP request to access each child. In our case, the single most common operation was accessing the list of services in Windows Azure and querying each of them to see what their status is. As we were very close to shipping, we didn’t have time to go rework our design to work around this so we put in what I thought at the time was a giant kludge - a ‘recurse’ parameter on the parent which expands all the children.

This surprisingly turned out to be very efficient and wound up making both devs and our servers a lot happier. The other feature we looked at was how to do partial responses, something GData now has support for. In each of these cases, the actual implementation wasn’t the cleanest (I tacked on a query parameter) but identifying a key usage scenario and optimizing it proved to be invaluable. I wouldn’t be surprised if this is saving millions of requests each day to our API services. More importantly, it makes clients faster and easier to implement.

One good forcing function to make this happen is to build prototypes of API clients as you build the API. For example, we were maintaining clients that pushed builds, monitoring clients and prototypes that simulated other common scenarios. It is amazing how obviously bad an API becomes as soon as you write some code to consume it. As a designer, you’re better off making sure you’re the one discovering it and not your users.

Measure and log everything

Something we were very good about is measuring things. We instrumented our API to the wazoo and tracked those numbers keenly. Everyday, we knew what users were hitting, what the popular calls were, what the common errors were and so on. This gave us great insight into how users were actually using the API (we could tell how much API activity we had saved using the N+1 hack, for example). When we rolled out new functionality, we could tell how users were adopting it and take corrective measures if we were not seeing what we expected. And probably most important of all, we could see what errors users were hitting and dig into those. We often changed or added error messages to make it clearer to users why they were seeing something based on this data. All this went a long way in increasing the usability of the API.

Versioning and extensibility

APIs change in unanticipated ways. Some of these changes are small - you want to add a new property to some data structure or add a new item to a list of items. Some are bigger - you want to change the authentication mechanism or drop support for existing APIs. For either of these, you’ll be thanking yourself if you future-proof your API from the beginning.

There are many ways to version your API to protect yourself. You could do what we did - the client sends a version header and the server shows the client behavior it expects. Or you could just add a version number to your path. In any case, just baking this into the first version of your API will save yourself a lot of heartburn down the road.

The other common change you want to design for is minor additions. Here, I think I could have done better with the Azure SMAPI. Something I like about the Foursquare API is the way they use generic structures and indirection (example - { type: “special”, item: { special: { ...specialdata... } } }). This lets them add new ‘types’ without breaking old clients which don’t anticipate those types. I wish we did something like this with the Windows Azure APIs. It would have made life much easier for us when we wanted to just add an item here or a list there without having to increase the API’s versioning number and breaking several clients.

# 21 Jul 2011 Comments

The publishing world

If you are a Harry Potter fan or even remotely interested in books, you’ve probably heard of J.K.Rowling’s decision to break into e-publishing. After several years of avoiding the e-book world (and rampant privacy), she has done something very interesting - forsake her print publishers and go it alone. A lot of main stream authors have dabbled in self-publishing - Stephen King wrote a short story years ago, for example. But J.K.Rowling is the first ‘superstar’ to go full fledged into self-publishing and ship her books from her website.

Some background

I’ve been digging into this space for the last few weeks for a variety of reasons. A bit of it is out of self-interest - I’ve been making use of my post-Microsoft vacation to pursue a long held dream and work on my first fiction book (a thriller, far from being complete, even further from being any good). Also, I’ve just been interested in how the publishing world works for a long time.

Backup a few years. For the last several decades, if you were a newbie fiction writer trying to get their first novel published, the process went something like the below.

‘The Process’ (as it used to be)

Bring on the revolution

All this changed due to two key developments, both caused by Amazon.

All of a sudden, writers could self-publish and not have to go through any of the gate-keepers. Just as importantly, they could now price their books at very low prices ($.99, $1.99) and rely on readers to make impulse purchases. It’s much easier to pull the trigger on a $1.99 thriller than a $9.99 thriller. It was a classic low-price, high-volume strategy. The publishing houses couldn’t match these prices or these royalty figures of course, since they had their own costs to worry about. Since writers need editors, cover-art, etc, a little cottage industry has sprung up where you can hire an editor, somebody to do graphic art, etc for low prices. A lot of these are still nascent (which is why most self-published books have terrible cover designs) but it doesn’t take a genius to see how these might evolve.

Also, the self-publishing world has been seeing their first breakout stars and big writers moving into self-publishing. There are three which are interesting, for different reasons.

If you’re interested in self-publishing, J.A.Konrath’s blog is a must-read.

How a big house can help you

The publishing world realizes all this of course. However, it is very unclear to me as what they intend to do about it.

There are still several things that only traditional publishers can do. Publishers throw in editing/graphics as part of the deal - you now have to find external people to do it for you. A good editor is invaluable and to be protected and cherished. Only traditional publishers can get you print distribution at scale. This counts since a lot of people still buy only books (I still prefer my books in the paper form). You still can’t get a review in the NYT or People magazine if you’re self-published, even if you’ve sold over a million copies. You still can’t get into Barnes & Noble or Borders or a small book store if you’re self-published. That robs of you precious real estate in book displays, etc (which the publisher often pays for). You can’t get other benefits - associations demand that you be published traditionally, you may not qualify for most writing awards, etc. You may not even be able to get your books into libraries. Of course, every author has his or her own priorities as far as these are concerned.

The biggest downside to self-publishing might be the marketing muscle you don’t get. Marketing yourself is hard and unlike John Locke, most people are not good at it and don’t want to do it. Especially writers, who often tend to be shy creatures who like the privacy of their dens. Selling well with self-publishing means constantly promoting yourself and putting yourself out there and not having a professional PR person line up tours/appearances for you. This is very hard work and work which most people don’t know how to do.

Apart from all this, there’s still the perception problem. Self-published long meant “Not good enough to be published” and that perception still holds for a lot of people. You will get snide remarks and get looked down upon by a lot of people. Whether that matters to you or not is completely up to you of course.

Make no mistake, traditional publishing is still the established, mainstream way of being a writer.

And the agents

The biggest people at risk right now are the traditional middlemen - the agents. They’re not taking this lying down however. Here’s a post from Rachelle Gardner, who runs a well-known blog on her life as an agent and the publishing business in general. I really like Rachelle’s posts in general but the below made me shake my head sadly. Quoting her

“With no more gatekeepers, no more exclusivity, no more requirement to actually write a good book, won’t published books lose value? If anybody can get a book published, doesn’t that diminish the perceived status of all authors?…Well, I have news for you. If you think the published books are bad now, just wait until self-pubbing becomes the norm. Holy cow. Folks, you don’t see an agent’s daily slush pile. Sure, some of it is good. But let me tell you. At least half of it is seriously not good. As I look at all the books I say “no” to, and then realize these books could be for sale within a matter of months, I get depressed.”

If you’re in the tech world, you probably see how wrong she is too. The AppStore and probably the web is an example of how this model will work. In fact, that’s the beauty of it - that anyone sitting in his pajamas can get their content out there immediately. And the best content will always rise to the top since people will find it and bubble it up.

If you see the comments, you can see how (some) traditional writers are now having to deal with their world upended. Some love the changes and are jumping onto the bandwagon. Some see it but still want their books published by a traditional house. Some of them have spent years (or decades) in the hopes of being published and now all of a sudden, that prize doesn’t seem as valuable. Here’s a sample comment from the extreme end of the opinion spectrum.

“I think this idea that “everyone deserves to get their book published” is fallacious and insulting to both the good, hard-working writers and more importantly to the readers.Everyone can open their mouth and make a noise. Not all of us deserve to stand in the Royal Opera House and sing to a paying public.”

Oh boy.

My thoughts

I have a jumble of thoughts on this whole space. From the top of my head

Now, excuse me since I have some unread books beckoning me.

# 23 Jun 2011 Comments