Programming by Wishful Thinking

This is a technique that I discovered several years ago while teaching an introductory programming course, and I was extremely pleased to find out that although it had already been discovered, I had given it the same name. Programming by wishful thinking is a style of top-down program design that helps you improve the structure of your code, isolate the hairy parts from the simple parts, think more clearly about data flow, and work faster. I find it particularly helpful for novice programmers who feel intimidated when starting a new project and aren’t sure how to get started, but I still practise it regularly as well.

So how does it work? I think this might be easiest to demonstrate by example. Let’s imagine that you are trying to write a Python program that checks the xkcd RSS feed and downloads new comics. In a bottom-up design, we can think about the different built-in functions of Python and its standard library that we could use. We’ll need to make a web request to the RSS feed, parse XML, download images, sounds pretty complicated. Wouldn’t it be great if Python had a few more built-ins, and we could just do this? 

items = fetch_rss('https://www.xkcd.com/rss.xml')
for item in items:
    src = get_img_from_item(item)
    download_image(src)

We’d be done! Although Python is a batteries included language, it unfortunately doesn’t include these functions. That’s okay though, because we can just define each of these functions. Let’s start with fetch_rss. This is definitely going to involve a web request, so we should figure out which library to use. Then we’ll need an XML parser to convert the response into a list of items. Sounds pretty complicated, and I really just want to download some images. Can we come back to this later?

For now, let’s just fake it. At the time of publication, the xkcd RSS feed contains the following four comics: Models of the Atom, Missal of Silos, Magnetic Pole, and Thor Tools. So we know that if we were to actually implement fetch_rss and run it right now, those are the results we’d get. 

def fetch_rss(url):
    return [
        'https://imgs.xkcd.com/comics/models_of_the_atom.png',
        'https://imgs.xkcd.com/comics/missal_of_silos.png',
        'https://imgs.xkcd.com/comics/magnetic_pole.png',
        'https://imgs.xkcd.com/comics/thor_tools.png'
  ]

There we go. Good enough for now. Next up, get_img_from_item. This is supposed to extract from one of the items in the result of fetch_rss the URL of the image to download. Well that’s easy, because each item in the result already is that URL. 

def get_img_from_item(item):
    return item

The last function to implement is download_image. We actually care about this one working properly, because I actually want to view the images on my harddrive. We don’t have much choice but to implement it properly. We can use urllib.request.urlretrieve for this, but we need to specify the target filename. Let’s just use more wishful thinking and move on. 

from urllib.request import urlretrieve

def download_image(src):
    dst = generate_filename(src)
    urlretrieve(src, dst)

I said the previous function was the last function, but we’ve shifted the goalposts now by introducing yet another imaginary function. No problem, we can implement it too. For now, we’ll just stick the download into the working directory using whatever filename the source URL uses (get the last element in the path). Maybe in the future we’ll want something a bit better than this, but since we’ve tucked it away in its own function it’s completely isolated and we can easily come back and change it. 

def generate_filename(url):
    return url.split('/')[-1]

Put it all together in a single file (in the right order), run it, and sure enough I have four comics to view locally. From Make it Work, Make it Right, Make it Fast, we’ve made it work, now it’s time to make it right by going back and properly implementing the functions we skipped over. 

from urllib.request import urlopen
from xml.etree import ElementTree

def fetch_rss(url):
    src = urlopen(url).read()
    xml = ElementTree.fromstring(src)
    return xml.findall('channel/item')

By correctly implementing fetch_rss, we have changed the return value and thereby broken get_img_from_item, so we should implement it now too. 

def get_img_from_item(item):
    desc = item.find('description').text
    html = ElementTree.fromstring(desc)
    return html.get('src')

And now we’ve made it right. The approach we have just followed pretty accurately reflects the actual process I use when developing software. Here’s why.