Magical Mystery Talent Algo Tour
The Beatles’ album Magical Mystery Tour as it is known to US fans is not the album the rest of the world. Capitol Records feared that the EP format of the film soundtrack would not sell in the US given that EP’s were not common in America.
To solve this, they added a four more songs from earlier singles and release the album as a full LP to the displeasure of the Beatles. When asked about this, John Lennon said, “ It’s not an album, you see. It turned into an album over here, but it was just meant to be] the music from the film.”
That is often our experience with algorithms. They turn into something we do not expect. We go in thinking they do one thing, and the output is something we may not even recognize.
Algorithms seem mysterious to most people. For us developers though, creating algorithms is what we do everyday. The simplest definition would be a set of rules to follow in order to solve a problem. In this case, logic written in a programming language that instructs a computer to do something.
The problem though is that algorithms have an image problem. People not familiar with technology view algorithms as some black box, complex, highly mathematical thing created by mad geniuses with ill intent. When any incident involving computer errors is discussed in the news, like the Knight Capital trading fiasco or various stock market flash crashes, algorithms are the culprit and guilty as charged.
Algorithms are complex because the nature of the problems we are trying to solve are sometimes complex. This means the logic to solve the problem can be complex and the inputs and outputs can be unpredictable. Often times we have to make assumptions during development just to realize in our testing that our assumptions were radically off the mark.
This is why it was refreshing to read this post on the evolution of crafting an algorithm by a developer at Stack Overflow. He lays out in an amazingly detailed and clear way how the proverbial algorithm sausage gets made.
As an aside, most people have no idea how Stack Overflow stays in business. Developers use it without so much as a thought beyond getting an answer to a question. You may have an inkling as a regular reader of this blog, therefore probably are at least aware of the business side of Stack Overflow compared with 99.99% of the developer world. If you are curious however to know how Stack Overflow manages to keep the lights on, their Director of Architecture shared this in a post “How We Make Money at Stack Overflow”.
Which brings me to the post on the Stack Overflow algorithm that digs into the code driving the Talent business. Their engineering team has historically been radically transparent about many of the things they do, so walking through the steps of how they built the job matching algorithm was quite informative. To deliver something relevant to users of their Talent product was not some instant revelation, but a series of iterations, and failures, to get slightly better results each time.
This highlights two things. First is the commitment to delivering a quality product, even if it initially leads down the wrong path. The Teams Q&A platform was itself a failed experiment several years back. The lessons from that effort led to a much better product for organizations and integrated more easily with other systems. Second is that the job matching algorithms are pretty spot on through the various iterations and experiments. This is the biggest takeaway for Enterprise engineering teams, that experimentation is the only path towards building high quality products.
What have you done to cultivate a culture of experimentation on your team? How do you test and ensure your algorithms are doing what they should?
Why does ++[[]][+[]]+[+[]] return the string “10”?
A fascinating question exposing some of the more arcane aspects of Javascript.
We help IT leaders in enterprises solve the cultural challenges involved in digital transformation and move towards a community based culture that delivers innovation and customer value faster. Learn more about our work here.
