7 learnings over 7 years at Pinterest (Part I)

Pinterest is a company where I had the most growth throughout my career so far, for which I have so many people to be grateful for because they believed in me, mentored me and gave me the opportunities to grow. In this post, I will focus on the learnings about scaling engineering and technology.

Since this is a somewhat long post, I will break it to two parts. Today I will only share the first three learnings, and the rest four will be shared in next post. Stay tuned.

Before we get started, I’d like to make a couple of clarifications:

1. This is not a post about specific technology choices we made to build an infrastructure that serves hundreds of millions of pinners at Pinterest, but rather a set of guiding principles which I believe are super important to embrace when making those decisions.
2. The guiding principles in this post may seem too black-and-white. That is intended, because I want to make it more thought provoking and actually useful. One important realization of mine lately is that if you want to turn to your guiding principles to help make tough decisions, then they have to be more explicitly tipping towards one end of the spectrum. If they were well balanced, they might seem to be “more” correct, but they would be less useful in actually guiding you towards a decision in case of ambiguity.

I started at Pinterest back in July 2012, when Pinterest probably had less than 20 engineers, with most of the engineering team hired within 3–4 months prior to when I got started. The stack was pretty primitive, although the foundational work on sharding MySQL had been done, the mid tier was mostly a monolithic Python Django application and we still had a lot of issues with both our caching tier and persistent storage tier. Fast forwarding to August 2019, when I left Pinterest, it was on a microservice based architecture, where 700+ microservices together served 400M+ monthly active users all over the world with highly personalized feeds of pins assembled in real-time at the moment when our pinners visit Pinterest. Not only was Pinterest a highly scalable and reliable service, it was also a thriving business. Pinterest IPOed in April 2019. By the time of my departure, it was a 20B company with a revenue growth rate of more than 50%.

Pinterest Team 2012

Ben Silbermann and Evan Sharp at NYSE Celebrating Pinterest IPO — 2019

However, the path to glory was not all roses, but also full of thorns. I will try to share some of the most valuable learnings I had along the way over successes and failures of my journey at Pinterest, so that you can hopefully take some shortcuts for your company or your team to achieve what you set out to do.

Without further ado, let’s dive right into the first part of my seven learnings over seven years at Pinterest:

1. Customer obsession. Notice that it is an obsession over customers, not technology. I don’t know how many times we as an engineering organization at Pinterest allowed our engineers to chase shiny technologies instead of focusing on solving the problems for our pinners and our advertisers, especially during the early days. For example, when we first got started with building our advertising platform at Pinterest back in 2013, we made the decision to develop all the core ads services in Golang, despite very little adoption of Golang at that time outside of Google. Some of our engineers were very excited about the new programming language, the powerful, natural and expressive idioms Golang provided to build large scale distributed systems. However, what we failed to take into consideration was that back then, there was not really any good open-source service framework for Golang developers to build microservices on, actually there was not even a good thrift Golang RPC client/server implementation. To adopt Golang at that time, we had to take on the challenges to build a lot of foundational libraries and frameworks before we could get to things that actually delivered business value. Also when we hired people who had experiences with developing large scale ads systems from Google, Microsoft, Yahoo, etc., since they were all overwhelmingly C/C++ engineers, they had to spend quite some time on learning how to be proficient in Golang, which arguably could be deemed as time being wasted in terms of GTM. As an engineer of a small startup, you have to be wise with whether to adopt cutting-edge technologies vs. use mature, proven ones, because even though there could be a lot of excitement for us as engineers to learn them and making significant contributions to those communities, they might not be the core competence of your team, and they might not bring your company any significant competitive advantages. In such circumstances, you might be doing that at the expense of value being delivered to business and customers, and sometimes even worse, huge opportunity cost caused by the delays in GTM. The end story here was that we had to spend almost 2 years with dozens of engineers on rebuilding our ads platform later.
2. Governance over key decision making. I know that this one could be very controversial. After all, don’t all the leadership principles tell us to empower our team members, and give them autonomy? I’d like to clarify that not all technical decisions need to be made by a governance body, only the ones that have profound implications to your architecture or your overall engineering organization, especially if the consequences of those decisions are hard to reverse in the future. For example, as an engineering organization we shouldn’t dictate whether people use Vim, Emacs, or some sort of IDEs to develop their code; however, when it comes to a decision like what programming language we should be using to develop our entire ads platform, or whether we should be using one particular database technology to empower all of our use cases, the decisions are better to be made through an engineering governance body with your most senior engineers in that domain together with the tech leads and the teams who are going to build and own those platforms and services. After all, it is not fair to have one individual make a decision that is going to impact everyone else in the engineering organization without asking them for their perspectives; and it is not fair to have an individual or a team to make a technical decision that could have a profound impact to whether the business could even survive in the future.
3. Keep it simple. As engineers, we take pride in what we build. But sometimes we take it too far, that is why there is such a word called over-engineering. I believe most engineers get why we shouldn’t prematurely optimize our code for performance. The same applies to other aspects of software development. When you are still in the phase of exploring viability, being able to quickly get to experimentation and feedback gathering is critical. It is not the right time to build craftsmanship into your system yet, because tomorrow those code could totally be thrown away. However, I do think it is important to embrace proper software design patterns even if you are in the phase of prototyping, so that it is easy for you to further extend the code and evolve the features. Also when you are developing an external facing enterprise grade API, you need to be super thoughtful before releasing it, because most of the time you need to maintain backward compatibility for a very long time.

That is it for today. If you enjoy reading this post, please follow me on Medium, give me a clap, and have it shared with others. That will help motivate me to find the time to have the second part published sooner. If you have different perspectives, please share them with me in the comments, so that we can have dialogues on them.