An Inside Look with Codeship: Peter van Hardenberg, founding developer of Heroku Postgres2015-08-12
Peter van Hardenberg is the founding developer of Heroku Postgres. We recently had a chance to catch up with Peter and talk about the challenges facing industry, the Heroku team, and how Mario influenced their software development.
An Inside Look with Codeship is a regular series providing an insider’s perspective on founding and building a tech company. Each session, we chat with some of the most exciting voices in tech and ask them where they’ve been, where they’re going, and what we could all be doing together. You can read all Inside Look interviews here.
Hi, Peter! Can you introduce yourself and what you’ve been working on at Heroku?
Sure thing. I’ve been part of Heroku for about six years and from about a dozen employees. I run product at the moment for our ecosystem and data products. I was the founding developer for the Heroku Postgres product.
When you started building Heroku Postgres, was there one major problem for your users you wanted to solve?
Heroku had been building this development platform and application runtime platform, and we realized at some point that, in order to do that, we were going to need a really strong story around data as well as around running people’s code. So the founder Adam Wiggins came to me, and he said, “Peter, I want you to look at the databases.” And I said, “I hate databases, but I agree that this is really broken, and so I will spend six months on it.” And that was five and a half years ago now.
Ultimately your code is important to your business, right? Your code is how you express value, but your real business is your data. If you lose your data, your business is over.
I had a friend who worked for a startup that, during a data center move, the servers literally fell off the truck. The truck went over a bump, the server fell out of the back, and all the disks were lost. That was the end of the company. For us, our primary goal has always been durability and ensuring business continuity for our customers, but that’s really just table stakes.
We observed from the beginning that databases were always something that intimidated developers and that had become governed by a priesthood of specialists who, by and large, were not well-aligned to business objectives. Not to speak ill of DBAs, but I think that the relationship between DBAs and developers was always very challenging. Our goal has always been to make every developer able to run applications at a very large scale, no matter how much experience they have with the data.
How did you approach problems you encountered by growing so rapidly?
Fundamentally, when you’re going through exponential growth, you need to be getting better faster than you’re growing. This is a real challenge that we understood from the beginning could only be solved by basically becoming the best at automation, anywhere.
We’ve got millions of databases and many, many thousands of servers; and we do it all with no ops team. Unless you count, obviously, our infrastructure as a service provider. Our team has always been integrated between development, operations, and user support. Everybody on the team does on-call. Everybody on the team does support, and everybody on the team does development of different varieties.
Obviously, as we’ve grown, we’ve specialized somewhat. But it’s still a guiding principle that everyone should be exposed to customers as directly as possible. This is a little bit controversial. I think it’s quite common for people to try and shelter their developers from users, but I think that you really only can build that true empathy by actually having to talk to and deal with users on a regular basis.
You talk about having your developers not so siloed away from the company. Does that change what you look for in your developers when you’re hiring new members of your team?
I think what we really look for is grit, the ability to learn, and gallows humor.
I think that gallows humor is highly correlated with the ability to survive in a high-energy startup. You know, you need to recognize that everything is miserable, and it’s unlikely to get better; but that, ultimately, that’s what we’re doing for our users. We suffer so that, hopefully, other people won’t have to. I think that if you can’t laugh at that, you’re really going to have a hard time. Databases in particular are miserable, you know?
One way we talk about our goal internally is to just try and claw back the misery of using databases just a little bit. Databases are still terrible to work with, but I think we have managed to make them more approachable and more joyful for developers to use. And, ultimately, that’s what we’re in the business of doing.
You’ve previously talked about the necessity of wearing many different hats while leading a product, but knowing how to take off the hats when it comes to people issues and culture. How did you come to that lesson as a manager?
It was advice given to me by Jim Lindenbaum, one of the Heroku founders. I’d been working with the team, and I was trying to figure out how to balance people’s individual desires for their own career development with what the team needed.
He gave me a really good piece of advice. He said, “No. You need to figure out what the team needs and get that. But when you’re working with people, you need to stop thinking about what the team needs and just focus on what that person needs.”
That sometimes creates a tension, obviously, between those things. If a person wants to go in a particular direction, but that’s not what the team needs right now, that can be challenging to manage. But I think that it’s important to be able to take off your “what does the product need right now?” hat and really focus on “what does the person that I’m working with here in this situation need?” It’s the same when you’re working with customers versus your business. I think the ability to view things through those different lenses is really essential.
I understand Heroku works with remote teams.
Absolutely. Heroku is more than half distributed. We have about 200 employees. About 55 percent work from somewhere other than San Francisco. This was a real challenge for us to overcome, initially. In the early days, I actually relocated to San Francisco because we were an all-local company. And as we grew, we realized just the incredible potential that was available by taking advantage of the global market, as opposed to just the Bay Area market.
How do you make sure important knowledge and information doesn’t get lost and reaches your entire team?
We use a whole variety of different tools to manage that. Everything we do is in a Google Hangout or is in email. It’s unsexy, but mailing lists are fundamentally one of the most key tools we use to run our business – and, of course, tools like HipChat and GitHub and everything else that everybody uses.
People talk in Hangouts, but decisions are made in mailing lists and on pull requests, so we always have a written record of these things. When I say “always,” I’m sort of rounding up, of course. The simple truth is that if you’re not in the office you’re going to miss some things. If you weren’t in the Hangout, you’re going to miss some things, and that’s just a fact of life. But that’s true whether you’re in the office or not to some extent. If you weren’t at the meeting, or if you were out at a conference that week. Ultimately, put things into writing and make sure that as much as possible is communicated.
Is there a unique technological difficulty your projects have had to overcome?
When we started this, no one had done this before. Heroku Postgres predates Amazon’s RDS service, and so at the time, there weren’t really any database services out there except for Amazon’s SimpleDB, which was not as robust or functional as was required for most consumer applications. Some of the difficulty is scale and availability. Some of it fundamentally is about understanding the user experience. It’s really about investing deeply and understanding what a developer needs to understand, thinking about what do they need to know and then automating that.
I think the greatest personal technological advantage is that my background is diverse. I’ve done video game development. I’ve worked in research laboratories. I’ve done Shakespearean scholarship. When I came to Heroku, I brought sort of a pretty diverse set of experiences with me.
The best trick that we pulled in was actually drawn from video games, which was to treat our databases a little bit like the enemies in a Mario game. All characters in video games are little state machines that travel around, and when they reach the edge of a platform, they turn around. Or, if they see Mario, they throw a boomerang. That kind of thing. We really treat our databases kind of the same way, which is that if a database has an interruption, then it spawns a little state machine, and it says, “Okay, I need to find a new server,” “I need to repair this disk,” whatever the task of the day is. And so really what we have is a very large, distributed job processing queue, where we’re continually taking every database that we have and saying, “What does this thing need next?” We’re feeding all the events that we get from the world into it, so we can think about it a bit like a very large game where the goal is to maximize the availability of our service. And I think we play it pretty well.
Is there a way that Heroku grows the team skills or mentors learning?
I think participation in a company like Heroku really exposes you to a lot of challenges. We don’t have any formal mentoring programs in place. Actually, we have historically biased towards bringing in fairly experienced engineers. One of the things I’ve seen in some of our teams is that bringing a more junior engineer can really give everybody else a lot more focus and discipline, because it forces them: 1) to work with that person and explain what they’re doing and 2) to be really conscious of how somebody who’s less experienced might perceive and understand a thing.
You know, you don’t want to have a team that’s unbalanced towards all junior people, but I think that having all senior people can actually be a big of a crutch as well. Building a team with a blend of skills is really valuable, not just in terms of cost management, but in terms of having a few people who are newer in the industry and are really passionate about growing their skills from nothing. They don’t have bad habits to unlearn. They see things with really clear eyes.
Is there a way that you and Heroku try to give back to developers outside your company?
Our mandate is to try and make databases less awful. We have a saying, “The worst code is code you write yourself, the second-worst code is code somebody else maintains, and the best code is no code at all.” So we always take the position that if you’re writing code, that’s probably a mistake. if you can get someone else to maintain that code, or get that code maintained by a community, that’s way better.
So to some extent, we upstream and open-source a lot of the changes we make in Postgres and in other tools. We try to open-source and release tools and get a community built around them. We see that, just from a business strategy perspective, as awareness that will come back to Heroku. But it’s also about fundamentally aligning to our goal, which is to make building software better, less miserable, more effective.
What do you think is the biggest challenge for the industry right now?
I think we have a lot of technical and social problems.
On the technical front, we’re going through a really Cambrian period right now, which is wonderful. If you remember, the stagnant — maybe “stagnant” is unfair, but there was a time when everything was Java. I wouldn’t call it monoculture, but it was certainly a less diverse technical world, say, 15 years ago than it is now.
Today, there are multiple package managers for Node. It feels like there’s a new programming language of the week that people are getting into. Is it Go, Rust, Crystal? I think that’s great, but it also introduces a lot of new challenges in terms of managing the complexity of projects over time. It’s wonderful that we have these new tools that make it possible to digest this complexity. It’s easier to build your product in multiple languages if you use tools like Heroku, which allow you to deploy all your applications the same way, no matter what language they’re written in. On the other hand, I think it’s getting harder and harder for developers to pick tools, because there are so many options which appear viable but that may or may not survive in the market.
But if you want to talk about what really worries me in technology, it’s that we have a relatively small group of undiverse individuals making decisions that will impact the direction of our society for a long time. There’s poor representation from women. There’s poor representation from a lot of visible minority communities, and we are not doing a good job of expanding that.
If you look at the Gamergate controversies of the last year, it’s tied into that. I see areas where we’re making progress. I want to give some credit to the Ada Initiative. I’m a big fan of the work they’re doing improving gender representation, but it really feels like we’re still in the Stone Age and that this is still a boys’ club. That needs to be fixed.
Is there something you’d like to see the industry as a whole doing more of in a practical sense?
We’re still losing the share in terms of the number of women in engineering, and I think that there’s a lot of people doing great work there. I’ll give credit to organizations like Hackbright, who are trying to shift the ratio, but I definitely want greater investment there.
I’d encourage any company to get involved with organizations like Hackbright and see what they can do to help include communities that are not well-represented in technology today in their company.
Is there recent advancement in the industry that is there right now that will transform the future of your work?
This is an industry that is in constant flux. It’s hard to pick out anything — as William Gibson said, “The future is already here. It’s just not evenly distributed yet.”
Broadly speaking, I’m really excited about the consumerization of developer technologies. Heroku has been a big part of this. That’s why I’ve been so passionate and excited about working at Heroku. If you go back a few years, developers were really treated like experts, and the idea was that, “We build tools to be powerful, and developers will figure them out.” I think that the new crop of companies coming up, Codeship among them, take a fundamentally different perspective, which is to say that developers are people who deserve to have great tools to use, and that those tools need to be approachable and easy to use and intelligible. Whatever we can take off a developer’s mind is something that won’t distract them from solving the problem that they’re there to solve.
Our goal is not to build user-friendly tools precisely, but it’s to make developers more able to build better software by reducing the number of distractions and frustrations in doing so. Five years ago, there were very few companies doing this. Today, there are dozens, hundreds, even, focused on so many different parts of that stack.
The other big trend supporting this is the move to the cloud. Now these services can collaborate much more effectively than they could a few years ago, because people are mostly hosted in Amazon. It means that low-latency interconnection between those services is possible, so you can buy your database from one person, you can buy your CI service from another, and you can buy your runtime from a third. You can run your own ersatz, custom stuff alongside as well and those things can all inter-operate. I think that that kind of marketplace and focus on developer experience is huge.
What do you think are the key skills required to be an effective product manager or engineering lead?
Know how to ask questions. The first question is always, “What problem are you trying to solve?” If you don’t know what problem you’re trying to solve — whether it’s a technical, product, or people problem — you won’t know if you’ve accomplished your goal. So many people get excited about an implementation, or an idea; but they run ahead of making sure that they’ve documented what problem they’re solving and also what problems they’re not solving.
A good example might be if you’re building a new feature to reduce the amount of time for a log-on. You know you’re building a new log-in page. Why are you building it? Are you trying to reduce the friction in sign-up? Okay. How are you going to know that you did that at the end? You’ve got to make sure that whenever you set out to do something you know it’s use. Is that actually going to be useful? Is it going to be used? What problem does a user have that this solves?
Number two is empathy. You need to have empathy for the user, for the team members, for management, for your reports. Ultimately, software is hard, and people are harder. And people make software, so you’re ultimately going to be solving people problems, whether you’re an engineer, a product manager, or a manager.
If you could go back in time to the start of your career, what one piece of advice would you give yourself?
I’ve been very fortunate. My whole career, I’ve basically followed a strategy of never being afraid to say yes to an interesting opportunity. Nothing I’ve ever done has been planned far in advance. It’s always been about recognizing a new opportunity and following it when you see it. That used to worry me more than it does now, but if I had tried to plan my career, I certainly wouldn’t be where I am right now.
So, I would say: take opportunities, feel comfortable with the uncertainty. It’ll all work out.