Voxelwise

Category: Pro Tips

Going to gemba, any day now

If you want to enable change then you must be an active participant.

It was the mid-2010s and I was working as the head of software for a local scientific instrument manufacturer. The executives were looking for ways to increase velocity and urgency. They felt that the company had slowed down significantly in the last few years. 

During an all hands meeting the President of the company got up and described how we would be enacting lean manufacturing principles. One element of this approach was “going to gemba”, with gemba representing where value was created within the company.

The President told anecdotes about leaders putting their office, or at least a desk, right on the manufacturing line. The goal was to see and hear the opportunities to unblock work and improve processes. This was part of genchi genbutsu, where the only way to understand the manufacturing floor was to actually go there. Toyota executives would talk about the daily “gemba walk” where they would move through the manufacturing center to observe and identify inefficiencies.

Following the all hands we collectively waited for a member of the executive staff to move their office from the front of the building to the manufacturing area in the back. At the very least we expected them to begin regular walks through the area.

It never happened.

Nothing undermines change management like hypocrisy. In this instance, the leaders of the company were asking everyone at the company to make major changes. People were expected to suffer whatever sacrifices were necessary to make those changes successful. We were told that this was required to ensure that the company would be competitive into the future. Then, when it came time to enact those changes, it was clear that the executives were not willing to make sacrifices themselves.

Why should anyone else in the company commit to a major change when the executives refused to change any element of their work?

Two years later we got a new CTO who was hired by the executives to improve R&D and manufacturing. The first thing he did was kick an engineer out of a well-placed R&D cubicle so he could sit at the intersection between R&D and manufacturing areas. This resonated with everyone in both departments. The message was clear: he was there to be a part of the problem solving.

In the end, there are two lessons to take away. First, leading by example is critical and nothing torpedos change management like hypocritical “for thee, but not for me” leadership. Second, it does pay dividends to get close to the value creation in your company. Maybe you don’t have to give up that front office 100% of the time, but set up shop in the back of the building at least a few times per week. Your gemba is likely there.

Photo: One last peek at my desk at Amazon before we moved to a new office in downtown Santa Barbara, CA. We were in the new office for about two months before COVID hit and we scattered to work remotely.

Iterate, iterate, iterate

Pretty much anything great comes from constant iteration. The vast majority of the time, it is better to start building and learning than taking forever to create a perfect plan that is uninformed by experience.

Last year I had to get a halloween costume with 48 hours of notice. I decided to go with something easy to pull together: a classic Star Trek uniform. Black shoes, black pants, a gold shirt (command staff, naturally), and a tricorder mockup. All I needed was the badge to put on the shirt.

There wasn’t time to order a badge online, so I turned to my 3D printer. I found a few badge files online and began the process of iteration:

  • #1: Too small, needs to be bigger
  • #2: Right size but had a printing error, so couldn’t really try the vinyl overlay
  • #3: Tried to use cut vinyl for an overlay, but it was hard to place and didn’t look good
  • #4: Try it a bit smaller. Ok overall shape
  • #5: Printed a gold final version, but the vinyl star still didn’t look great
  • #6: Made a window in the top layer and put a black layer underneath. Still looked messy.
  • #7: Combined elements of the previous tries: larger badge, flatter height, and a smaller window for the black layer underneath. SHIP IT.

If there is one thing that 3D printing has taught me is that you often need to just give it a try and see how things turn out. All the preparation in the world in a CAD program helps, but nothing is better than printing your current best guess and holding that iteration in your hands. Sure, you will print a few things that get thrown immediately into the trash can. Fortunately, plastic is cheap and the rate limiting step is just the time it takes to create a print.

I had been at my current company for just a few months the first quarter of 2020. We were working on a major new vendor integration. A staff engineer on my team came to me and said “we have been planning the project for four weeks, but we will need four more weeks to complete the planning and be ready to start coding”. I weighed my options. I trusted him, and I trusted his assessment that more time was needed for planning. At the same time, I knew that we had to begin the process of iteration so we could learn how the product plan would come together in real life.

“No. I’d like you to work with the team and start coding next week. Even if we have to throw away the code, we need to start building.”

It was a risk on my part, but a calculated one. In the end, I asked what the team needed more. Did it need greater knowledge of the vendor interface or did it need greater knowledge of using the interface in real world circumstances? I decided it was the later. While it is impossible to say if this was a key project decision, we did end up shipping the system on time that quarter.

It would be cavalier to leap into a big project with zero planning. Likewise, if your goal is to reduce risk to zero then you will be planning forever. The key element then is finding the right balance between speed and risk. One of the best ways to reduce risk is by gathering data on real world operation through constant iteration. Gaining experience and letting data guide you is worth many weeks of conversation on theoretical points.

Photo: The iterations on the badge, starting from #1 on the left to the final #7 version on the right. Hot glued some magnets to the back and I was in business.

Director: Other duties as assigned

At some point in your career the job description becomes 100% “whatever it takes for your team and the company to be successful”. The people that get ahead are the ones that realize this fact early.

Steve Jobs is always a controversial character when evaluating management styles, but I have always loved this story regarding his expectations for VPs:

Steve Jobs told employees a short story when they were promoted to vice president at Apple. Jobs would tell the VP that if the garbage in his office was not being emptied, Jobs would naturally demand an explanation from the janitor. “Well, the lock on the door was changed,’ the janitor could reasonably respond. “And I couldn’t get a key.”

The janitor’s response is reasonable. It’s an understandable excuse. The janitor can’t do his job without a key. As a janitor, he’s allowed to have excuses.

“When you’re the janitor, reasons matter,” Jobs told his newly-minted VPs. “Somewhere between the janitor and the CEO, reasons stop mattering.”

“In other words,’ (Jobs continued,) “when the employee becomes a vice president, he or she must vacate all excuses for failure. A vice president is responsible for any mistakes that happen, and it doesn’t matter what you say.”

John Rossman, Think like Amazon

There is a lot of power in the Jobs story. As a senior leader you own it all – both successes and failures. The reasons for each stop mattering. My addition to that perspective is that ownership shouldn’t start at the VP level. You should be going above and beyond to deliver results, and owning all outcomes, wherever you are in your career journey. This trait is rare and puts you far above your peers in any industry.

I have had numerous direct and indirect reports that requested a kind of checklist for promotion. “Just tell me what to do and I will do it” is a common refrain. This is relatively easy to provide to recent grads up through Senior engineers. The expectations at these levels are more straightforward, almost mechanical. At the Staff+ levels it becomes much more difficult to create that kind of to-do list. This is because managers are indexing less on your technical knowledge and project completion rate. Instead, we are focusing more on your soft skills, including communication, collaboration, and ownership. Establishing strong ownership early shows a senior level of execution, accelerating your career growth.

This same ownership effect occurs within the management ladder. While software management is already a bit like four different jobs masquerading as one, at the Director+ levels is starts boiling down to “whatever it takes”. This explains why so many Director+ job postings are oddly vague, with terms like “create a healthy team culture”, “collaborate with product management”, or “ensure technical excellence”. As a job posting there has to be a job description, but how do you capture the reality of “responsible for everything”?

One of my personal Top 50 management books is Extreme Ownership by Jocko Willink and Lief Babin.  You can distill the entire book down to a single view: you always have some control over a situation, so stop giving excuses and pretending that there was nothing you could do. The individuals who get ahead quickly are the ones that internalize this fact early.

Photo: The Cessna 152 that I did most of my flight training in. I have a lot of fond memories in that plane, including my first solo flight. On any plane the pilot-in-command (PIC) is the individual directly responsible for and the final authority regarding operation of that aircraft. No excuses.

Scrutinize advice from the lucky

Never accept career advice blindly.

Sometimes I get asked to do something like a brown bag lunch talk on careers and career advice. The above comic is always the first thing that I show the audience. If I leave them with nothing else, I want them to not simply follow the advice of some schmuck like me just because I/we have achieved some measure of “success”. If I can get the audience to think critically about my presentation then I have succeeded in the most important goal.

I like to think that I am reasonably good at what I do. After ten years in academia and twelve years in industry I have learned a few things that make me more effective as a researcher, engineer, and manager. Still, a significant degree of luck has also shaped my career path. Here are a few, off the top of my head:

1) Having my grad school interview schedule changed at the last minute and randomly meeting the person who would be my future advisor.

2) Finding a great postdoc position because I literally ran into a professor at the Cognitive Neuroscience Society conference in New York and he said that his lab had a job available.

3) My first software manager deciding to take a risk and hire me even though I had no formal training as a software engineer.

4) Getting my first software management position because the CEO happened to remember that I organized large research projects as a postdoc.

5) The startup I joined getting acquired by a FAANG nine months after they almost ran out of money and laid off 30% of the company.

Hard work is always necessary. A little bit of luck can make all the difference though. Survivorship bias pops up when only the successful people are around at the end. Put another way, the people who have gotten lucky are much more likely to be giving presentations than the people who never got a fortunate nod from the statistical universe. Always know that.

Comic from XKCD: https://xkcd.com/1827/

Associated explainer: https://www.explainxkcd.com/wiki/index.php/1827:_Survivorship_Bias

Shine bright, but don’t burn out

I have ridden the burnout train before. It was a miserable experience.

Rewind to 2020. The startup I had joined in 2016 was acquired by a FAANG company in 2017. The new company gave us a healthy grace period to ramp up, but as time marched forward the corporate expectations on our org grew ever-larger. Information security demanded an audit. Finance dictated we shrink cloud spending by 10%. API response latency was up and we needed to bring it down by 50ms. I was running the SRE team, and it was becoming a lot.

The point I knew something was wrong was when I got paged during Thanksgiving 2019. We had traveled to the Sierra Nevada mountains with friends and the feast preparations had begun. My pager went off. I sprinted upstairs to grab my laptop, sliding in my socks on the carpet. My knee went down hard into the floor. 

I opened my laptop, saw that a service had run out of memory, restarted the instance, and went back to my holiday business. Total distraction time: 15 minutes. As I nursed my rug burned knee something was clear to the people in the room that was still dawning on me: I was hating every minute of my job.

The hate I felt wasn’t toward any person or any situation. It was a generalized frustration and irritability. I hated that I was underwater. I hated that I felt frustrated by my workload. I was quick to anger when things didn’t happen properly. I wanted to blame my manager, but I knew that I had willingly taken on my responsibilities. I hated that I shot myself in the foot by not communicating more effectively with my manager. That fact made me even more frustrated, but purely at myself this time.

The next week I (finally) talked with my supervisor and let him know that I was burnt out. He moved quickly to lighten my load and divide some of my responsibilities. This didn’t fix my burnout, but it did provide some breathing room. It was also a great career advancement opportunity for my peers. In hindsight, by holding on to so many responsibilities myself I was preventing others from demonstrating the skills to get them promoted. That is a blog post for another time…

My burnout recovered bit by bit. It took six months, but by the next summer I was feeling back to “normal”. Still, that July a former coworker let me know about a great new opportunity with their company. If I didn’t just go through burnout recovery I probably would have ignored him. Because I was still harboring some of that frustration and irritability I ended up saying “yes” to the recruiter when they reached out. The rest is history.

What can we learn from all of this? I think there are several lessons, but we can break them down into three for now:

1) Be vigilant for signs of burnout in yourself and your employees. Steer clear early.

The total cost for a bad case of burnout is high, for both the business and the individual. It is far better for everyone involved to steer clear of burnout as soon as the early signs are present. Don’t wait for them to say they are overloaded. Constantly ask them about their stress level, their workload, and week-to-week changes in their relationship with their work.

2) Burnout may hit your most-engaged people hardest.

In retrospect, the root cause of my burnout was a failure to say “no”. I wanted to help the org whenever we were in a tough spot. I had the skills and the knowledge to address problems that were constantly popping up. Why shouldn’t I help? I felt needed and I felt important. It felt good. However, the constant piling-on just served to erode the foundations of my motivation until it all collapsed on itself.  If you have someone that is constantly volunteering to take care of challenges make sure to check in with them more often.

3) Recovery from burnout can take ages. 

It is not a matter of just fixing the issues or dialing down the total amount of work. I can say with authority that it is an effortful journey back to an engaged state of mind. Time away from work, new challenges, resetting expectations, and setting boundaries are just a few things that can help.

None of this is pleasant. None of this is happy. I think the best step toward addressing burnout as an industry is to talk about it. If a manager decides to consciously ignore burnout as a factor and optimize for short-term deliveries from their team then, hey, more power to them. If you have a desire to optimize around the long-term success of your organization then navigating the burnout cycle is a critical pillar of sustainable engineering.

Photo: The Amtrak Pacific Surfliner stopping at the San Luis Obispo station. It’s a beautiful ride north from Santa Barbara.

Sitting with Uncertainty

The ability to sit with your anxiety and bathe in uncertainty for a limited amount of time can be a managerial superpower.

“Project Blue” was going to be a major effort. I would need to marshal the combined resources of all my software teams to make it happen on the CEO’s demanding schedule. There would need to be all kinds of project alignment, team alignment, people alignment, and change management if the company went all-in on Blue. I was ready. There was only one problem: the executives hadn’t decided to move forward yet.

For two weeks I waited, then I waited, and then I waited some more. My hand was metaphorically hovering over the “GO” button and I needed an up-or-down decision. We were either moving 100% of available product development time to Blue or it was business as usual. Because the decision had so much gravity, with an untold number tradeoffs, the final decision was not forthcoming.

One of my favorite quotes ever is “nothing clears the mind like no choice”. While I did push the CPO for a decision, I knew that the executives had to complete their discernment process to know with confidence which direction to go. This would take time. The decision was out of my control.

What is a manager to do at this point? My advice is to hone the fine art of sitting with your uncertainty and anxiety. Yes, absolutely take action where you can and advocate for the direction you think is best. At the same time, also understand that it may be days or weeks until the uncertainty is resolved.

If you let the uncertainty eat at you then you only end up worn down and burnt out. Not the best way to start a major project. Instead, clear your mind, be comfortable that you have done everything you can, provide additional input where asked, and wait for clarity.

As an aside, we did end up moving forward with Project Blue but with a new project definition that meant the effort from my teams would not be needed. Business as usual…

Photo: Tail rotor assembly from an MH-65E “Dolphin” helicopter of the United States Coast Guard at the Camarillo Air Show. Don’t stick your hand in there.

The power of a manager README

The first 90 days of a new hire matter as much for you as it does for them. Get started on the right foot by providing documentation on your management style.

I’ve had periods of managerial stability in my career and periods where I didn’t even know who I was reporting to. In the twelve-month period from 1/22 to 1/23 I ended up reporting to four different people, including one CISO, one VP of Engineering, and two CPOs. That was… a lot.

Every person you report to is as unique as a fingerprint. To have any hope of building an effective relationship you must learn about them and get to know their values, style, and preferences. For example, I had one manager that wanted a one-pager filled with discussion points prior to every 1:1. I continued this practice for my next manager…and they never read it. Instead, each week they simply wanted a verbal update on how they could best support me and my teams.

The core problem isn’t so much that every manager is different. Instead, the problem is that it takes TIME to understand them. That time can vary from weeks to months in duration. What would you pay for a magic wand that you could wave to cut that time in half? No joke – what would that be worth?

What if it cost nothing? What if all it took was 1-2 hours of effort to fill out a doc template?

The Manager README doc is a widely-known management tool, but it is one that I have only seen rarely in practice. In short, the goal is to put a doc in front of your new reports during their first week that empowers them with information about you, your role, your expectations, your quirks, and more. It contains all the little nuggets of wisdom that someone would learn across dozens of 1:1 meetings, all condensed into a few pages of text.

Here is my README. It is far from perfect. Parts are incomplete, and some sections are getting a bit dated. Still, it is a great doc for me since it consistently delivers on its purpose: generating high-quality conversations with my direct reports. When you finish reading mine, maybe consider writing your own.

https://github.com/prefrontal/README

Some other READMEs to check out:

https://hackernoon.com/12-manager-readmes-from-silicon-valleys-top-tech-companies-26588a660afe

A site to write and share READMEs:

https://managerreadme.com

Photo: A help kiosk at Los Angeles International Airport, being as useful as it can.

“What else?”

Ask “what else?” and keep asking until the room is silent.

Program management gets a bad rap at times. Ask twelve people what a program manager does and you will get a dozen different answers. The best description I ever heard was from a career talk during an internal Amazon technical program manager (TPM) conference: “I don’t know what program managers do, but things seem to go far better when they are around”.

One of the best TPMs I ever worked with had a pro-tip that he swore by. It is so simple that it seems obvious, but it is still a rare occurrence in meetings. After we had worked through the meeting agenda he would ask “ok, what else?”. That would inevitably lead to a missed agenda item that should have been discussed. “Great, thanks. What else?” Someone would volunteer that there was a risk that hadn’t been documented that should be. “Awesome, what else?” Suddenly an engineer would share that we need to talk with our third-party vendor. “Fantastic, what else?”

“What else?”

I am convinced that these two magic words have saved more projects than I have fingers.

Asking “what else” changes the focus of any meeting from simply completing the agenda to truly understanding the state of the project. It also communicates that the goal is to suss out all challenges and blockers, not just to check things off of a list. This subtle shift is powerful.

Don’t be a slave to your agenda. Focus on success of the project. Make sure that all challenges and blockers have been discussed. Ask “what else” until it hurts. Ask it three times in a row. Ask each person individually. Whatever it takes to get a shared understanding of ground truth.

Photo: Ball made of used bike tubes in Breckenridge, Colorado.