How I ship projects at big tech companies

> Shipping is a social construct within a company. Concretely, that means that a project is shipped when the important people at your company believe it is shipped.

A lot of the ideas in here resonated with me a lot. This is the kind of article I wish I'd read before I spent a few years leading an engineering team at a medium-sized tech company!

  If you ship something users hate and makes no money, but 
  your leadership team is happy, you still shipped. You can 
  feel any way you like about that, but it’s true. If you 
  don’t like it, you should probably go work for companies 
  that really care how happy their users are.

  Engineers who think shipping means delivering a spec or 
  deploying code will repeatedly engineer their way into 
  failed ships.

EDIT: clarified the ethical question.

Rarely in large enterprise environments are projects greenfields. The new system is often there to replace some existing Line of Business system.

Inthese cases the pervers but common shipping strategy is:

1. freeze all maintainance and evolution of the existing system. Seems reasonable until you realize that these enterprise projects are multi year calendar endeavours and the business will not stop changing during that time. The idea is not realy to halt all development, just to inctease the friction of getting anything done on the now declared 'legacy' system above an administrative pain threshold.

2. Commandeer all the IT staff to serve on the 'new' system's project team and overwhelm them with tasks. Make sure they remember any time spend on the 'legacy' (keep hammering in that term) codebase is basically waisting time assuring you'll end up at the bottom of the corporate dinosaur tar pit at best.

3. Force they 'new' system into production long before it is ready. Use all the leverage you build from point 1 to promise 'going forward'. When essential gaps to support the business are pointed out en mass week1, reverse the tables and intigate a very heavy administrative process demanding proof that the lacking functionality is actually required with a full ROI breakdown coutering your handwoven 3 minute overinflated guestimate for implementation. Again, you know essentials are missing, you are just trying to gain time as the longer you can stay deployed the higher the chance you will.

This process is very destructive to not just the business, but also moral of most involved. Most projects taking this scorched earth shipping strategy fail regardless. This is no small contribution to the cynicism oftem found in veteran battle hardened corporate IT shops.

- Have endless arguments about what to name the product

- Have industrial design team enforce the latest design language

- Create product variant SKU numbers, get them loaded into ordering system. Get buy-in from sales on these variants.

- Create part numbers for the FRUs and CRUs.

- Create multi-level BOM for all the variants.

- Buy replacement parts and push them into parts depots

- Calculate reliability so that warranty department knows how much to charge you

- Get agency approvals

- Create packaging

- Test packaging

- Translate any words on the package into french (for Canada)

- Get vendors to sign your big company environmental requirements document

- Write legal notices and get it translated into many languages

- Work with documentation team to get printed (usually short) manual translated into many languages

- Get safety engineer to sign off on products, usually means you need to make a bunch of stickers and include them in the BoM.

- Figure out the part numbers for the power cords needed for each country

- Figure out the part numbers for the rack mount rail kits for various types of racks that you support

- Write announcement letter

- Create videos and documents for sales team training

- Write troubleshooting guide

- Train repair center

etc..

You’ll never ship a perfect product; no such thing exists. But if you ship early (and incomplete), users will be delighted—and often change their minds about what they want. That’s great, because you didn’t spend forever on their initial (flawed) requirements.

It sounds to me like this person is good at getting paid no matter how it goes for the shareholders or the rest of the team.

Yes that is pretty much what I see as well, I see how things go to nothing with wrong people who think they can make bunch of requirements and dump them on dev/qa - then be surprised nothing works - and also how much stuff I try to hand over ends up not happening despite people claiming that they are very motivated to pick up a project. If they are motivated and I see nothing happens in 3 weeks I just take it over again and don't even talk to the guy anymore.

I am also quite often seen as an asshole because I don't ask "everyone" for permission and mostly care about my actual boss.

Generally in my book projects should only be assigned to teams, and managing that project to completion should be part of that team's way of working.

Doing this in an agile manner generally means smaller projects, one at a time.

Well now, hold on a second. You didn't tell me this was about consulting.

If we're being paid like pirates and fightin' like pirates thats one thing.

> Project competence. You want to aim for something like a NASA mission control vibe

Having read a little about early Apollo-era (and earlier) mission control this was a fun comparison. The astronauts and flight controllers had no idea what they were doing! There was no textbook – they were performing a human first. They were generally young and very cocky people hired out of university, but were successful anyway, for some reasons, including:

1. They were further culled through harsh adversarial simulation training. This made sure only those that could cut it came out the other end.

2. They approached each new mission phase with a trial mission, each time extending the envelope in a controlled manner. (The cowboy moon touchdown on Apollo 11? It's often described as the first moon landing, but it really wasn't. It was just a test to see if they could reach the surface and get back up. The real first precision landing came on Apollo 12. Oh, but that was also just a rehearsal for the first scientific mission on Apollo 14, etc.)

3. They were fluid in how they organised themselves, and allowed mission needs to dictate which positions existed, rather than trying to impose a hierarchy onto a mission.

Going back to shipping, we can squint and look at this as:

1. Fake it, either until you make it or get tossed out.

2. Prefer to ship many smaller things than one big thing. This builds confidence in your abilities, gets you experience shipping things, and allows you more feedback to adjust to what the real project is.

3. Build coalitions across department boundaries, and draw on their support to ship.

Analogies are like scissors: more fun when you run with them.

I could really relate to this. Having been part of incidents, I fucking hate them and try to anticipate them with various safe deployment strategies. To leaders though it apparently seems like Im not doing anything because “nothing ever goes wrong” lol.

This article is not about “shipping”software.

It’s about pleasing upper management. Those are not the same things. The article even says it explicitly - if your users hate it and the market laughs, but executives like it, you’ve shipped. If you’ve given users the best software ever but your executive management doesn’t know, or even actively hates it, you haven’t shipped.

Blech.

Specifically then, as a technical person, this means you don't ignore the requirements and just think about the code. Instead, you need to communicate with both your higher-ups and your end-users, in order to make sure the code you're writing matches their needs - in my opinion, requirements is the part that most developers are weakest in, but if they strengthen would make the biggest difference. Get it as close as possible to being right on the first time (it's very rare that this happens but focusing on doing really good requirements increases your chances that you don't create the wrong system).

Definitely. This can work very much to your favour.

There have been times when I have happily shipped a product containing issues and bugs that would be catastrophic for any customer using it, because I know that no customer actually ever will. The product is shipped and released, everyone is happy.

The catastrophic issues won't ever come up because that shipped product will never be used by anyone.

Big people are happy because the product is shipped.

Team are happy because big people have stopped causing so much friction in their desire to ship.

When the measurement becomes "have we slapped the label shipped on it", then that's what you get. A label.

Pigs can fly. I know this because if it's got to ship and it's a pig, then sufficient thrust will be brought to bear to achieve flight.

Shipping is some liminal space between sheer will, recklessness, and tautology. It ships because it has to ship, because to not ship is to fall, and failure is not an option.

My biggest career mistake was a project that I had no way to know was the CEO's pet project.

Can't say where (non-disparagement agreement), but oh boy was that a lot of messy fallout — QA awarded me a prize for least bugs on launch, yet at the same time line manger also immediately went from giving me bonuses to putting me on a PIP for having too many bugs in launch.

I have been shipping (as opposed to "writing") software, for pretty much my entire adult life.

A lot of it has involved holding my nose, and gingerly handing it off, dangling between my thumb and index finger, while wearing rubber gloves.

This chap has a point. "Ship" is in the eye of the managers.

Which is why good management is so important.

(more amazing: 54 ships, but not a single leader-ship joke. that takes nerve.)

You're quite likely to be handed projects that make no sense. Bad ideas are created every day. Even if you ship them, you may be associated with their failure.

1. Focus on stakeholder management

2. Know and understand what you build end-to-end

3. Because of 2. you can quickly assess the impact of various options to resolve a problem and communicating this back to stakeholders, generating trust

I like the example of 3. In the blog post.

So many people run projects like they just coast around. Never a moment to pause, reflect and proactively thinking about potential risks and possible mitigations.

Oh my, we never saw this road block coming, how could we have known? Well..

“I ship at big tech” is a broad statement… as a reader, I need elaboration or I tend to question the rest of the article. Is this just a guy bloviating or does he really know his stuff? And bold claims need to be back up.

Some of my observations:

- Often it is about taking a leadership & ownership of fixing if it fails in the production

- Often it is about, guiding team not to overengineer for the future scenarios

This becomes 500% difficult when you are building for someone else:

In our current scenario, we are working with a semi-government organisation who goes by the RFP fine print then an actual project needs.

In general when I ship projects in the context of the org I with for, I _do_ meet the requirements of higher-ups/stakeholders, but not because that's my goal -- but because we're aligned on those requirements. Shipping to make management happy feels kind of incestuous, you should be shipping because you and management both agree on what the end goals of the project are.

This might seem like a subtle distinction, but it can create issues. If you're using your management as your metre-stick without truly understanding and believing in what they want, you'll likely over-optimize for the project's goals that they've stated, but miss the target of actually making the project successful.

This approach is fine if you're primary goal is to get paid, get promoted, please your managers, etc. But if your goal is to create great things, I think you need set your metre-stick based on the project not on the management. From the outside these approaches look very similar -- which is why I largely agree with the "destination" and a lot of the tips in this post, otherwise.

And you will feel the toxicity created by this kind of poor leadership by way of constant blame games, misalignments, and unnecessary stress over garbage.

The bad faith interpretation is that this is corrupt and misaligned.

The good faith interpretation is that each leader has context and motivations that are important.

The right interpretation depends on the exact circumstance.

Getting leaders aligned on the right definition of “shipped” is a separate problem to the one described here, and it’s not the project lead’s direct responsibility. It’s the responsibility of the leadership team, the CEO, and the board.

But the realism proposed in the post is useful for project leads. Meeting leadership’s expectation is not necessarily corrupt, and if it is corrupt, it can be improved by directly discussing and negotiating what it means to ship a project.

> you have to get clear on what the company is looking to get out of the project.

Regardless of politics, starting with why we are doing this project at the first place is often overlooked.

True, but that doesn't mean you have a good leader. They are hard to find, but not impossible.

* Your first bug report that you file against yourself which you fix yourself. Your project is stable enough to be usable with a known bug, but you go back and fix that bug when you have time to dedicate to it. Contrast with nothing working until everything works.

* Your first time seeing someone find a flaw that lies so far out of your own mental model of the project that it causes a very jarring feeling, hopefully followed by excitement that the project is now taking on a life of its own outside you / your team as the sole developers and users. An example of this might be someone filing a bug that image rendering is missing from a git visualisation tool where you’d never even considered using it for projects with version controlled image assets.

I feel like there’s an analogy here to a baby’s first detectable heartbeat and a child’s first original that-never-occurred-to-me thought. There must be a lot more in between too, with the social acceptance of a project becoming shipped as the child entering the adult world as its own person.

Feature flags are a super weapon if you are trying to keep the path to production clear, especially if you have a diverse deployment or customer base. They don't have to be a constant binary value either. I've set up "flags" that were SQL queries which produced the actual boolean fact based upon customer-specific data and requirements.

These things do tend to accumulate, but cleaning them up is easy as long as you have a way to report on the current configuration across all deployments. If two or more instances have different settings but their users seem happy with this arrangement, then you go the other way and promote the flag into some kind of official configuration abstraction.

Customer and environment specific branches quickly become a death spiral if you are trying to grow a B2B/SaaS product. I'd much rather fight a spreadsheet of flags & config than rebase branches and resolve conflicts that have silently piled up for the last several months.

We call this role something like "lead developer" and it's as much about people, relationships, logistics, and documentation as it is about technical systems — but like TFA says, it's also your job to have the best holistic understanding of the technical system. A lot of more junior engineers are likely to work on a large project, and they will inevitably need guidance to satisfy the constraints of other parts of the system. You are responsible for knowing those constraints and, as much as possible, preventing mismatches when different components come together.

Meanwhile, the non technical staff (whether it's a manager or someone from the sales team, documentation team, customer support team, etc) are going to need someone who can explain the technical system to them in plain language. "Can it do this? How fast? Why can't it do this?" You tend to become their main interlocutor.

It's kind of your job to anticipate problems before they happen, to look into the future so you see what's coming and solve it in advance, and then to show up all the time when something needs help around launch, which it always does. It's very undefined but there's always something that needs doing.

This role doesn't sit well with some engineers who think their job is basically about coding and staying within their lane — those engineers don't get to be lead dev too much, because they don't do well with the organizational side of things. (It depends, of course, whether there is a hands-on project manager who can pick up a lot of that slack, but where I work, we don't always have that.)

(Context: I have worked in a large tech company a little more than 3 years and shipped some things. And before that I shipped a lot more things in smaller places – I like to think that shipping in small shops was a good prerequisite for shipping in large ones.)

This is not universally the case, and I've been much happier when I find teams and orgs where this is not true.

Good leaders understand the details. Find them. Be one.