Lockheed-Martin’s Skunk Works has a hallowed place in the annals of engineering. Established in 1943 near a plastics factory in Burbank, California as a secretive, elite engineering group they built the first jet fighter for the United States, the P-80. Led by Kelly Johnson their chief engineer they completed this amazing feat in 143 days from the start of the design process to the first production model flying, under the severe resource and personnel shortages you would expect in a nation at war.

In 1951 led by Kelly Johnson the Skunk Works developed the iconic F-104 Starfighter - affectionately called “the missile with a man in it” which set many speed and altitude records.

In 1955 they producing the U-2 - an ultra-high altitude reconnaissance aircraft, or “spy plane”, if “ultra-high altitude reconnaissance aircraft” is too much of a mouth-full, for the CIA. Capable of flying above 70,000 feet (21,300 m) - well above the range of soviet ground defences and fighters - it helped to pierce the iron curtain by providing high resolution pictures of the soviet union in an era before satellites.

While the soviets raced to build weapons that could shoot down the U-2 [1] the Skunk Works created another spy plane - possibly the most amazing aircraft of all-time - the SR-71 blackbird. Capable of sustained speeds of Mach 3.2, at heights of 85,000 feet (nearly 26 Km from the surface of the earth), the SR-71 was the world’s fastest and highest-flying operational manned aircraft throughout its career. Although 12 of the 32 that were build were lost in accidents, none was ever lost due to enemy action. To evade any missile the SR-71 just needed to accelerate. Everything about this aeroplane - from its massive J58 engines that got so hot they became almost transparent, with movable cones at the air inlets designed to precisely position shock-waves within the air inlet, to its corrugated titanium skin (the materials for which had to be secretly sourced from the USSR, the world’s major producer of the metal at the time, and which was very difficult to work with), to its low radar cross-section shape was revolutionary.

Then in the 1970’s the Skunk Works made another amazing breakthrough. Using the erroneously-unclassified work of a soviet physicist/mathematician specialising in calculating radar-cross-sections of objects, they were able to design an aeroplane with a radar cross-section 1000 x smaller than any other aircraft. This design was eventually evolved into the F-117 Nighthawk, which the U.S. used to great effect against Iraq and Saddam Hussein in 1991 to perform a first strike against critical pieces of defence and communications infrastructure.

On the recommendation of my friend Chai I recently read a book about the Skunk Works by Ben Rich, president of the Skunk Works from 1975 to 1991 who was picked as the successor to Kelly Johnson, the founder of the Skunk Works and principal engineer on the P-80, SR-71, U-2 and over 40 other planes. The book, simply and unsurprisingly titled “Skunk Works” details Rich’s time as both the Skunk Works director, and earlier as a rank-and-file engineer. I found it a great read, filled with many tales of ingenuity, amusing anecdotes from test pilots, and a liberal dose of the “right stuff”.

Lessons Learned

One thing I didn’t expect when reading this book was how familiar some of the Skunk Work’s practices seemed compared to Software Development. Perhaps this is an indication of how much the aircraft industry has changed, but after hearing about the design and testing process for the F-35 which has taken about a decade, and the problem domain, where errors could certainly result in loss of life, I was expecting things to be much more “process-heavy” [2]. Instead I found a few parallels with current software development practices which focus on rapid iteration, flexibility, collective ownership, and building quality throughout the process. Although in the past I’ve been fairly critical of software development relative to other engineering disciplines, I think there is a lot of inspiration we can draw from them. Here are some of the similarities, and/or things we should aspire to, that stood out to me.

Prototype

Build prototypes quickly to prove out concepts, using off-the-shelf components where they don’t compromise the overall design. The F-117 was a great example of this. The Skunk Works built 2 prototypes, which ended up being slightly smaller than the actual F-117, code-named Have Blue to prove out the stealth capabilities of their design. They built it using standard avionics, landing gears, flight control systems, engines and other parts to allow them to build the prototypes more quickly.

Own the End-To-End

The Skunk Works founder Kelly Johnson thought design engineers should stay very close to the production line, to allow them to work closely with the people building the plane they’ve designed.

We created a practical and open work environment for engineers and shop workers, forcing the guys behind the drawing boards onto the shop floor to see how their ideas were being translated into actual parts and to make any necessary changes on the spot. –Ben Rich, Skunk Works

He built up the best engineering organization in the world. Kelly’s rule was never put an engineer more than fifty feet from the assembly area. –Norman Nelson, CIA Engineer at the Skunk Works for the SR-71

Johnson ensured this “end-to-end” perspective extended beyond just design and construction. He had 14 rules that he used to run the Skunk Works by. Rule #9 stated that:

The contractor must be delegated the authority to test his final product in flight. He can and must test it in the initial stages. If he doesn’t, he rapidly loses his competence to design other vehicles

In other words divorcing the design and build process from the operational testing of the design robbed “the contractor” (in this case the Skunk Works) of the ability to close the feedback loop and learn the lessons from this aircraft design. Designing something and then throwing it “over the wall” for someone else to test and operate means you have no idea if the design you came up with is correct. Did you make the right trade-offs?

The Skunk Works also preferred to support their planes themselves, providing flight crews to the CIA or Airforce as well as the planes and parts. Ben Rich cited orders-of-magnitude efficiency improvements in terms of # of crew required to support a single aircraft, rather than using Airforce staff.

Stop the Line, So the Line Never Has to Stop

We made every shop worker who designed or handled a part responsible for quality control. Any worker — not just a supervisor or a manager — could send back a part that didn’t meet his or her standards. That way we reduced rework and scrap waste. –Ben Rich, Skunk Works

inspection and quality assurance branch checked the product at all stages of development. That was also unique with us, I think. In most companies quality control reported to the head of the shop. At the Skunk Works quality control reported directly to me. They were a check and balance on the work of the shop. Our inspectors stayed right on the floor with the machinists and fabricators, and quality control inspections occurred almost daily, instead of once, at the end of a procedure. Constant inspection forced our workers to be supercritical of their work before passing it on. –Ben Rich, Skunk Works

The Skunk Works made quality everyone’s responsibility, much like the Lean manufacturing processes used by Toyota, and subsequently adapted for lean software development.

In spite of this, Ben Rich thought modern aircraft development focused too much on arbitrarily high quality targets. He recounts a rare meeting at an aviation industry event, where he met the great Soviet aerodynamicist Alexander Tupolev. Over dinner Tupolev said:

“You Americans build airplanes like a Rolex watch. Knock it off the night table and it stops ticking. We build airplanes like a cheap alarm clock. But knock it off the table and still it wakes you up.”

Ben Rich thought by lowering quality - for example making aircraft tyres that are good for 10 landings, instead of the 1000 that the airforce mandated - could be used to save money overall, even though you would throw tyres away more often, because the cost of those disposable tyres would be many orders of magnitude lower. This reminds me of uptime requirements in hosting - you can get 99% up-time for $5 a month. 99.99% might cost you a few hundred dollars. 99.99999% might cost you millions. Similarly many developers design for scalability targets that are driven by optimism and vanity, rather than based in reality.

Final Thoughts

The term “skunk works” has rightly become synonymous with astounding breakthroughs created by brilliant engineering teams, freed from bureaucracy. While us software developers have to be content with making a page load faster, or a data store be resilient in the face of failures these guys and gals - armed mostly with slide rules - were hacking the laws of physics, chemistry and materials science. It’s humbling to hear of their successes and learn from their failures.

We are defined not by the technologies we create but the process in which we create them. –Kelly Johnson

Footnotes

[1] The USSR eventually shot down CIA pilot Francis Gary Powers in 1960 by simeltaneously firing 14 SA-2 missiles into his flight-path, which they were able to predict because a similar one had been used on a previous mission days before.

[2] On later projects such as the F-117 the amount of beuracracy, oversight and red-tape at the skunk works did increase, forced on them by the U.S. government, and is one of the things Ben Rich rallied against in his ‘suggestions’ towards the end of the book.