How will the wars of the future be fought, and who is heading these advancements in technology? Back in 2017, the US Air Force created a program called Kessel Run, which aids war fighters in the realms of DevOps, Agile, and UX, and the head of this project was an analyst by the name of Adam Furtado. In February of 2021, we interviewed Adam on the Dev Interrupted Podcast and shortly afterward hosted an AMA on our community Discord server.
Adam is the Chief of Platform at Kessel Run, and his story of how he almost single handedly led the US Air Force from 1970's software delivery methods to modern DevOps is one of the most incredible episodes of Dev Interrupted we've had. Adam talks about translating engineering to military officials and how he had to shift his mindset from application development to creating a system of systems. Listen to the episode here:
This Community AMA took place on February 26, 2021 on the Dev Interrupted Discord.
Necco-LB: 📢📢Community AMA📢📢 @everyone
Topic: Scaling Agile & DevOps
We're getting started in 15-minutes! Adam Furtado joins us to share his experience and expertise in scaling his organization (Kessel Run) from 5 >> 200+ developers!
Necco-LB: Let's get this thing started! @here
Welcome to our little community Adam! I can honestly say your episode of Dev Interrupted this week was one of the most interesting episodes I've produced.
Adam Furtado: Thanks for having me! I'm happy to hear that. Fighter jets are inherently cool.
Necco-LB: I don't think anyone can argue with that. To start things off, Adam can you give the community some quick context about Kessel Run? How many developers in your organization, what you’re building, etc.
Adam: Sure thing, KR is an Air Force organization proving that government-led software development will lead to better mission outcomes than outsourcing our software to companies that specialize in building airplanes (and using the same processes for their software). We build applications for warfighters to more efficiently strategize, plan, execute, task and assess the complexities of air campaigns. We have grown to about 1300 people… I’d guess. 400 of those are developers.
luisfernandezbr: Adam. What are the top 5 tech/dev metrics that you consider important to measure on a dev team? (Not product metrics like MAU, MRR).
Adam: I think they change as an organization changes... but for the most part I love the DORA 4... I think when used properly (and together!) it can tell you quite a bit about where you need to invest in your organization. The relationships between the metrics are what drives the value and I think often get forgotten about.
Necco-LB: Were you looking at different things (metrics or ways of visualizing work) when KR was smaller vs today?
Adam: For sure. I led most of our app development at first and we were/are an XP shop. Our teams were always very diligent about pulling the next story from the top of the backlog etc., so we never really had a WIP problem. When I moved over to lead our platform org, they were using a poorly-executed pseudo-scrum model and all of a sudden all of the DeGrandis/Kersten/Kim stuff I have been reading my whole career started to make a ton more sense. In building internal services, it was amazing to be able to see why work visualization matters and SEE the constraints building up. I'm so glad that I made the switch to build the empathy needed to be a more effective leader.
Necco-LB: Sounds like a big jump indeed. I have to say I’m wicked curious about how software development is different from within the military vs. the corporate environments most of us know.
Adam: Traditionally, the DoD was a case study in poor waterfall dev. Years of requirement development by people very removed from the work, leading to a contract being put in place that could only feasibly be won by a big defense contractor, years of development to "deliver" the "finished" software to be tested by separate government test organizations for a year or so and then "fielded" manually by folks traveling around the world putting CDs in machines. We've proven that all that risk avoidance actually INCREASES risk and we've had it backwards all along. To biggest thing we focused on early was how to reach Continuous Delivery with the heavy GRC requirements that we have in Defense (and rightfully so). So we worked with a forward-leaning IT leader in the Air Force to create and pilot the first Continuous Authority to Operate in the DoD. So instead of an approval to deploy to classified systems at the "end", we got our processes approved so everything coming out of our org was approved to go into those production environments. That’s prob the most unique thing.
luisfernandezbr: How you measure the evolution of your dev teams? And what initiatives and practices you use to grow them (like Dojo's etc)? What content do you recommend about DeGrandis/Kersten/Kim?
Adam: Deploy Frequency, Lead Time, Mean Time to Restore and Change/Fail Rate... Accelerate is the bible on this one (Forsgren, Humble, Kim)... Phoenix and Unicorn Project for Gene Kim's take on how to transform IT to DevOps approaches in big, slow companies... Making Work Visible by Domenica DeGrandis is a fantastic book on understanding what keeps us for being as productive as possible.... Mik Kersten's Project to Product on increasing flow. There are a ton of others, but that's a good start.
Read the unedited AMA and join in the discussion in the Dev Interrupted Discord here! With over 2000 members, the Dev Interrupted Discord Community is the best place for Engineering Leaders to engage in daily conversation. Join the community >>
drdwilcox: Thanks for joining us. During the podcast episode you talked about gaining momentum with some early wins. How did you keep that momentum going?
Adam: We struggled there, to be honest. The early wins were so much easier to "sell" to stakeholders. "There wasn't an app before... now there is. So I'm impressed". The first year we were deploying MVPs left and right and were the Belle of the ball. However in building large scale systems, we started focusing a lot more on our infrastructure, data models, optimization, internally efficiencies... and things that were providing real value- but weren't as visible. Those things are a lot less interesting to stakeholders. The Government has a very output-centric approach to value. We have focused on building an outcome-driven organization, so there is always a conflict when discussing what is or isn't valuable.
drdwilcox: I don't think it's just the government, to be honest. I have the same struggles in my private company. Output as defined by Product are sexy, all the other things are not. What was effective for you in getting the stakeholders re-engaged?
Adam: It's still a work in progress, to be honest. We constantly harp on the risk of NOT transforming in this way. The 2018 National Defense Strategy hits on this hard and all of our Senior leaders are pushing the same message. So that has been really helpful. General Brown, AF Chief of Staff, has done a great job of being clear about where we need to drive, so that allows us a bit of a trump card when we come into contact with someone who is trying to hold back progress.
Necco-LB: That idea of selling to stakeholders is really interesting, especially in the military. What did you have to say or do to convince your higher-up that the counter-intuitive dev methodologies like releasing more frequently was worth a try?
Adam: We had no support early on. The incentive process in the military rewards people who follow the rules and work within the system. We sort of worked quietly off to the side on a project nobody really cared about to prove the value once delivered. Once we got that delivered… we had MASSIVE dollar savings, so we started to be loud about it. In fact, we were told not to use the “Kessel Run” moniker by higher ups… we decided to do it anyway and started promoting pretty hard. By the time our first FastCompany article came out, all those senior leaders changed their tune and now they will say they were supporters all along. I am constantly doing that translation/evangelism work. And in the military, people swap out of positions every year or so generally, so some new person will get dropped in with no idea what’s going on and we need to start over again. “Nope, the cloud is a real place…” Continuous Delivery has broken every gov process. The test community doesn’t know what to do or look for… Requirement Managers don’t understand their place. Configuration Management is just…. Different now. I don’t need some guy managing a spreadsheet of what versions of software are deployed where. We are in a weird transition period right now. In a lot of ways those stakeholders are sick of hearing from me. I'm sure they hear Charlie-Brown-teacher-voice when I try to discuss this stuff at this point. So we have worked on finding the proper champions in higher up places to do that work for us. The very top of the Air Force totally gets it. It's everyone in between who need to keep their head down to keep rising in the ranks. (Tale as old as time...)
Necco-LB: Geez, what a thing. I can't believe you have the energy to continuously fight these battles within your own organization.
Read more about Kessel Run and smuggling DevOps into the Department of Defense
Adam: Someone once told me a story about this dad who brought his family to the beach. They had this big, pink inflatable bunny that the kids were using in the water. Every so often the bunny would deflate and the kids would run back up the beach to the dad and he would huff and puff and blow it back up. Kids would be happy and go back in the water. An hour later, kids are back again and the dad is blowing it back up. This person said "that is what innovation in the government is like". Every once in awhile you need someone or something to "pump up your bunny". The work I do is so exciting and fulfilling, all the BS that I have to deal with, all the money that government employees are leaving on the table, the bureaucracy ends up being worth it.
Necco-LB: That is a great analogy. Reminds me of how you talked about the mission driven culture at KR on the podcast. Can you talk about why you believe the culture of your organization is so important? And any advice you might have for organizations who are bifurcated?
Adam: Organizational alignment is incredibly important. One place we have struggled is that we put such an emphasis on teams, that teams built strong individual identities. They were empowered to solve their problem, but over time became less concerned about other teams' problems. This was never more evident than working with the ops/platform teams. The app teams knew what their users needed and all they cared about was meeting their needs. Meanwhile, we had a whole organization with organizational outcomes that were the priority. Let to a lack of empathy across teams and the communicate at the seams of teams was challenging. We are still digging ourselves out of that, but one thing we focus on is that mission-driven culture. All it takes is a day like yesterday, with airstrikes in Syria, to level-set everyone on the seriousness of our work. The mission aligns the teams towards a common goal and common outcomes.
luisfernandezbr: Adam. Thanks for the great tips. What were the big challenges that you had when increasing the dev team? Things like knowledge sharing, share learning and maintain quality and excellence. Could you share some tips about this, if it is the case?
Adam: We sucked at all those things. That mission-driven culture led us down the unenviable path about feeling so much pressure to deliver and support our users, that tech debt mounted and documentation suffered. We struggled investing in automation in favor of getting short term wins. The last year we have really rebalanced and ensured that we are providing space for our teams to organize their time better. None of it was intentional, but regardless of what we said, we (leadership) were giving off the vibe that teams couldn't possibly slow down to invest in tech debt or spend time focusing on automating toil away. We have had to be super clear that it is EXPECTED that teams work at a sustainable pace, invest in their code bases, invest in their professional growth and personal health and be okay saying "no". We have a lot of military members on our team, so saying "no" to your superiors is always a culture change we have to work on internally.
Necco-LB: Working on technical dept and automation vs. new features is something everyone can relate with for sure. I think we'll let Adam get back to his far more important job at Kessel Run. One last question, if someone here wants to get involved with Kessel Run, where can they go? Should they reach out to you?
Adam: This was fun- thanks for having me. You can follow me on Twitter at @adamsfurtado or you can reach out directly at afurtado@kr.af.mil. To follow along with KR, you can follow @kesselrunAF on most social media platform. I'd also like to plug that we are currently hiring for a bunch of roles from product leadership to engineers. It's an incredible place to work and you can make a real impact. Please take a look and reach out to me with any questions! Thanks again!
Antonette: Job opportunities at Kessel Run here: grnh.se/3201d1713us
Starved for top-level software engineering content? Need some good tips on how to manage your team? This AMA is based on an episode of Dev Interrupted - the go-to podcast for engineering leaders.
Dev Interrupted features expert guests from around the world to explore strategy and day-to-day topics ranging from dev team metrics to accelerating delivery. With new guests every week from Google to small startups, the Dev Interrupted Podcast is a fresh look at the world of software engineering and engineering management.
Listen and subscribe on your streaming service of choice today.
