Sustainable Software Requires Deep Agility & Expertise

There are many ways to obtain software. An “agile” process or “rock star coders” are not enough to create long-lived software assets that increase in value as business needs evolve.

With over 35 years in the software industry, I am no longer amazed but rather saddened, at how often unrealistic and poor choices are made in software investments. Many established businesses want to buy versus build whenever software is mentioned. There are many good Software as a Service offerings to handle commoditized processes for small-to-medium businesses and larger enterprises. There are certain types of software (e.g. marketing and simple self-serve websites) that can be built effectively, quickly and relatively inexpensively by someone who is proficient at a particular platform (e.g. WordPress).
Yet, many businesses have at least one way they differentiate themselves from their competition that is not easily served through commercial off-the-shelf (COTS) software or customizable platforms. They believe they have some expertise that their competition does not. Yet, they shy away from investing in unique software that can capture their unique expertise and leverage it in ways that only software can and settle for “me, too” mediocrity (or worse).

Many business leaders want to believe that this expertise they’ve spent enormous amounts of cumulative time and money in gaining can be converted into software in a way they would never treat any other capital investment. Perhaps they have read about agile processes and think it will help them avoid catastrophe, but try to exercise it with unproven talent. Or they hire a bright person or two — or maybe even a big name company — without really digging into their track record to see if they’ve really ever delivered software that leverages expertise in an effective way for the long haul.

I hope that this article will help others avoid big mistakes in the investment of custom software. Whether that investment is on the order of tens of thousands, hundreds of thousands, or millions of dollars, it should get a return that exceeds the investment. If done well, that should be an outsized return that continues to increase in value over time: a sustainable software asset that is a strategic advantage, rather than a frustrating liability that never delivered on its potential.

Sustainable Software Explained

There are many kinds of software ranging from simple to complex and short-lived to long-lived. At first glance, without giving the differences much thought by the naive, the differences may not be obvious. However, an “information only, publicly accessible” website is radically different from a business application that extracts and codifies deep domain expertise and provides a delightful user experience for secure creation and management of complex concepts which require a level of essential compliance to significant rules.

But the parts of software beyond what a user sees on the screen are mostly invisible and a mystery to many. But let me offer an analogy that may be easier to understand yet extremely similar in many ways.
A simple dog house and a custom-built home may have some resemblances. However, the inherent complexities of the custom-built house require a different approach and set of skills to produce, though some of the same skills used in building a dog house may be useful in building a custom home. While you might safely employ a handy teenager to build a dog house with little risk to meet the basic goals of a dry shelter for your dog, knowing that the cost of future replacement is minimal, you wouldn’t think of asking the same person to design and build your custom dream house that you plan to be an intergenerational home with much more complex desires.

One can build a dog house in a “quick and dirty” way for tens or hundreds of dollars in parts and labor, ignoring the effects of moisture and weather, and it will meet the immediate requirements. For hundreds or thousands of dollars in parts and (more skilled) labor, a premium dog house might be built with much more care to stand the test of time, dressed up with a lot of unusual features. That premium dog house, though, still doesn’t begin to compare to the simplest of custom homes. Again, what it takes to build the simplest and smallest of homes versus the most majestic dream homes will vary greatly in both process and required skills, though some of the same skills used in the simpler home can be employed in building the dream home.

The two biggest factors in determining both the cost and skills needed for building projects are complexity and expected longevity. And many have learned the hard way when they’ve hired people who lacked skills or integrity to do the job well.

Software is much like building in that basic way, though much more adaptive to change if done well. Any significant software investment will also need maintenance in various ways, just like any house that is lived in or subject to the environment will over time.

There’s a bit of a fuzzy line between consumable software and software investments. But the key difference is a mindset. The “consumable” mindset is more of a sprint, producing as much as possible to meet immediate needs without a lot of thought to whether the software being produced is sustainable… Will it scale? Can it be easily changed or extended while there are active users? Are there hidden bugs swept under the rug that could cause significant future damage? Is there a suite of regression tests that prevent bugs from popping up when someone changes something in the code? Beyond the inherent complexity of the problem space, how much complexity has been introduced to the codebase that makes it harder to approach?

The Importance of Sustainable Software Expertise

The sad thing is that many software developers, software product managers and stakeholders don’t recognize the things that impede the sustainability of software because they are really Advanced Beginners with respect to building sustainable software. The Dreyfus Model of Skill Acquisition and studies since then have identified the different levels of expertise and the implications of each.

It is easy to find a Novice and many do. We are all Novices at something to which we may have been barely introduced. For example, if you have had an introduction to HTML or programming, gotten a degree in IT or Computer Science, or been to a “coding bootcamp” you are probably still a Novice software developer. You have not actually developed any software beyond the software equivalent of a simple dog house. If you have ever been involved in software that is actually used by someone else on a regular basis, you may have moved on to being an “Advanced Beginner” (whether or not you have spent much time actually supporting the software you were involved in creating). Novices and Advanced Beginners often hire Advanced Beginners at significant pay rates.

Unfortunately for a Novice, anyone who seems to know more than them looks like an expert. But this is just not the case. And an Advanced Beginner without the proper perspective of where they are on the learning curve may think they are farther along than they actually are. There is so much that goes into building sustainable software than just adding some new features. Every team member has different levels of skill with languages, tools, general problem solving, and the understanding of the specific domain. The implications of the application of those skills can have significant impacts, both positive and negative.

Luciano Passuello points out, “There’s much more to mastering a skill than just acquiring more knowledge. Just like adults are not simply bigger children, experts are not only smarter, more knowledgeable or faster than Novices. The differences can be found at a more fundamental level, such as in how they perceive the world and approach problems.”

In Andrew Hunt’s “Pragmatic Thinking and Learning”, he introduces the Dreyfus model and points out that many people never make it beyond Advanced Beginner. This resonates with my experience of 35 years in the software industry. I’ve often been amazed at the surface-level skills demonstrated by someone who has been in the industry for even ten or twenty years. I’ve also often been disappointed to see bright people who don’t shine in their early introduction to programming or a new software development skill who don’t put in much effort to really get it.

My hypothesis is that no one likes feeling clueless, which is where all Novices begin. Some stay Novices because they let their fear of looking and feeling clueless drive them towards activities in which they are more comfortable. But, if they get over that fear (whether through a nurturing mentor or inner resolve) they end up being productive at an Advanced Beginner level. If their new productivity level is readily recognized by others, they begin to feel comfortable again. This often keeps them from pushing through the next, and often larger, barrier to true competence… or at least broader competence.
It is like the Advanced Beginner carpenter who becomes a framer who has learned that if they “put enough nails in it and it will hold”, and then rely on the finish carpenter to cover it up. When I was building my house a master carpenter pointed out the saying, “a little putty, a little paint, makes that carpenter what he ain’t”. There are many good framers that you should never want to hire to do anything else in your house. As long as you recognize what they are good at and what they are not so good at, you will be fine.

In residential housing, the “inspection process” has two real steps. The first step is done (well or not so well) by the person in charge (supervisor, general contractor, et al) and the resulting quality varies by the skills and standards of the person in charge and those doing the work. They are held to a minimum standard by the independent building inspector who knows the rules that have been built up over the years to greatly reduce the amount of bad stuff that might happen in the longer run. Those rules have probably been produced by competent people so even Novices can follow them. If the framing job was done in a week by a novice who had to tear out and reattach every other board, or a couple of hours by an experienced framer, it passes as long as the end result “will hold.”
The rules rarely take into account the expertise a proficient or expert builder may have that may transcend the rules in a given context, but at least they help produce “competent” buildings. Unlike in the world of residential housing, the process of “quality inspection” in software is generally lacking. If the software seems to work, it often ships. The “weight bearing” nature of the code is not given much thought, and bad data entry, or the simplest perceived change might grind the system to a halt, though the root problem might be difficult for other Novices or Advanced Beginners to find and correct. This could be analogous to giving a certificate of occupancy to someone because the walls sitting on a shaky foundation currently seem to be holding up the roof. And when the storm comes, and the house sits in a pile of rubble, the root causes are many, but the poor homeowner has to deal with the consequences and the “professionals” have moved on to other jobs.

Using Advanced Beginners may be perfectly fine for putting together a quick and dirty demo. It can be disastrous for building a system that needs to constantly evolve to meet the needs of its eventual users and stakeholders. An Advanced Beginner doesn’t know what they don’t know. We often see people with 1–4 years of experience calling themselves “rock stars”. We wonder whether the “Rock Star” analogy is a good one for Advanced Beginners. At a superficial level, they seem impressive, but their experience is often shallow. They might focus on appearances and entertaining with the “magic” they wield. Perhaps they build simple demos at amazing speeds. Don’t confuse speed of coding with real progress.

I would challenge investors in start-ups to consider how much of the high failure rate has to do with bad business models, and how much has to do with the make up and approach of their development team. Often one leads to the other, e.g. not enough funding to hire the right development team. But in the infamous words of Red Adair, “If you think it’s expensive to hire a professional to do the job, wait until you hire an amateur.”

I’ve seen many start-ups whose development team was mostly made up of recent graduates of college or coding bootcamps, or retreads from other failed start-ups. In addition to this collection of Novices and Advanced Beginners, they may even have an experienced developer who is Competent or even Proficient. However, they are often so busy meeting with other principals or customers as the Chief Technology Officer or System Architect, that they actually spend very little time in the actual development of the software. These start-ups end up “shipping a demo” which couldn’t satisfy the demand of real users who could have benefitted from a more solidly built piece of software that adapted to the dynamic needs of the target market. Without real customers who realize the real benefit the start-up wanted to deliver, the funding dries up and the “pile of software rubble” is just as useful as a house built carelessly on a poor foundation.

And, by the way, this is not just a problem in start-ups. In some corners of the industry, it is even worse when it comes to internally built software. I know of one company in the insurance industry that has 95% or more of its developers having less than 5 years of development experience. Since management is rewarded significantly better, it is rare to find a Proficient developer anywhere in the company. The cost of maintaining software in that particular company is huge with the number of QA people far outweighing the number of developers. It should go without saying that this company is not known for its innovation in technology or customer satisfaction.

What really matters in building sustainable software effectively and efficiently can be read about in books, but must be proven by actually building, supporting, and growing software that can continue to be supported and grown by others effectively and efficiently.

There are patterns that have been proven to work well in various contexts, and the contexts have to be recognized in real time. There is rarely a single context in any software project. Although there are various catalogs of patterns, it takes a lot of discernment to know what the true context is and whether the pattern fits or the context requires a new or different pattern to be effective. There is no “required reading” in most places where software development is done. Even when there is, without a guide as to when and where to apply what is being read, the application is often unchecked. What might be a powerful technique in certain contexts actually introduces complexity in others. Unlike in most physical building projects, a software foundation can also be malleable and built up as the rest of the hopefully well-built, yet malleable software is built up. But that doesn’t happen when Novices and Advanced Beginners are the builders.

It has been acknowledged by many who have been involved in sustainable software that code is read more often than it is written. Depending on the longevity of the development, the amount code is read (in order to figure out what and where changes need to be made) is greatly increased. If the code is more complex than it needs to be, the impact is huge. If an expert can keep complexity in the code to only what is inherent (and managed well), navigating through the code is simple. But, if it is a jumbled mess, navigation can be a nightmare. And if it is navigated by someone who isn’t very proficient, they are most likely to either reproduce bad patterns or introduce new ones, continuing to introduce more complexity.

Just like poorly built physical buildings, the best thing to do with poorly built software is to demolish it, and start over. Very few people can stomach that. But woe to those who, instead of demolishing it and starting over, try to salvage it and “remodel”. There may be times it is the right thing to do, but there will never be a time that it will be an inexpensive thing to do.

Growing Expertise

At RoleModel, we recognize that producing software is more than just coding and getting cool looking stuff to show up on a screen. (Though, we certainly know how to do that). We know that someone with three years of experience, no matter how well-trained, has not had the opportunity to learn in enough contexts to become an expert in software development. There is a progression of skills that need to be learned and proven in order to become truly proficient or an expert in the context of building software assets. The basic skills range from adding well-defined features in a test-driven manner with intention revealing names to debugging difficult production problems and removing the root cause(s). The more advanced skills range from turning ill-defined requirements into helpful features that users and stakeholders love to determining how to navigate the politics of the company and industry you serve in order to provide real value to both. The level of expertise needed to tackle features or problems is dependent on the inherent complexity of the project as a whole.

If you have properly identified the level of expertise needed to complete a task without significant risk of introduced complexity, everyone does not need to be an expert. Instead you just need to identify the levels of expertise on your team, and assign tasks appropriately.

And, there is a level of expertise needed to identify and manage inherent complexity that may not be obvious to those who do not have the appropriate level of expertise.

Our core values of Character, Collaboration, and Craftsmanship have driven our development of a culture of continuous improvement and service to our customers. We recognize the value of staying humble, not believing our opinion is more important than user feedback. We learn the skills to extract useful feedback from users and stakeholders and only work on the things that provide the highest business value and/or delight the users who may not be like us. We also need to learn from the feedback of those who have produced software before and recognize good and bad patterns in code and approaches to code to avoid introduced complexity.

Only after our developers have mastered all the skills necessary, and proven themselves in real world development, do we recognize them as Craftsmen. This can be a process that takes five to fifteen years or longer. But time doesn’t necessarily make a Craftsman. We’ve made our Craftsmanship Levels Templates available to the industry, and update the scores of our team members every six months or so based on demonstrated results, not just potential. We don’t even call someone a Developer until they have shown competence in our best development practices on software projects (usually internal). It doesn’t matter what degree they might have, we want to see effective delivery. And we want to help them continuously grow in their expertise.

Agility

I was a pioneer in the application of agility and even co-authored the highly acclaimed book “eXtreme Programming Applied” with our own Roy Miller. If used well, agile practices provide feedback loops at all levels of a project to help continuously refine requirements, design, plan, results, and people! But, as Andy Hunt points out in Pragmatic Thinking & Learning, “Proficient practitioners can take full advantage of the reflection and feedback that is core to agile methods.” A corollary to that is that Novices and Advanced Beginners cannot. Although there is never anything wrong with reflection and feedback, two Novices don’t produce expert quality by pair programming.

Some think “agile” = “SCRUM”

But, many who claim to be “agile” are missing a lot of important feedback loops if they just use SCRUM. Some of the practices originally suggested by Kent Beck through eXtreme Programming and refined by many in the industry today are not being employed. These practices matter more, the more inherently complex and long-lived one’s project may be.
The Customer Feedback loop is so important, and I’ll address this in another article soon. But a team that doesn’t have the expertise to take that customer feedback and doesn’t use practices that build it into a sustainable software system is only going to be so effective.

But, many who claim to be “agile” are missing a lot of important feedback loops if they just use SCRUM. Some of the practices originally suggested by Kent Beck through eXtreme Programming and refined by many in the industry today are not being employed. These practices matter more, the more inherently complex and long-lived one’s project may be.

The Customer Feedback loop is so important, and I’ll address this in another article soon. But a team that doesn’t have the expertise to take that customer feedback and doesn’t use practices that build it into a sustainable software system is only going to be so effective.

Yes, there are times when Pair Programming is not the optimal solution to getting two sets of eyes on all production code. However, there are at least as many times when asynchronous pull requests, popularized by Github and its variants, isn’t the optimal solution either. And who should we rely on to know the difference? Certainly not a Novice or Advanced Beginner. The code will only rise to the level of the best developer involved in creating the solution. If you want Craftsman-level code, you’ll need a Craftsman involved in order to keep quality high, and especially to keep introduced complexity low.

A Craftsman Centered Approach

At RoleModel, we basically have three ways we approach projects:

  • Craftsman-focused
  • Craftsman-led
  • Craftsman-supervised

These types of projects often correspond with the various phases of our proven approach of build-measure-learn each step of the way

  • Conceptualize — count the cost before starting a project collaborating with stakeholders to make sure we are on the same page.
  • Explore — identify key aspects of the system and mitigate high risks through various forms of prototypes and experiments and build a Test-Driven Walking Skeleton upon which we can start fleshing out the application with the most useful features.
  • Craft/Rollout — build the software with a continuous delivery of business value getting feedback from users, subject matter experts, software craftsmen, and stakeholders all along the way.
  • Sustain — when the desires and efforts to put in new features dies down and the full-time development team is moving on to other things, continuously monitor what is happening with the system and its users making important upgrades and adjustments.

Conceptualize & Explore

Craftsman-focused approaches occur when the inherent complexity and risks needs to be identified and a development plan sorted out. Sorting through this requires a high level of expertise to avoid introducing complexity or missing important challenges. We might put multiple Craftsmen and Sr. Designers/Developers on it. This is almost always true at the beginning of a project and, depending on the inherent complexity, it may continue to be necessary to have a higher percentage of Craftsmen into the Craft/Rollout phase, but usually, enough has been sorted out to proceed as a Craftsmen-led project.

Craft/Rollout

Once it is clear what we’re building, where the inherent complexity is, less expert developers with good sustainable software habits can be brought in.
Craftsmen-led approaches typically have one craftsman allocated full-time and other developers/designers of various levels. (The ratio of Craftsmen to less experienced developers will change based on the inherent complexity of the project). The development plan is typically fairly well sorted out, and the Craftsman is there to handle the hardest problems and make sure all the other contributors don’t unwittingly introduce complexity.

Craftsman-supervised projects typically have a small team of appropriate skill level with a Craftsman who checks in and has their back. These are typically projects that are not adding significantly complex features and the work often includes making minor adjustments.

At RoleModel, a new software system often starts as a craftsman-focused project and evolves into a craftsmen-led project and then possibly a craftsman-supervised project when the major features are in place. Proficient and Expert developers use the feedback they get throughout the software development process to refactor code to increase clarity, performance, quality, supportability, and extensibility. Doing it this way minimizes the amount of introduced complexity. As new opportunities arise, new or simplified features can be added with confidence. The cost/benefit analysis isn’t encumbered by an inordinate amount of “technical debt” that needs to be paid down in order to move forward.

However, we have seen that software systems are sometimes created in the opposite way. An Advanced Beginner may start a project and “get it to work” and the system that goes into production is full of introduced complexity and has not addressed some challenges. We’ve had potential clients bring these kind of systems to RoleModel hoping we can “finish it” or “fix a few problems” who are surprised to hear that they’ll need experts for an extended period of time to sort through the mess. Instead of being in a place where they can easily grow the system, they are in a very scary position of finding out their system is on a shaky foundation that needs to be carefully engineered to avoid breaking what appears to be working. The money “saved” up front is completely lost as the amount of “technical debt” that needs to be paid off is huge. Think of it as paying 30% interest on your credit card while you are struggling to make your minimum monthly payments and you are already at your credit limit.

Sustain

Our Supportability Scorecard is a tool we use when software we build goes into production and we continually revise it. We verify that we have all the important monitoring tools, tests, and documentation in place in order to get a good to excellent score. When new features are desired, we can move it forward if the original developers are not readily available. This can be done with a Craftsman-supervised approach for simple features or in a Craftsman-focused approach if managing the inherent complexity needs more expertise.

If someone brings us software that we didn’t create (or perhaps we created an earlier version and someone else took it from there with less care to important details) we would first score it and then give the client an estimate of what it would take to get it up to a passing (or better) score. Then, we analyze what the available monitoring tools are telling us (along with some manual inspection) and help the client understand their risks in both keeping the system going or improving it as they might come to us complaining about some problems their users are having.

We prioritize getting the system up to a sustainable score and determining a plan to help move the system forward in both sustainability and meeting new customer demands. The scariest situation is when automated tests are missing. The danger of breaking something (or a lot of things) are huge, and the effort to make sure no one does is very expensive… manual tests of the entire system or the chance that existing users will not be able to perform what the software was supposed to deliver. We’ll slowly build up automated system tests until we can reduce the number of necessary manual tests to a reasonable level of risk.

Agility and Expertise from Concept to Production

There is a wide variety of software and many ways to approach the building of it. Through years of honing our craft, building software systems that stand the test of time, and observing the consequences of poor practices or under-skilled software leadership we’ve come to recognize the value of both agility and expertise throughout the software lifecycle.
Using the word “agile” and even doing some of the accepted practices is not enough.

If you are looking to invest in what you hope will become a long-lived software asset, make sure you have both agility AND expertise. If your so-called experts haven’t supported software in production and don’t advocate a set of automated tests, preferably test-driven, be afraid. Be very afraid.