Dr. Dobb's Digest October 2009
Ivar Jacobson is a father of components and component architecture, use cases, aspect-oriented software development, modern business engineering, the Unified Modeling Language and the Rational Unified Process. Ian Spence is Chief Scientist at Ivar Jacobson International where he specializes in the agile application of the Unified Process. They can be contacted at a_theory@ivarjacobson.com
Bertrand Meyer and I recently presented an article entitled Methods Need Theory which outlined why we believe that methods need theory. In this article, Ian Spence and I expand on the foundation provided by the previous article to examine more detail at why we will all benefit from establishing such an underlying theory.
--IJ
Do we really know how to develop great software? The answer for many people is clearly yes. But do we know how to communicate and continuously improve the way that we develop software? Do we really understand the best way to communicate and share our knowledge? The answer, as we saw in the previous article is clearly no!
Do we stand on quicksand or the shoulders of giants?
Have you ever taken the time to investigate a new method or practice only to find that it is just the re-branding and regurgitation of ideas that you have seen many times before?
Have you ever got frustrated that every new idea about software development seems to be at the expense and in aggressive competition with everything that has gone before?
Does it seem to you that following that latest software development trend has become more important than producing great software?
Have you noticed how in their hurry to forge ahead people seem to throw away the good with the bad? Instead of learning from their own experience and building on top of the good things that they do, they heedlessly throw everything away and start again with something they believe is new. It is as though they have no solid knowledge to stand upon.
This behavior can be seen in the way that many teams carelessly discard expensive process and tool investments, almost before they have even tried them. Every project adopts a new approach. Every time someone changes their job they have to learn a new approach before they can get on with the real task at hand. This is not effective; people cannot learn from experience as they are forever starting over.
The bottom line is that nothing new ever becomes properly established -- even today after a number of "modern" software development trends the most popular approaches to software development are prescriptive waterfall development or uncontrolled hacking. As an industry we cannot get anything to stick and nothing really seems to change.
But now we're all Agile surely all our problems are solved?
The latest trend to sweep the industry is "being agile".
Now let's be quite clear here, the "agile" movement has made a very positive contribution to the software industry. (If you have seen any of our recent presentations at conferences around the world you know that we are both big fans of Agile and supporters of the Agile Manifesto.) It has reminded us that people matter first and foremost when developing software. This is not really new but it is important, and it did seem to have been neglected by the previous, more technically oriented, fashions such as object orientation, and programming in Java. By presenting a set of values the agile manifesto created something robust and resilient that can withstand the waves of change that will be bought on by the next trend. (A new trend is as sure to follow the agile trend as night is to follow day.)
It's a shame that the same cannot be said for the many agile methods that have promoted themselves as supporting the agile philosophy. For a movement that values people over process and tools it has certainly given us a lot of "new" processes and tools. The majority of these have been effective by putting the team back at the heart of what is done to develop software but in bringing this back into focus, much has been lost or obscured as new terms are introduced for old things, creating the illusion of something completely new.
This constant re-packaging and re-branding of old ideas produces a constant churn in the way that software development teams work. Rather than re-focusing peoples' efforts away from wasted work onto the production of quality software all that seems to happen is an arbitrary renaming of the things they do and produce.
Even with something as correct and beneficial as the agile philosophy the underlying message can get lost in the churn and hype. We are already starting to see a backlash against agile and our fear is that the benefits will be lost as early adopters move onto the next bandwagon and the late majority re-assert their right not to change to something that has obviously gone out of fashion.
All that seems to happen is that we add more buzz-words and conflicting terminology, and we are all becoming deadly tired of all the buzz!
Addressing the challenge: developing a basic theory describing what software engineering actually is
It is clear that we need to stop chasing after fads and easy answers that forever disappoint, and that we need to do it without discouraging innovation and the generation of new ideas. People need to stop constantly re-packaging and re-branding old ideas just for the sake of it. Instead they should focus on helping people understand how to build great software. But how can we facilitate this change in emphasis? In our opinion, this theory is right in front of us -- we just need to grab it.
First we should start with all the popular methods, processes, and practices out there and find the "truth" of software engineering. We can then describe and capture this small set of essential concepts in the form of a minimal, practice independent process -- we call this small set of essential things our "kernel". Then using this kernel as our starting point we can analyze existing processes and methods and identify the practices that they contain. By starting from the kernel we can describe the practices in a way that allows them to be compared and composed.
Now this way of producing our theory is not in its self a theory. This is something that we have already done. By studying a number of methods, including XP, Scrum, and the Unified Process our team has identified a kernel with some 20+ elements, things we always do or always produce. On the surface, there may appear to be large differences in these studied methods and the ways we work with them but under the covers they share the same DNA. As an example, you can capture requirements with features or with use cases or with user stories, and you can use these requirements without a lifecycle, with the Unified Process lifecycle or even (as some people still insist upon) a waterfall lifecycle. There is definitely a common basis for the many methods, which can be captured in the form of a small, simple set of kernel elements.
Now we are not presumptuous enough to propose that our kernel provides the needed theory. More and greater minds than ours are needed to do this. But we would present it as evidence of what can be done and that the theory we need is within reach.
This isn't just something that affects methodologists, process geeks, and academics; it is something that will benefit everybody involved in software development. But what difference will it really make?
What's in it for the industry?
Most large companies have their own homegrown method or process, one which is a soup of standard ideas complemented by some of their own more business specific ideas. Usually these processes are described in a thick book or a website, and lots of money has been invested to document them. Sometimes people are trained on the process, sometimes they are just pointed toward it.
In reality the process is often ignored; the only parts that are actually used are the parts that form the "oral tradition" of the organization. This is explained by the rediscovered law of nature: people don't read process books. New ideas are introduced into the organization, and the old processes fall out of fashion and their books become shelf-ware.
In these large companies there may even be more than one process. For example the big system integrators may have 10 or 20 different processes. Sometimes they are quite similar, but the similarities are hidden behind all the cosmetic differences.
If your company was to adopt the practice idea, you wouldn't need to throw out your entire way of working because something new and sexy is becoming popular. Instead you would improve your existing way of working one practice at a time. You could even adopt practices used by other companies without throwing out those existing practices that seem to be working well.
To get started you would look at your existing way of working as a set of practices. Then you would look for your pain points and complement your existing way of working by removing any practices that are not working and replacing them with practices that address the weaknesses. This is easy to do once you have understood the kernel and its usage.
In a large organization with many different ways of working, you can use this approach to successively improve each way of working without forcing everyone to use the same method or process.
This approach will make it easier to adopt new practices without having to change the other practices you have. Imagine that you have already introduced the kernel and described your practices a couple of years ago. Then you would be able to introduce Scrum easily by replacing your existing practice for project management by Scrum without having to make any major modifications to your other practices. Looking into the future, Scrum will most likely be replaced by a new practice, and you will be able to do that easily, too.
What's in it for the academic and educational community?
It would be excellent if our technical institutes or universities would educate students in the basics of software engineering, followed up by training the students in a set of good practices using that base. Education will become more logical since it focuses on individual ideas instead of the particular soup of ideas that forms every method, process, or methodology. I believe students will love it. There is also space for a lot of relevant research here.
Remember Kurt Lewin's words: "There is nothing as practical as a good theory." A good theory makes it easy to learn and develop your knowledge without going overly religious. That would be smart.
Most university professors have made an academic career and have never really had a chance to practice software development on a large scale. Still they have to teach software engineering, which of course is not easy or stimulating. They just do it because it is in the curriculum and not because they have something to contribute. They have no theory to teach, just a set of ideas or one specific approach. When challenged, one professor, who is successful in computer science and who teaches software engineering, said: "Amazingly, the students like to bathe in the mess we ship to them". I realize this was not serious, but for sure this teacher was not proud of what he was doing.
A theory will fundamentally change this situation. Students will learn the basics of software. They will get a language to communicate about software process, practices, patterns, etc. I can imagine that they will get a language with a grammar that represents the kernel and language constructs to describe the practices being constituents of the process. The language needs to be executable so that the practices can come alive. By this I mean that practices are not just specifications but also executables. When running a project, the practices will start to run and instances of activities, work products, roles with competencies will be created and populated by real things. Aspects seem to fit well to model practices. There are very interesting semantic rules that need to be identified and defined. There is whole new world that opens up to the students to understand the fundamentals of software engineering. Not to mention the whole new world for practically and theoretically interested researchers.
What's in it for the methodologists?
Thinking back on my own career from 1987, I was asked by many people to write a book on the methodology I created. At that time Objectory had a number of new ideas such as use cases, use case driven development which is a kind of test-driven design, collaborations, sequence diagrams, components and component based development. Most of the rest was not special. Implementation, unit testing, system test, performance test, configurations, planning was quite conventional. Of course, I had experience from the whole lifecycle, but I was not a world-class expert on everything. However, to write a book I had to cover the whole lifecycle, even if much of it was not my expertise.
With the new theory we are looking for, there will be no need to describe anything else than what is new. You won't need to write a book to publish your new ideas and put them in the context of all the other things a software development team needs to do. You will just describe your new practice or your new pattern and it will be available to the whole world the next day. Anybody in the whole world with great ideas can contribute and become successful.
What's in it for the teams developing the software?
Finally they will be able to escape from the endless churn caused by slavishly following fashions and become proper software engineering teams. Teams that build and extend their knowledge by practicing good software development based on a solid foundation, one that is not constantly changing and forcing you to learn the same things over and over again. One that lets you demonstrates your expertise by the results you've achieved rather than the courses you've attended. One that lets you bring in new ideas and new team mates easily and seamlessly without performance dips or wasted effort.
Finally teams will be able to continuously improve and adapt the way that they work to meet the challenges that they face on a day-to-day basis. They will be able to develop their knowledge and skills in a way that will enable them to work successfully with other people from different backgrounds, teams and organizations without having to continually relearn the same things over and over again.
Our understanding of software engineering lacks a basic theory. As a result, we keep reinventing old approaches with slightly different words, obscuring the real innovations and making it harder to use the good parts of the new while discarding the bad parts of the old. The theory will help us to substantially improve our education in software engineering. It will help us to be less naive in how we react to new ideas popping up around us. Finally, it will also help us to adopt new ideas faster than we can today.
The real beneficiary of this theory will be the software industry, as has already been proven in many companies. We will be able to easily educate our people, get them up to speed, improve the way we work with our products, reengineer (a stronger word than refactoring) our products systematically, and continually improve the way we work.
The result will be better software, significantly faster and at dramatically reduced costs. As mentioned above this is something we need to work on together. As seen from Scott Ambler's article The Theory Needs a Strategy, momentum is starting to build but there is still a lot to be done.
We have demonstrated that it works, but we still have much to do to establish an accepted standard. This has to be done by consensus building among a group of experts and authorities. We look forward to working with such experts.
To find out more about how you can get involved visit www.ivarblog.com and give your feedback to the blog "In need of a theory for software engineering" or e-mail us at a_theory@ivarjacobson.com.