Software projects are often problematic. They take a long time, are expensive, need a lot of maintenance, tend to have bugs in it, and quite often the final result does not work according to the original expectations: intuitive to work with, without any failures or bugs, and fast.
The elements of software are small and many
Software consists of lines of code. A software package like Microsoft Office contains millions and millions of lines of code. Not all of it is handwritten (some code can be generated by computers), but a lot is. To compare with: a book of 300 pages contains around 10 thousand lines of text. So, to write a software application like Microsoft Word, you need to write at least a thousand books in computer language. Not only is this a lot of work, every line has to be correct as well, a failure in a line could potentially disable the whole software program.
Software depends on hardware
Software needs a computer or a server. It uses computer memory, disk space, network connections. Any failure here will stop the software from working, at least partially. Good practice is to have backup hardware. The complication here is that you need software that manages an automatic transition to backup hardware in case of a hardware failure. So software that makes sure the principal software will still run in case of hardware failures. Complicated, right? Yes, and therefore sensitive to errors.
Another factor that relates to the hardware dependency is the fact that hardware developments go fast. It is not unusual to replace hardware every 4 years with more powerful and faster hardware. Of course, this has consequences for the software as well.
Software depends on other software
I just mentioned this point already: most software is not standalone: it depends on other software. Most commonly it depends on an operating system. An operating system is a software package that operates the hardware the software will run on. Windows is an operating system, and Linux as well. Operating systems are among the largest software packages you will find. It consists of billions of lines of code.
Failures in the operating system could have consequences for the functioning of a software program that depends on it. Upgrades of the operating system (all operating systems need frequent upgrades for security or other reasons) also might affect the working of a software program that relies on it.
Similar to developments in hardware, software developments go quite fast as well. Operating systems are replaced about every 5 years, having consequences for all software that relies on it.
Software applications are prone to functional changes
Software is very much related to social and work processes: for working processes you can think of applications like Enterprise Resource Planning (ERP) or Customer Relationship Management (CRM), for social processes world wide known examples are LinkedIn and Facebook. Any kind of social or work process is dynamic, and therefore software will need to keep up with this dynamics. New requirements for software appear frequently, which causes most software to be in a continuous state of change. What does this mean? It means software should always be built in such way that it can be adjusted relatively easy. What makes this a very difficult task is the fact that it is unknown what the future adjustments will be like exactly.
Building a good software application with fixed requirements is already quite difficult, building it in a way that adjustments are easily possible is even more difficult.
95% of a software program is invisible, and people don’t realize this
The visible part of a software program is called the user interface. However, most of the program is not visible: it’s behind the screen. Complex architecture and an enormous amount of code lines are hidden from the user of the program. At first sight, a program may look simple, however, behind the screens it can be very complex. People don’t realize this, even managers of software projects don’t always have a clue about the complexity of a software program.
This becomes a problem when decisions or estimates have to be made regarding a software project. Budget estimates, project schedule, skilled people to be hired. If one does not understand the complexity of a software program, it is easy to make judgmental errors, resulting in problems of several kind (projects getting out of budget, instable software, etc.).
Many of the problems in software projects are related to this: decision making by people with insufficient knowledge or insight.
Anyone can develop software, very few are able to develop good software
Anyone with a computer is able to do software development. Software development tools are available on the Internet, most of them for free. Nowadays it’s quite easy to build your own website for instance.
However, as the size of the software application increases, the learning curve and the skills needed to be a successful software developer, do rapidly grow. To be able to work within an environment of millions of lines of code, you need to know about software architecture and design patterns, and know how to work with testing frameworks, for instance.
Not all software developers have this knowledge and these skills, and it shows, efficiency and productivity among software developers and architects differ by a factor 20.
Good developers are hard to recognize by people that do not have software development knowledge themselves (like most recruiters), so it happens quite frequently that software developers with insufficient skills are assigned to ambitious software projects, resulting in problematic software development.
All together, many factors why software development is difficult and can become problematic. To a certain extent, things can always be improved, for instance by improving IT knowledge of people in and around the software development process. Another option would be standardization. However, this contradicts in a way the power of software development: its flexibility. There are no physical boundaries, possibilities are endless: software can be developed totally tailor made.
The counter side of this flexibility may be clear by now: low level of standardization and intransparent complexity. My tip on how to control this should also be clear: skills and knowledge!