A Discipline of Software Engineering

481

OVERTURE

Outline

INTRODUCTION TO OVERTURE

Chapter 1: The State of the House

INTRODUCTION TO OVERTURE

• If the System is a potato, then I don’t know what I’m writing about.

• If the System is a load of bricks, then you should not read this book

CHAPTER 1

The State of the House

Publisher Summary

Programming is an engineering craft, which is based upon a formal apparatus of great depth that has every chance of being complete and correct. Theory gives correctness proofs. But they are of complexity in all practical cases, and cannot be used. Composing a program is an act of creation that can be compared to that of solving a problem of geometry. In some way, the composer must see a potential solution to the problem on hand. An important aspect of software engineering is quality control. The quality control work is eased by an order of magnitude if a prototype is approved and when a solid design language and programming language is chosen and algorithms are well documented. A way is to use refinement: give a global functional description of which the phrases are refined into program sentences, thereby guaranteeing readability. It might be advisable to use data-driven practices rather than code-embedded ones, because such programs are more stable, and the external data structures, being very descriptive, also act as the documentation of the process.

Curtains for programming

Many still claim that programming is an art. Today it is more accurate to say that programming is an engineering craft which is based upon a formal apparatus of great depth, that has every chance of being complete and correct. But theory does not do the one thing that everyone expects: explain how one programs. True enough, theory gives correctness proofs. But they are of overwhelming complexity in all practical cases, and cannot be used. Composing a program is an act of creation which can be compared to that of solving a problem of geometry. In some way the composer must see, almost feel, a potential solution to the problem on hand. Next, he must fill in the necessary material which establishes the truth of his solution. Some people can do this, some can’t and never will be able to.

That programs have a structure is not at all surprising, but structure is not the target in itself. In fact it is the problem to be solved that contains the structure. Perception of the structure, which is a work of abstraction, is the creative act. And again, there are those whose mental powers allow them to see that structure, even when it is masked by problem specification of an intricate nature. For a long time, many people have believed that structured programming was going to help them. How wrong they were! Structured Programming is nothing more than style. And style alone does not allow the novelist to write a successful book: the story must be conceived before any point of style appears … It suffices to take a look at many programs to see what has happened: an effort to disguise bad concept (non-solutions in fact!) by flourishing texts, written in an extravagant style. Scratching only the merest bit reveals the emptiness of the program: many flag settings for later testing, merely avoiding non-structured writing, but not fundamentally solving the problem. Programs of today are just as error-prone as before. Just as difficult to maintain or expand.

An impressive number of authors have tried their best to improve things. They have invented methods. We must certainly give credit to the people who did so: their work was and remains extremely valuable. They, the visionaries, realized an important fact: a problem must be analyzed so as to make structure appear. The problem must be decomposed into more elementary structures that are somehow connected. And we should note that it is the problem, i.e. the real world, not the program, that must be so mastered. In almost all cases, the problem will appear to have an overall structure, but it becomes confused because of the number of exceptions. These must be analyzed just as deeply since they have an important influence upon the whole body. The authors therefore proposed disciplines, steps as it were, to do the work. But the major problem remains: analyzing (decomposing) an amount of work must have a starting point somewhere. It is not clear how to begin the decomposition. In other words, in order to do it, one needs a global insight into the structure from the start. Which completes the circle. Alas, no method gives that initial spark.

Moreover, methods are largely subjective: the methods proposed by the authors are those that fit their own intellectual approach. That particular author is very good at his particular method because that is the way his brain feels structure. And this is a process that cannot be transferred to other people. It can definitely not be taught. In a way methods have failed.

Fortunately, the application world abounds in paradigm situations. Quite a number of problems are more than similar, and a global solution certainly can be adapted to many particular situations. The systematic approach, via a method, at least prevents DP people from re-inventing the wheel. That alone, of course, is an interesting aspect. But whenever a totally new problem occurs, the programmers are at a loss. They have nothing to rely on: the whole thing is untouched wilderness in which roads will have to be cut without any means of orientation. This also explains the failure of relying upon experience. Experience induces duplicate solutions. A new problem will be solved along the lines used for all or some of the problems previously solved by that programmer. As a result, experience only helps when one remains in the same domain. It is no help for novelties … And, education is just as bad. First: not everyone can be trained to become a programmer. It is a story of haves and have-nots. But next, whatever education has brought is either specific to a given domain, and thus confines the student, or it is very general but then it has no depth so that it remains void … Moreover, many educational systems lay the accent upon rather trivial matters, idiosyncrasies of the tutors, so quite often biasing results.

There is another point that has considerably confused the issue: the abundance of programming languages. A very common mistake is the confusion between fluency in a language and aptitude at programming. Surely, a problem that gets solved must eventually be expressed in a programming language. The argument should however be: choose the language at the end of the conception process and select the one most indicated. Unfortunately this is never really allowed. A language is imposed by company standards (and usually it is COBOL, of all things!). As a result, the idiosyncrasies of the language affect the design in the earliest stage, since it is definitely not true that languages are general purpose (with the possible exception of ADA). Educating programmers so that they become good Cobolists (or anything else) is a gratuitous exercise and is harmful rather than solving anything.

believe that correct data specification is a key factor in success. Identification of problem data is certainly a large part of the analysis. Indeed, programs are data transformers, in other words they are transfer functions that can be described as the ratio between output data and input data. Therefore, data analysis is an essential aspect. Not only should one devote careful attention to the static structure of data (as enforced by data-base technology) but also the evolution of data must be taken into account. In that field also, errors are commonplace. Methods abound, but, again, they only help if the designer has the initial spark. Data structures are not there for the fun of it. They must express a reality. The same problem exists as for programming: normalization (i.e. structuring) of data is seen as the target. It should be the means instead.

In the methods jungle

Every DP professional uses the words method, methodology, technique and technology. These words have become very fashionable. But they serve to cover up one of the major crises of DP: a lack of solid tradition. After all, DP is still very young; there can not yet be traditions…

Let us be clear about the terminology: a method is a set of rules (a discipline as it were) which, when followed, will achieve a very precise (and predictable) result. The method may be facilitated by means of a tool that implements it¹. Methodology is the study of methods. The usage of the word as a synonym of method is utterly wrong. A technique is the sum of all methods, theories and knowledge that concern and build a well-delimited view of reality. Technology, then, is the study of all the techniques covering a consistent and closed part of reality. By metonymy, the word technology is also used to stand for the set of techniques. Thus, 4th generation development is a technology. Structured programming is a technique; Nasi-Shneiderman diagramming is a method to achieve structured programming. As usual, definitions like these, which are somewhat philosophical and not without emotion, are debatable. But I think the real point is the following: method = discipline.

In no other domain of technology² do we have as many methods as in DP. There is no way to compare them. Most of them are named by acronyms: SDM, LSDM, ISS, ISAAC, NIAM, … a new vocabulary, used in a way similar to car names. Although there is almost no difference between two cars of a same category, you’ll find that both have their totally convinced drivers. Emotion crept into the game. The users of one method explicitly despise those of other methods. Moreover, methods are made fashionable by containing the usage of brilliant techniques! See how it evolved: a method contains techniques! It should be the other way around … Methods have overshot their goal. Some of the best ones tend to confuse project management and development technology in one single (huge) set of commandments. If they could, they would tell you how to set up the furniture of your office! Most of the methods get clobbered by the incredible amount of documentation they produce. They become document-management systems, and get hopelessly lost in the volumes involved … I still have to meet an analyst who actually re-uses the documentation. I remember one particular DP project, of rather average size; its documentation covered 5600 pages (26 binders!). There were endless discussions about storing away this volume.

Methods have one advantage, though: everyone can use them. Just follow the rules. Don’t try to understand them! Many of the methods have killed creativity and reduced the demands upon human intelligence. Rather sad, this. Admittedly, there are methods that work in another way. They tend to channel creativity rather than crush it. Such methods are much more acceptable, but unfortunately, they are somewhat ill-considered because of the degrees of freedom they contain. In fact, the management feels a method is good when it is heavily prescriptive. Descriptive methods, on the other hand, are seen as rather worthless. Still, it is my belief that descriptive methods are much better in guiding the developer to a quality result. Such methods tell you what to do, but don’t impose how.

There is another point as well: whatever method is chosen (and everyone chooses a method), the issue is further obscured by standards. All entities used in a DP development trajectory, to start with fields, have names. As an example, a very obvious way to give a field a name is: ENM-0415C3-BW035. In more human terms, the field means employee-name. Don’t look for any cryptographic explanation, however. It is easier than that: ENM means Employee NaMe; 0415 is the identification of the record of which this is a field (this record would be called R0415, of course!); C3 indicates a computational-3 usage of the field (Cobol stuff); BW are the initials of the developer and 035 is the project for which this field was conceived. Obvious, no? Someone has clearly overlooked the fact that we have data dictionaries and repositories in which fields can be registered with a convivial name and attributes that further connect the field into its environment. A good dictionary will deliver any cross-reference required, just at the stroke of a key. Other standards: how is the address of a person recorded? 20 positions or 30 positions? House number before or after street name?

The common pitfall of standards is that they are too drastic, too involved. There is no incentive to apply them other than reprimand. One wonders: what is the true need for a standard? From the answer follows what the standard should then be, provided all other ways to achieve the same goal have been explored, and found unsatisfactory. It is my belief that the best effect standards can achieve is to avoid quality degeneration by imposing one choice at places where the development systems offer a multiple choice possibility. Something like:

– all relational tables should have a primary key (even if the system does not require this);

– there should be no sorting in on-line programs;

– for reasons of performance a 4th generation application module should not have more than approximately 400 code statements (notice the word approximately: a standard must be flexible)…

To cut a long and painful story short, I could cite a classic aphorism: It doesn’t matter which method you use, provided you use one. There is a lot of truth in this sentence. Indeed, known methods intend to achieve a stated goal. In essence, the method is guaranteed to reach that goal. So, if the goal is not debatable, the method is OK as far as I am concerned. However, use sound judgement along the way. The interpretation of the discipline should be indicative rather than imperative. Deviate where it is indicated. Allow for creativity. Having no method, on the other hand, means that one is going to re-invent the wheel, most obviously by trial and error. A time-consuming and demotivating process. Adopt a method, but don’t spend too much time in selecting one. I repeat: they are all acceptable (provided you throw away the heavy body of standards).

The worst one could do is see the method as a goal in itself. Unfortunately, this happens more often than not.

Not programming but software engineering

Creating a good program appears to be a rather delicate job. But making programs is not the only thing a DP professional undertakes. There is more, much more that has to be done at an earlier stage. Programming is the last part of the job.

In fact, what people create today are systems. In their technological implementation, systems are huge conglomerates of many programs that together deliver a solid and well-defined closed functionality. The making of such a system creates immense problems. The major one is in mastering the sheer size of the whole thing. Just imagine that an application of rather conventional nature takes something between 10,000 and 100,000 lines of (3rd generation) code! And that is small compared to some medical systems or NASA systems (tens of millions of lines of code!). Such volumes cannot be mastered by just one person any more. An accepted statistic states that one programmer produces 2000 lines of code in one year (from pre-study to accepted result). In order to solve this kind of problem, we need many people working together. And this is not a linear rule either: people have communication problems which increase quadratically with the number of people. A team of 10 people is considered a maximum. When a project is bigger than 10 people can manage to produce in a reasonable time interval, the project must be cut up in separate pieces and given to various teams. And this creates new communication problems. But even for a smaller subject, there is the problem of how one divides the project over the team members. This in itself is far from trivial, as we will see in the next section.

The situation further complicates because the business problem we try to automate is itself moderately to very complex, due to the great amount of detail one must master and the incredible number of exceptions that must be taken into account (many of them with related second-order effects). The result of such endeavours constitutes a package which does not have a linear behaviour: small changes in the code may induce disproportionate consequences in the outcome, so that one can never assume that safety margins are really obeyed. And if this weren’t enough, systems once finished start to evolve, either because one must correct errors they still contain (corrective maintenance), but mostly because the reality evolves so that the systems must be continuously adapted lest they become obsolete much too soon (perfective maintenance). Also modifications for the sake of performance must be brought into the system (tuning). As a result, there is a continuous drop in quality because of the unsound haste with which such corrections and adaptations have to be realized. Implemented systems obsolesce very fast…

The discipline that groups all techniques and methods used to create a computerized system is called software engineering. Unfortunately, software engineering has no scientific grounds whatsoever. Apart from planning methods that allow a project leader to keep (some) track of what is currently happening, there is no method guaranteeing a quality result. At present, therefore, the only way is to achieve quality control at every moment of the implementation. This is done by continuous prototyping (evolutive prototyping) and keeping the knowledgeable end-user tightly involved. Nevertheless, managing a software development project is one of the most nerve-wrecking jobs ever.

The fact that most budget makers have unbelievably unrealistic requirements as to deadlines, does not help at all.

The life-cycle

The creation of a software system (even a system comprising only one program, however simple) is a work which proceeds in phases, not unlike the phases we find in, for instance, automobile industry when a new car type must be developed. In the automobile analogy we can certainly recognize three major phases:

– a phase of study, where the needs for the new type, its constraints, the wish list and some state of the art considerations are put together; this is the analysis phase;

– a second phase during which the new model is conceived, plans are made, prototypes and models are built and tried; this is the design phase;

– the last phase, when the first production car of the new type is assembled; this is the implementation phase.

Although the phases of software development are not exactly identical in detail, we find the three same general steps: analysis, design and implementation.

1 The analysis phase

A problem to be solved needs a clear problem statement. Usually however, this is missing or is formulated in the vaguest of ways. Therefore, it is required to push the investigation: what is really needed? How is the problem handled today (i.e. before automation)? Is there a problem solution (even if only in principle) without the need for a computer? Is this solution adequate on a computer also? Can it be improved by computer usage? If there is no exact manual solution, is there an approximate (or heuristic or good-sense) solution? Can this solution be computer-based?

In order to answer these questions, one has to go and capture the reality, in the world of every day³. In a business application this can be a straightforward activity, although it often involves a vast amount of work. Most of such applications are a link in an information flow network. Information coming from the outside world is input and results in the creation of transformed and usable information, which in turn serves as input for yet another process. Thus, investigation of the functional information flow in the company’s departments is the key to this part of the work. This will require interviews, observation and quite a lot of communication psychology. The result is a set of specifications, usually (but not necessarily) expressed as a set of input values (possibly with an involved structure), corresponding to a set of output values. In this light, an automated package is a transfer function, creating a mapping between the input and the output set.

If the analysis work is to be complete, the model of information flow should be analyzed so as to find unproductive situations: processes that are never executed, data that serves no one, loops that do not really produce anything. Formal methods such as Petri networks (and its derivatives such as event diagrams) may be used here.

The mental process needed during analysis is one of abstraction or, put more pragmatically, generalization of the detail information gathered by investigation. Here it is of paramount importance to perceive the fundamental structure in the mass of detail (and exception) cases. The information needed by the processes (and external to them) must also be investigated, so as to detect its static structure. It is important at this stage to perceive the data structure as independent from the programs: it pre-exists by essence and should therefore be represented in a natural, application-independent way.

The major part of the analysis work relies upon a carefully conducted dialogue with the concerned (and hopefully strongly involved) end user. Such a dialogue is difficult, even under ideal conditions, because the end user is not very apt at expressing abstractions. A way to structure the dialogue is by breaking the functions down into manipulations of user views. This is certainly a good thing to do. But being able to build a prototype, even at this early stage, would help even more. A prototype is a quick and easy (hopefully not too dirty) mixture of means that emulates the problem solution (preferably on-line) and shows the user what functionality can be expected. The prototype is a truly operational object, and will serve to refine both the user’s and the analyst’s perception of the problem. It is precisely by using fourth generation techniques in association with relational data-bases and a number of tools (such as query languages (e.g. SQL, report writers, screen mappers, spreadsheets, graphics,… not to forget data dictionaries) that prototypes can be realized, the cost of them becoming justifiable.

The analysis stage ends with the production of a set of specifications and requirements. It also produces some models: a process model representing the activities with their information-carrying user views such as take place in the application domain, an information model which represents the major entities carrying the information, their fields and their inter-relationships and finally an organization model representing the relationships between people and the company’s activities. It should also deliver a choice of solution outlines.

In some connotations, the analysis phase is called study phase, conceptual level⁴, enterprise level. A question of what’s in a name.

2 The solution design and data structure design phase

Having established the problem specifications and requirements, one now determines a solution. In essence, the work proceeds from the models obtained during analysis, by chosing where choice exists, by filling in detail where detail is missing, by correcting erroneous situations where they exist. More specifically, the design phase will represent the solution in some normalized model, that is a model which obeys easy and strictly enforceable rules. This model will emphasize the functions which will be offered to the users of the application, the user interface (screens, reports,…) as it will manifest itself to the user. It will also indicate the new information flow. Where appropriate, standards will be formulated: for instance in order to achieve a consistent user interface throughout the system.

The design of the fine data structures that must hold the long term information constitutes half of the design work: programs become less complicated, more visible, when the data structures are correct.

The methods used for designing a software system are essentially the same as those that can be used for the analysis. It is a matter of shifted accent.

IBM’s AD-cycle methodology speaks about an enterprise level (perception of a software system as it is articulated in the enterprise reality), a design level (how does it work), a technology level (how is it built).

3 Program Implementation phase

This phase contains two strongly connected activities. One is the creation of the algorithms⁵ and of the physical representation of the data-base (if any). The other is the actual writing of the programs and the setting up of the data-base (if any). While on-line algorithms in business DP are usually quite simple, algorithm creation for batch processes calls on all the craftsmanship of the programmer. A great amount of knowledge is required, especially of typical solutions to a vast number of situations. However, if the problem to be solved has been decomposed deeply enough and the associate data structures and resources have been well described, most algorithms will usually be mere transducers and fairly easy to establish. The more difficult ones tend to be isolated; it is not every day that a programmer must solve Hanoi tower problems.

On the other hand, if the problem to be solved is in a specific area, such as text processing or syntax handling, there are sufficient well-documented solution profiles in literature. Writing the algorithms should be straightforward.

There is another important aspect: notwithstanding the claims of formalists, the algorithm’s profile depends on the chosen programming language. This is a rather sensitive field: instead of allowing programmers to choose the most appropriate language for a given algorithm, there is usually a company-imposed language such as COBOL. There may be economic arguments for this (although I strongly doubt their validity) but it is a truly sad fact. No language is perfect, so that the mere choice of a language entails the need to use a number of tricks. The major responsibility of the programmer is to produce highly readable code which espouses the algorithm as closely as possible. This is a matter of producing good syntax: bear in mind all the syntactic structuring items the language offers and use them in the cleanest of ways. Don’t misuse a structure to simulate something with! Good programming standards (but do they exist?) are certainly helpful. A rule of thumb is to isolate black boxes in procedures which are replaceable as a whole (or paragraphs in COBOL). This, incidentally, is the gist (in reverse) of stepwise refinement. And don’t forget the programming of the accesses to information (pragmatically: data access). Acceptance of and commitment to the concept of data independence is an absolute must: a programmer should never go physical on data. Instead, use SQL or something similar (and better). And don’t ever forget this basic truth: structured programming is not a goal in itself; a program must be structured so as to reflect (in some way) the structure of the objects it works with (data, knowledge, events, information as it were); structures in the syntax (connectives and the likes) are only a way to divide and conquer, but of course, you must know what to divide and how to divide