In the previous two articles in this series we have tiptoed around the discipline of objective knowledge. How can system administration be kept within the boundaries of science and protected from the spectre of personal opinion? The answer was straightforward: the scientific method, i.e. a mixture of inspired inquiry and exacting self-critcism. To round up the series, I would like to present a number of challenges to the system administration community, challenges which indicate a few of the important problem areas we have to examine in the coming years. This article will be of a more speculative nature than the previous pieces, with the aim of stimulating activity in the field; the list will not be complete, but you can fill in the blanks yourself. Speculation provides the creative impetus for a rational inquiry. System administration touches on a wide realm of topics; it is loaded with technical and social issues; it requires creativity and analysis, and we know very little about the empirical (verifiable, factual) basis for the subject. In short, it is a jungle waiting to be explored (ref 1).Theoretical
Perhaps the greatest challenge for system administration is the development of theoretical models. For some theory is a dirty word: it signifies the ivory towers of irrelevancy: charmless academics who have never gotten their hands dirty, preaching to the workers from their soap-boxes. But this popular affectation is unfair. No field of study exists without a theoretical framework, a frame of reference--and you can be sure that no one gets into system administration without getting their hands a little dirty. The challenge of theory is to come up with ideas which can be tested by the kind of rational enquiry that I have been discussing in this series. If one can build models, they can be tested by simulations or by experiment. Indeed, without a framework for inquiry, experiments are often meaningless.As I explained in the last issue, there is no sense in collecting data and trying to guess what they might mean. You have to start with a question and then see what the data tell you about the question. Analysing data blind is rather like like Schiaparelli looking at Mars through a blurry telescope and seeing canali, which became even more blurrily interpreted as The Canals of Mars. It is easy to see things which are not there, if you don't have some theory to interpret what you see. I have repeatedly emphasized that theoretical work for system administration is about building models which link cause and effect. In some cases that is easy, in others it is complex. Computers are dynamical systems. One of the first tasks then is to develop models of competition for resources (ref 2,3).
Take, for example, the immunity model which I presented at LISA 98 (ref 1). This is based on the idea that a computer systems has an ideal state. Recently I showed how such a state can be defined from a given system policy by mapping the average condition of a computer system onto a `state space', or lattice of possible states, where the origin was defined to be the ideal (ref. 3). As faults build up in the system, the actual state gets farther from the ideal, moving through the lattice. This is a now a precise model, not just words.
There is more to do. I did not show that the definition was the only possible definition. We need to know how the system migrates through the space of states in order to detect anomalous or harmful behaviour. Can certain regions of state space be classified into ok and dangerous? Is there a continuous blend, or are there rigid boundaries? The answer to these questions would be of enormous importance to anomaly detection, and thus also security. Is threshold behaviour the answer, or something more subtle? Can immunological models be built based on costimulation: a kind of lock and key signalling which switches on immune responses in the body only for a very specific signature (two or more complementary keys are required for confirmation).
Empirical
Experiment and theory are symbiotes, feeding off one another in a constant dialogue. Just looking for trends in data can lead to canali. To test theories one needs to collect data and see how well those data support or contradict the theories. The data feed back into models and refine them. An important issue here is scale - finding the right scale for right problem. As in the examples from part 2 in this series, the first step in analysing a problem is to determine the time-scale at which it occurs. Problems therefore need to be classified by scale. There is no sense in measuring variations once a week if you are trying to see how users behave, since the time-scale relevant to users is minutes, not weeks.Sometimes it happens that problems arise interpreting data. If the measured data are troublesome, measure something different. For instance, many researchers have measured process lifetimes, or the lifetimes of network events. These fall into divergent distributions, since some processes or events hang or never die. This causes a problem. Solution? Try looking at a different variable, e.g. the number of events, a quantity which cannot easily diverge.
Experimentation is already tied to many issues like anomaly detection, time series analysis. Use of neural networks is common, but neural networks are not well understood. Try to keep within the bounds of what you understand. Studies of reliability, effectiveness of certain policies, determination of what the relevant quantities are to measure in specific instances, dependency in causal webs. The possibilities are almost endless.
Psychological
The behaviour of users is central to the behaviour of computer systems. Users are as much a part of the system as the programs they manipulate. Can user behaviour be understood in any convenient way? There is almost certainly a relevant literature on this subject in connection with user-interface design and other issues. It needs to be traced and examined. It will not be completely relevant to understanding how users interact in network communities, but it will pave the way forward and inspire new work.What shall we measure? Can user behaviour be classified into groupings with particular characteristics which are useful when defining system policy? Psychological behaviour can be useful in several circumstances: predicting user behaviour in a crisis, adapive security measures. Do users fit profiles? Is it possible, and if so useful to put users into categories? Should the categories be different for different system policy models? What are the ethical issues here? Can profiles be kept and stored in a form which is only meaningful to the computer system, which cannot be examined or used by humans for scurrilous purposes?
Conflicts of interest develop in any community where limited resources have to be shared by several players. How do such conflicts develop, what are ways of extinguishing them before they escalate? What is the effect of trust and privlege between users and the system? Does increased trust and privilege lead to fewer conflicts, or does it simply open for abuse? I do not know whether studies of this kind have been performed in other fields of research. Certainly the results have not been applied and made available in the field of system administration. Note how this final point is related to the issue of `security through obscurity'- if one attempts to conceal something, it is like stamping something TOP SECRET in flashing neon lights.
Technological
Research into smart/safe algorithms which implement aspects of theoretical models will lead to an improvement in basic technology. For example, I have personally worked on the idea of `convergence to the ideal state', using cfengine as an embodiment of the immunity model. Load balancing or resource sharing can be implemented algorithmically. Network middleware already creates technological layers which help to standardize these issues for network services. Can similar middleware technologies (like cfengine) assist with the core problems of system administration?Many problems in system administration would be solved more readily if there were standard log file formats, or sharing mechanisms which allowed programs to share log data without having to reparse log files. Today, programs which want to know about the log output of other programs (e.g. intrusion detection programs) have to parse each others' log files. This is a meaningless waste of effort, made necessary by inadequate infra-structure. A signalling nexus for trusted processes to share their state information would assist in the task of anomaly detection and security issues.
Changes in operating system technology can have a huge effect on the ease with which system administration is conducted. Often system administration is a work-around for operating system problems: fixing symptoms rather than cause. Operating system technology at present pays little attention to the issues of system administration. If anything Unix-like operating systems are taking steps backwards in trying to emulate Windows (as if that were the standard for operating system design), giving users privileges which they should not have, in order to make things `easy'. The careful analysis of operating system technology in relation to system administration is long overdue.
How will encryption affect the issues of system administration? If system processes cannot detect threatening programs or data files because they are encrypted, they cannot protect against them. For instance, if system policy dictates that huge mp3 files should be deleted after 24 hours, but a user conceals these by encryption, tyhe policy cannot be policed . This is reminiscent of the attitude of governments towards wiretapping. It is not as straightforward an ethical or technical issue as some would like it to be. It represents a conflict of interests.
Load analysis
This is the area where the most work has already been done. Internet traffic, process lifetimes, capacity planning. High performance and real-time computing issues for specialized problems. If you are going to analyse load, ask yourself why? What are you going to do with the data? What are you trying to show?Databases
Bioinformatics is the new discipline in biology related to the creation and interpretation of databases of genetic and protein data. This a crucial sharing of important empirical data about the mechanisms of developmental biology, virus signatures and disease markers. Creating and maintaining important public databases is equally important in the computing community (e.g. DNS, RBL, virus signatures). What new databases do ew need? How will such databases be managed? What about trust and security issues relating to the database contents?The very first database which I would like to see created is a database of references to system administration research. Without such a library of what has been done, we will be doomed to repeat earlier work. Such a database would be a valuable resource to the field. At Oslo, we have plans to set up such a database in BibTeX format.
The shape of windows to come?
The day to day tasks of system administration change constantly and we pay these changes little attention. However, improvements in technology always lead to changing work practices, as humans are replaced by machinery in those jobs which are menial and repetitive. The core principles of system administration will remain the same, but the job description of the system manager will be rather different. In many ways, the day to day business of system administration consists of just a few recipes which slowly evolve over time.It is difficult to articulate just why the administration of computer communities is an exciting challenge, but if we are to succeed in pushing through programmes of research which will bring about the level of automation we require, then it will be necessary to attract willing researchers. Fortunately, today there is a high proportion of system administrators with scientific backgrounds with the will and training to undertake such work. However, only the surface has been scratched. The tendency has been to produce tools rather than to investigate concepts, and while the tools are necessary, they must not become an end in themselves. A clearer understanding of the problems we face, looking forward, will only be achieved with more analytical work.
In many ways system administration is like biology. Animals are machines, just billions of times more complex than our own creations, but the gap is closing and will continue to close as we enter into an era of quantum and biological computing techniques. The essence of experimental observation, and of the complex phenomena and interrelationships between hosts is directly analogous to what one does in biology. We may have created computers, but that does not mean that we understand them implicitly. In our field, we are still watching the animals do their thing, trying to learn.
Acknowledgement
Parts of this article have been adapted from the author's new book: Principles of Network and System Administration (J. Wiley & Sons) ISBN 0471823031, by kind permission of the publishers.References
To keep this list short, I have given only a few pointers. For a full list of references, please look to the sources below.
- M. Burgess, Computer Immunology, LISA 1998
- N. Glance, T.Hogg, and B.A. Huberman. Computational ecosystems in a changing environment. International Journal of Modern Physics, C2:735, 1991.
- M. Burgess, On the theory of system administration, submitted to Journal of the ACM
- M. Burgess, Principles of Network and System Administration (J. Wiley & Sons) ISBN 0471823031 (Chichester 2000)
- http://www.iu.hio.no/SystemAdmin/scisa.html At this location, you will find many references and pointers to the above. I would like to build a full list of key references to central issues. You can contribute to this database by sending references to work which you feel to be important.