Issue No.03 - July-September (2004 vol.26)
Published by the IEEE Computer Society
Laurie Robertson , Virginia Tech
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/MAHC.2004.3
Relocation bits, which became necessary when modules could be compiled separately and placed in a library for integration at runtime, are still relevant today. Two letters from 1961 demonstrate the kinds of analysis that led to the evolution of relocation bits in the early years.
The earliest programs, or routines, in the 1950s, were written in assembler language and were usually conceived and written as single, large modules. Later, subprograms, or subroutines, were created to modularize functions within larger programs. Programmers then realized that they could save these modules in libraries and call them whenever they were needed. These library routines included mathematical functions, input and output routines, sorting routines, and so on.
The main program, or starting program, needed to call on the library routines to execute, and the library routines needed to call each other, so it was necessary for each to know where the others were stored at the time of program execution. At first, programmers needed to carefully plan and lay out the program to avoid overlapping these routines in storage prior to execution. With the advent of higher level language translators, such as Fortran, storage allocation was delegated first to the translator and then to the operating system. Originally, Fortran required all the modules to be translated at the same time, so the translator would have all the information available (such as the modules' length, which ones called on which others, the types of parameters, and so forth) to lay them out successfully.
With the arrival of Fortran II in the early 1960s, the creation of subroutine libraries, and the need to accommodate other language translators, the task of loading and linking the various modules was delegated to the operating system. Loading and linking were the processes of bringing in a program's modules and the required library routines and arranging for each to know where the others were in storage. Usually, each collection of data and instructions would follow a set of bits, called relocation bits, which described the machine words that followed. The bits contained information about the module name, the amount of required storage, the amount of relocation (from the module's origin) to be associated with each subsequent data or address field, and so on.
As operating systems became more elaborate, and as virtual storage was introduced, more of the references to other modules became symbolic, separate from any particular storage medium or organization. This underlying idea of loading and linking independent modules, created and translated at various times (perhaps even during the execution of the program in question), is still relevant today.
The problem was how to compile a program as if it were loaded at one address, then run it after it had been loaded at a different address. Programmers initially solved this problem by compiling the program as if it were loaded at location zero, then associating with it a bit map (perhaps compressed to save space), which specified for each possible location or field where an address might occur in the program, whether that particular location contained an address. If the bit was on, the program's loading address was added to the field value; if the bit was off, the field was left alone. For example, on the IBM 704 and its derivatives, each word had two possible locations for an address, so there were two bits per word indicating whether the address or decrement fields should be relocated. Hence, the name relocation bits.
A relevant exchange
Tom Martin of Westinghouse Electric Corporation and Doug Eastwood at Bell Telephone Laboratories were members of the SHARE Fortran Committee, 1 which was trying to standardize the use of relocation bits across various IBM products. The first letter I provide in this article is between them, dated 3 April 1961. The letter discusses a recent proposal made by IB (the SHARE symbol for IBM) to the committee, the history of relocation bits, the reason for various evolutionary decisions, and the new IBM proposal's impact. The scheme Martin describes was the most ambitious plan to date.
April 3, 1961
Mr. Douglas EastwoodMathematical Research DepartmentBell Telephone LaboratoriesMurray Hill, New Jersey
During the recent SHARE Meeting in San Francisco, IB presented to the FORTRAN Committee, a proposed new relocation scheme patterned after the suggestions you made in New York during January. Many of us on the Committee were very enthused about the way IB took up this proposal; and, after what must have been a good bit of study, came up with a scheme which will remove many of the deficiencies imposed by, or inherently contained in, the present BSS loading method. Further, the new blocked COMMON which will be handled by the proposed loader brings back the power of a multiple entry origin table to a large extent. I believe IB is to be congratulated for arriving at a proposal which will remove inconsistencies contained in the present methods even though this will force recompilation and reassembly of many existing programs and general use subroutines because of the obvious object level incompatibility.
As you remember, some of us, at the suggestion of Bernie Galler, met for a short while to discuss IB's new relocation proposal. As I understood it, the motivations for this meeting were that it was known that some other systems and some installations had particular needs which it was thought would not be served by the new proposal, and further, that some of us felt that we shouldn't allow our enthusiasm over the benefits to be gained to lead us to adopt a system which did not provide for possible future expansions which might be brought about by the advance of the art. At this meeting, I made the beginning of some suggestions which I promised to describe in more detail. This letter is written for that purpose.
As a starting point, I think it would be useful to investigate previously implemented relocatable formats and loading procedures, comment on their advantages and inadequacies, and then see whether the new IB proposal meets the objections to the previous systems and possibly whether it introduces any new objections. The writer is certainly not familiar with the loading schemes of all systems so there can be no claim to a complete survey; however, I feel that enough can be set down to provoke some thought on the subject.
The original relocatable [punched] card format as presently described in the green loose leaf "SHARE Reference Manual for the IBM 704" had the usual loading control information in the 9L word, 36 bit checksum in 9R, relocation bits in the 8's row, and data words in rows 7-12. (It should be noted that the new IB proposal will rule out the use of the punch in bit 2 of the 9L word to cause checksum verification to be bypassed since a three prefix will denote a relocatable card containing data to be loaded into one of the COMMON regions.) The relocation bit patterns allowed for three possibilities; namely, 0 for an absolute field, 10 for relocation direct of a field, and 11 for relocation complementary. The decrements and addresses of the data words were each separate fields. If, at the time any one record is being loaded, there is only one relocation amount to be considered (i.e. if the origin table is limited to one entry), the only objection to this system is that it is possible to construct a card which will require more relocation bits than the capacity of the 8's row to describe the information on the card. This situation, which will be referred to as objection I, is present to some extent in almost all relocatable formats. In most cases, and certainly in the present case, the usual distribution of information in a record does not cause the above described condition.
In almost any useful application of the relocatable card format, it is found necessary to employ multiple origin table entries. The origin table is a means of specifying a partition of the set of integers (from 0 to 32767 on SHARE machines) and associating each section of the partition with one or more relocation amounts. The card described in the SHARE 704 Manual under the title "Origin Table" allows the introduction of up to eleven entries in an origin table, each entry of which will associate a reference origin with a pair called the loading origin and the operating origin. In what must have been a widely used relocatable loader (PKCSB4), the origin table had a capacity of over thirty entries which could be partially or completely modified at any time between unit card records. With this loader, a field (loading address, or data word decrement or address) could be relocated by an amount determined by the block specified by the origin table into which it fell. If it were a loading address, reference would be made to the loading origin table, otherwise the operating origin.
It should be noted that the PKCSB4 loader did a very incomplete job of referencing the origin table; the card loading address was located in the origin table, it was then relocated by the current amount specified by the loading origin table and all data fields were relocated by amount specified by the corresponding operating origin amount. This method leads to very rapid loading at the expense of radically reduced utility of the possibilities inherent in the relocatable format.
Two other loaders of which the writer knows (PKCSB3 and WHCSB4) were developed to overcome the above objection. In these loaders, the loading address for every word to be loaded was referred separately to the loading origin table, and both decrement and address of each data word were referred separately to the operating origin table for modification. Some very dramatic results can be obtained from such a complete utilization of the relocatable format; certainly this capability is necessary for any modify and load scheme that allows insertions at load time and operates in terms of relocatable records.
In order to effect any dramatic results in the above described system, the programmer had to put in a good bit of effort in planning the deck setup for a particular run. The FORTRAN method of controlling relocation which will be described later was able to make the job of deck setup very simple indeed at the expense of removing a great deal of facility. It is my hope that a scheme can be worked out to allow this ease of loading for the run of the mill job, but to also allow sufficient control to be retained for special cases in which it is useful.
With the introduction of the multiple entry origin table described above, a very serious objection was encountered, i.e. the occasion of an addend on a relocatable symbol causing a field to appear to fall outside the correct segment of core as specified by the reference origin table. A typical example of this situation will be found in the following example (see Figure 1).
This situation (incomplete field specification capability) will be referred to as Objection II.
With the advent of the original FORTRAN for the 704, a new concept in relocatable loading was introduced. In this scheme, the origin table card was replaced by the program card which specified the length of the subprogram which followed it. The loader allowed what corresponds to a two entry reference origin table (one entry being zero; the other, the program length or program break as it is called). The relocation amount for the first section was the sum of an initial value and the program breaks of previously loaded routines; for the second section, i.e. values greater or equal to the program break, an amount, usually zero, could be specified on the program card. It can be seen that flexibility of the origin table was sacrificed for ease of loading in this case. Although the positive aspect (ease of loading specification) should be emphasized, the case of a restricted origin table will be referred to as Objection III.
The authors of the FORTRAN relocation scheme realized the situation described above as Objection II; and with this in mind, the relocation bit scheme for the 8's row of the relocatable data card was changed so that the bits had the following significance: 0 indicated an absolute field; 10 indicated that the field as punched should be compared to the program break; and if it were less than this amount, relocation would be referred to the low entry in the two entry origin table, otherwise, to the upper entry; 11 indicated that the results of the above described test should be reversed. Relocation complementary was abandoned. One now notes that a change of the program break (which was commonly done at many installations to admit patches) can cause a deck to be misinterpreted. The situation of dependence of data to the origin table in this manner will be referred to as Objection IV while the loss of complementary relocation will be called Objection V. The BSS loading scheme issued with FORTRAN II, while it introduced the new concept of an external symbol in the transfer vector which has been found to be extremely useful, did nothing to meet Objections III, IV, or V. The current and historic FORTRAN loading methods then have created III, IV, and V and removed II.
As an aside comment, it seems fair to say that when the FORTRAN loading schemes were conceived, the authors considered that they had a closed system and could adopt any conventions that they desired so long as these conventions met their needs. The fallacy of this reasoning has been demonstrated quite sufficiently. I don't believe that such a closed policy applied to a system which expects wide distribution can be objected to in terms which are too strong. One must keep the implementation of a system in mind but should not abandon useful ideas or dictate against expansion to facilitate this implementation.
This, I believe, covers the various relocation schemes which have been implemented on SHARE machines. There are variations on the above working on other hardware; but I don't believe they indicate any new concepts to be considered. I understand that the Commercial Translator relocation scheme will contain an ability to make reference to a table built into the loader. I can see that this would be advantageous in the case where it is usual to make symbolic references to specific systems functions which may for one reason or another change location. This is, perhaps, an ability to be kept in mind if we are trying to establish a format acceptable to other major systems.
The new loading scheme described in the document titled "7090 FORTRAN Preliminary External Specifications" meets Objections IV and V completely and goes a long way to meeting Objection III. In the proposed new blocked COMMON idea, in which there are an unlimited number of origin table entries, every field will be completely specified as to reference without a comparison to the program break; furthermore, complementary relocation for any field, referred to any block, will be possible.
The only really major omission from this scheme is the ability to specify a separate relocation amount for the loading address. I realize that it is quite useful to have the loader take over the job of making room for various subprograms as it encounters them, but it has been maintained by others and myself that such a scheme cannot anticipate all of the needs that various installations may find necessary. As a very primitive example of what I am driving at, it would be very nice for the routine (FPT) to cause a trap transfer to its entry point to be loaded into absolute location 8. It can be seen that this would require a card with an absolute loading address containing one word with an absolute decrement and an address relocatable direct by the current non COMMON amount at the time of loading (FPT). I can also visualize relocatable library routines being written which would require certain parameters from other relocatable routines to be loaded into specific locations in the original routine.
With reference to the second paragraph, it was stated that provision should also be made to allow for expansion of the system. It can be argued that if we clutter up the current system with bells and whistles for contingencies which haven't even been dreamed of yet, we will be reducing the efficiency of current practice for what might well turn out to be no gain. However, it does not seem too much of a price to pay to set aside one of the remote possibilities of relocation bit patterns for this purpose. Actually, from one interpretation of the material on Page 33 of the 7090 FORTRAN Preliminary External Specifications, it could be inferred that there are three possibilities of bit configuration which will not be used.
At any rate, let me proceed to document some possible extensions to IB's new system, and then I can hope that this will at least stir up some comment. If these extensions are implemented by a loader which refers the loading address for every word and each field of the word separately to the origin table, the modification possibilities at load time will be quite good. It might be pointed out that with such a system, the implementation of the debug package on the 704 would have been quite simple since it would not have been necessary to have the STR instruction to introduce a break point. Instead, the program could be expanded by one word at each break point to allow the insertion of a transfer.
I shall detail most of the cards which will be used even though some are an exact duplication of the material contained in the copywrited International Business Machines Corporation document "7090 FORTRAN Preliminary External Specifications," March 1961. Permission for this reproduction was obtained from Harry Bekish.
An effort was made to stay strictly in line with the IB presentation, even though I have some reservation about some duplications of information contained therein; it will be noted that most of the new features are additions rather than changes. Indeed, I believe the only changes are the change of the 9L prefix on the COMMON Data Card to allow the standard use of the checksum negation bit and the restriction that COMMON block zero not be used. I imagine that IB intended to use block zero for "blank" COMMON although they don't specifically mention this. The system to be proposed would require that "blank" COMMON be denoted as block one. The difficulty referred to as Objection I will be raised with this system, but I would counter that this would be a small price to pay if it allowed the concurrent loading of subroutines produced by some of the other major systems now in existence.
Note that in the card formats described below, unspecified bit positions should be available for other use if future modifications dictate (see Figure 2).
The idea here is that the loader can build up an ordered list of subroutine names (to allow one and a half pass loading these routines must have previously been loaded). It will then be possible for a card, containing information to be relocated according to amounts in effect for the current programs, to specify that the loading address should be referred to the entry point of the nth entry in the above mentioned ordered list. From experiences using WHCSB4, I would guess that n would rarely exceed 1 or 2 when this facility is used.
For the standard relocatable binary card, there will be no change in format except for the relocation bit scheme of the 8's row (see Figure 3).
Two other card forms which are fairly well known and will not be described in detail should remain in consideration. These two are the 22 word/card absolute information card distinguished by a zero prefix and non-zero word count in word 1 and the symbol table card with a six prefix in word 1. I firmly believe that there cannot be an overemphasis on the necessity of retention of the symbol table with the relocatable binary deck. The Debug Subcommittee has barely scratched the surface of possibilities of putting this combination to use. At the recent SHARE Meeting, Fernando Corbato (MI) mentioned that he had come up with some very nice uses for just this combination. As the casual user is removed farther from the hardware by the introduction of compiler languages, it will be more mandatory that debugging information be made available to him in terms of the language in which he constructed his code. The symbol table makes this necessity a possibility. The absolute card format because of its compact specification of data and ease of loading will of course have to be retained also.
The relocation scheme proposed by IB is so complete and exact that I see no need to suggest any changes aside from the fact that the "zero" block could be reserved for future expansion and mixing of output from other systems. For example, in integrating Commercial Translator programs with FORTRAN programs, the 10 bit combination tagged by the indicator of the "zero" block could be the signal that the word in question had an address which referred to an entry, specified by the punched field, in the table of variable system symbols.
For the sake of completeness, I will include IB's description of the relocation scheme since some readers may not have a copy of the Preliminary External Specifications available (see Figure 4).
You will note that I have not specified a method for introducing an expansion ability into the middle of a particular subprogram. This was done intentionally since it seems to me that information concerning such a procedure should be supplied symbolically (probably by indirect reference to the symbol table). However, if this possibility is deemed necessary for inclusion on binary cards, the program loading card could be expanded by setting aside word 3 to specify the number, n, of break points of this type in the decrement and the number, m, of special program names in the address. Words 4, 5, …, 4 + n – 1 would then contain the address at which the break is to occur in the address and the amount of expansion in the decrement, while words 4 n, …, 4 + m + n – 2 would contain the ordered list of program names described above.
I'm a little sorry that this letter has turned out to be so lengthy; but, as long as there is going to be a change in binary format, I think that all possibilities should be considered. I would hope that this might provoke a few letters from other installations which have some feelings on the subject.
On rereading the various features I have suggested above, I get the feeling that a full use of all the relocation possibilities will require a very large loader and that this loader would require some pretty close coding to keep up to tape time particularly as tape reading becomes faster through the use of high speed units and blocked records. These are valid objections but should be weighed against the extra facility which can be realized.
As a final remark, let me say that I have certainly not included all the possibilities of actions which can be taken with the above described formats. I shall leave the derivation of all infinity of these as an exercise to what someone has called that oh so elusive interested reader.
Very truly yours,
Thomas W. MartinEngineer-in-ChargeSystems ProgrammingA.S.E. & A. Department
cc: Mr. Frank Engel, Jr.SHARE FORTRAN CommitteeMr. Bernard A. GallerMr. Morris AckroydMr. Harry Bekish
The second letter I provide in this article is from me to Tom Martin, suggesting a generalization of the proposal, in particular, to eliminate the distinction between program and data, as far as loading blocks of information into storage for execution. This was intended to simplify the scheme for relocation bits. This correspondence documents the state of module relocation at the time.
April 21, 1961
Mr. Thomas W. MartinA. S. E. and A. DepartmentWestinghouse Corporation
I thoroughly enjoyed reading your letter, and I would recommend that it go into the SSD, if only for its education value for newer members. Let me say, before I launch into some constructive criticism, that I endorse your proposals, and I certainly appreciate your careful work on behalf of our ad hoc committee.
I do have one suggestion which has rather philosophical roots, but which should lead to some practical benefits, such as a simpler loader, etc. I have thought for some time now that everything would become simpler both conceptually and operationally if we treated blocks of program and blocks of data exactly in the same way. Many of the things we do to data are also done to programs, and we are forever making special cases out of them. For example, your proposal for inclusion of a loading table for blocks of program emphasizes that IBM was again thinking only of relocating data with respect to data blocks, and it recognizes that we should be able to relocate data and/or program relative to data and/or program blocks.
My point is that we should consider core to be divided simply into blocks. The blocks to be referenced by a particular program should all be listed on the program card (with zero size for blocks of program, which would be expected to be in core already). This would obviate the need for your PROGRAM LOADING CARD and make the job of the loader much simpler. One could still differentiate between data destined for COMMON blocks and data destined for program blocks by means of bits 18-20 of the SPECIAL DATA CARD (so that data for COMMON blocks could be loaded backwards), but I would vote for forward storage of arrays, which would again tend to reduce the differences between various blocks of storage.
Another point on which I would like to comment is the ability to simply advance the current relocation amount to leave a gap in core. In our system we often have occasion to leave gaps at the low end of core, and I'm sure one could find other uses for this ability. This is currently available via the COMMON REASSIGNMENT CARD, but I see no provision in either IBM's proposal or yours for this. It will be suggested that a single program card, without any data cards behind it, will increase the relocation amount in the right way here; but a single program card followed by the program card for the program that follows will be considered part of the program card for that next program. I could put a dummy data card in between, but where should the data on that card load? This is not a very clean solution. Perhaps some way could be found (bit 12 of word 1?) to distinguish the first card of a program card from succeeding ones. Then a single one which reserves space would not be run together with another first one which follows.
In conclusion, I'm still reluctant to change relocation schemes without any good reason. Remembering that other people will want to extract FORTRAN in order to imbed it in other systems, we are herewith committing them to changes in their loaders, assemblers, etc., to keep in step with a particular FORTRAN system. We are going to take a long, hard look at it before accepting the new FORTRAN for our own system. If, however, the changes are approved, as it appears they will, I hope the comments I have made above will help.
Sincerely,Bernard A. GallerBAG:fjm
Subsequent to this exchange, the relocation concept was generalized to allow relocation with respect to something other than the program's loading address. Later, machine architectures also separated address information from machine instructions more cleanly, so instructions themselves could be executed without regard to where they were loaded, and all relocation changes were made in the program's data sections.
The term relocation bits actually fell out of use in the 1960s, when the relocation data became more complex than a simple bit map. The IBM 360/370 object file format contained a relocation dictionary (RLD) that described how to modify a program to take into account its environment as it was loaded. The current common object file format (COFF) used by PowerPC loaders contains a relocation data section, which is the clear descendant of the original relocation bits.
The impact of these two letters is now lost in history; perhaps a reader can carry the story further. Relocation bits are still here in some form, as I mentioned earlier. Where these letters and their ideas fit into their evolution is unclear. Comments and suggestions are welcome!
This recounting of the history of relocation bits is intended not only to record the then state of the art but to help others remember the lessons of history. These issues and concepts will always be with us, whether or not we recognize them in their current form.
Reference and note