From mcw@rice.edu Sun Feb 21 23:20:49 1988 Return-Path: Received: from anl-mcs.ARPA by antares.mcs.anl (3.2/SMI-3.2) id AA15614; Sun, 21 Feb 88 23:20:46 CST Received: from rice.edu (rice.edu.ARPA) by anl-mcs.ARPA (4.12/4.9) id AA26946; Sun, 21 Feb 88 23:20:31 cst Received: from iapetus (IAPETUS.RICE.EDU) by rice.edu (AA20615); Sun, 21 Feb 88 23:20:45 CST Received: by iapetus (AA08157); Sun, 21 Feb 88 23:21:26 CST Date: Sun, 21 Feb 88 23:21:26 CST From: Michael Wirth Message-Id: <8802220521.AA08157@iapetus> To: dongarra@anl-mcs.arpa Subject: Here's that C compiler benchmark memo Cc: mcw@iapetus.rice.edu Status: RO Posted: Tue Feb 9, 1988 8:58 AM CST Msg: JJII-2727-4700 From: CCULBERT To: software Subj: C development environment comparison The following memo FM7(87-411) is an electronic reprint of a survey done by Chris Culbert of the Artificial Intelligence Section in the Mission Planning and Analysis Division at Johnson Space Center. The copy of the memo or additional information is available from the author at FTS 525-8080. INTRODUCTION The computer language 'C' was developed at AT&T's Bell Laboratories to further the development of the UNIX operating system. Since that time, C has grown in popularity for a very wide range of applications, well outside the bounds of UNIX. There are many reasons for C's popularity, including generally high performance, comparatively high portability, good modularity, full structured programming capability, and it's ease of use for low-level programming. Within NASA, C has been used for a number of applications. The Artificial Intelligence Section (AIS) used C to develop CLIPS, a portable expert system tool. In the process of developing and porting CLIPS, the AIS has used a large number of C development environments. This memorandum compares a number of these environments on a variety of computers. The emphasis of the comparison is on programmer productivity and not strictly performance benchmarks, although some are provided. ENVIRONMENTS REVIEWED This review covers all the machines and compilers easily accessible to the AIS. These are listed in Table C. For the SUN, the Apollo, and the HP9000, the C compiler used was the standard C compiler provided as a part of the UNIX system. METHOD OF COMPARISON During actual code development, the average programmer spends 90-95% of his/her time cycling through the edit/compile/execute loop. This review is primarily concerned with the speed which the programmer can move through this loop and the selection of tools he/she has at hand to apply to programming problems. The speed is a function of basic CPU processing speed and the rate at which the compiler/linker executes. The tools available to programmers should include such standard UNIX tools utilities as grep, a debugger, an object code library manager, a dump utility, and a utility to compare the contents of two files, and full screen editors. Also of concern in any compiler comparison are the size of the executable code, the speed with which it processes, and the error detection accuracy of the compiler. Although these factors do not affect programmer productivity as directly, they represent useful information. Some of these factors will be discussed as well. PERFORMANCE The performance of each compiler was compared by compiling a configured pro- gram in each environment. The program used was CLIPS (version 4.10), a portable expert system development tool. This program consists of about 22000 lines of commented C code in 18 files. On each system, the code was compiled using that system's version of make. Each test compiled all the code in all the C files and linked with all libraries needed for execution (with the exception of Lightspeed C on the Macintosh, discussed in the following paragraph). Table A lists the performance of each system in order. Two clarifications should be made about the times listed in table A. First, the Lightspeed C compiler on the Macintosh computers treats linking differently than the other compilers. Lightspeed C allows full execution of the program from within the development environment without linking all the necessary libraries for stand alone execution. Creating a stand-alone executable is a separate step, presumably done only once at the end of all development. All other compilers link all libraries all the time. To make the comparisons more accurate, both times are listed for Lightspeed C, the time to compile an executable within the environment only and the time to compile an executable for stand-alone execution. The second clarification concerns the times for the multi-user systems (SUN, HP9000, VAX). The times in the table are for a system with only one user (the person doing the compiling). Therefore, the compile and link time during normal usage periods, when competing with numerous other users, could be considerably longer. SUPPORT TOOLS A good C development environment should also include a number of tools which the programmer can apply to solving problems. To some extent, these tools can be divided into two groups: (1) tools which are directly involved in the edit/ compile/execute loop and (2) tools which are used less frequently. The first set are considered more important for this review and primarily consists of an editor and a make tool. Full Screen Editor The most important tool is a robust full screen editor. Most of the machines reviewed here provided many choices of editors, such as EMACS and vi under UNIX, ED, EMACS, and LSE under VMS, and numerous choices under MS-DOS. Turbo C provides an integrated editor which can be partially tailored to a users preference. It also allows command line compiling which would let a programmer use any MS-DOS editor he or she desires. Lightspeed C provided an integrated editor which could not be changed or reconfigured, but was designed to the Macintosh standards for text editing. Most Macintosh users would not consider the editor a drawback in Lightspeed C. Make Another important tool is a make facility. This tool allows a programmer to describe the components of a program and how they should be constructed. When executed, the tool will examine the current state of each element, decide which ones need to be reconstructed and performs only those steps required to bring the program up to date. Proper use of make is an essential productivity step since it removes the need for the programmer to remember and reconstruct each changed part on his or her own. All UNIX systems have a make tool, and both MS-DOS compilers provide one. VAX VMS does not typically have 'make' as a standard tool, but a comparable tool has been developed by the AIS and was used for this review. Lightspeed C provides this capability, but in a slightly different manner. Programs are described in terms of a project which contains all the essential elements. The development environment automatically keeps track of the current state of all program elements and their relation to each other. When the user wishes to execute the program, the environment acts like a UNIX make and reconstructs only those elements which need to be updated. The project in Lightspeed C is less flexible than a UNIX make but considerably easier to use. Other Tools The second set of support tools include such as grep (a multi-file string search utility), make (a utility to automate recompilation of code), a debugger (source code level), an object code library manager, a dump utility (to view byte by byte listings of a file in hex, decimal, ASCII, etc.), and a utility to compare the contents of two files. Generally, these tools are used less frequently than those directly involved in the edit/compile/execute loop (with the possible exception of the library manager, since some programmers construct libraries as a part of standard development). However, even with infrequent use, these tools can prove invaluable for solving coding problems. The full set of these tools are standard in UNIX environments. VAX VMS also provides a full complement of these types of tools. Although not as well organized, most of these tools are available in MS-DOS, either bundled with the compiler or through public domain source code. The major exception is a source level debugger. Microsoft C includes one; Turbo C does not. A third party debugger is available for Turbo C, but its availability is not well known. Version 1.0 of Turbo C also does not include a library manager, although one is to be included in the next release of Turbo C. These additional tools are the area where the Macintosh is weakest. Lightspeed C provides a full library manager. It also provides a multi-file search for all files in a project, but it can not be used outside the project. Lightspeed C does not include a source level debugger in the current release. Some of the other tools (file dumps and file comparison) are available through desk accessories. OTHER FACTORS As mentioned previously, any compiler comparison should also mention the size of the executable code, the speed with which it processes, and the error detection accuracy of the compiler. The size of the executable element is listed in both table A and B. With the exception of the SUN, there were no significant differences. The speed of execution is considerably more dependent on the type of machine, as table B shows. In general, the compilers were all accurate with regards to detecting errors. Most of the compilers also detected potential portability problems and poor coding constructs. The exception was the DEC C compiler, which had a tendency to be too lenient (i.e., it let slide or corrected small problems for the pro- grammer without advising him or her). This is not a problem when writing code for execution on the VAX. However, when developing portable code, the VAX compiler allows many non-portable constructs through without warning. CONCLUSIONS There are two clear results of this study: (1) the state of the art in C compiler technology is highest on personal computers, (2) the high powered personal computers (80386 machines, 68020 based Macintosh II) are the most productive environments for development of C code by a single person. The fact that compilers run 5-15 times faster on personal computers that are 3-5 times slower than workstations clearly demonstrates the advanced compiler technology. The compiler industry is focussing their best efforts on these types of machines, probably because of the potential market size (millions of PC's compared to thousands of workstations). The advantages of using PC's for development is also clear. The productivity gain is more than an order of magnitude when compared to multi-user engineering workstations like the HP9000 and the VAX 11/780. Even higher gains are likely if the comparison examines the multi-user machines at their normal workloads (3-10 users). Execution speed for the applications can also be better on the high powered PC's (see table B). For programs which must be accessible by a large number of users, a workstation may still be the best delivery environment, but development can and should be done on the personal computers. The Macintosh II is the clear standout among the PC's, both in speed and in environment. Although Lightspeed C is somewhat weak in the support tools, it more than makes up for it with gains in the edit/compile/execute loop. Many users will also find the windowed/mouse environment considerably easier to use than a command line system. Turbo C and the next release of Microsoft C are also moving toward integrated environments with window and mouse support. It is important to note that although this type of development environment is highly machine dependent, the code developed using it is not. Among the workstations, the SUN 3 is the clear winner, mostly because it runs a high performance CPU (the 68020). Finally, we must discuss the limitations of this review. The results are clear, but they only apply to development of C code on projects with one or two programmers. Large projects which require many teams of programmers may be more suitable on workstations where they can take advantage of configuration management tools and improved code access. Networked PC's may soon provide the same level of code access, but PC development environments are definitely intended for single users. Also, this review only considers the actual code development portion of the software lifecycle. Tools which aid in other portions of the software lifecycle are not as common on personal computers. To take advantage of these results, the majority of the code being developed must be portable, machine independent code. If a significant portion of the code is machine dependent, such as interfaces, you may not be able to use the most productive development environment. Also, developing portable code requires a higher level of skill and knowledge on the part of the programmer. The implications this review has for development of code in other languages is unclear. Certainly Pascal comparisons will show similar results. However, a comparison between FORTRAN or COBOL development environments may not favor the personal computers. Ada development is still an open question. It is unlikely that any existing Ada compilers, PC or workstation, could demonstrate the type of speed state-of-the-art C compilers routinely provide. Ada compiler technology is much newer, and the language is inherently more difficult to compile. Performance Tables Table A - Ordered by Compile and Link Speed Time to Size of Time to run Compile & Executable Benchmark Computer Compiler Link (min:sec) (bytes) (sec) Macintosh II Lightspeed C 0:49[1] 164976 0.85 Macintosh II Lightspeed C 1:23[2] 164976 0.85 Macintosh SE Lightspeed C 2:15[1] 164976 2.40 Compaq 386 Turbo C 2:26 180208 0.82 Macintosh SE Lightspeed C 3:30[2] 164976 2.40 SUN 3/75 UNIX C 4:28 262144 0.58 Apollo 3000 UNIX C 5:39 232372 0.93 IBM PC AT Turbo C 6:18 180208 2.58 Compaq 386 Microsoft C 13:45 197190 0.82 VAX 11/780 Dec C 15:48 aprox 165000 1.23 HP900 (500) UNIX C 17:08 222769 2.13 IBM PC AT Microsoft C 36:45 197190 2.58 [1] Execution within Lightspeed C environment (i.e., without final link step) [2] Stand-alone execution (i.e., with final link step included) Table B - Ordered by Benchmark Speed Time to Size of Time to run Compile & Executable Benchmark Computer Compiler Link (min:sec) (bytes) (sec) SUN 3/75 UNIX C 4:28 262144 0.58 Compaq 386 Turbo C 2:26 180208 0.82 Compaq 386 Microsoft C 13:45 197190 0.82 Macintosh II Lightspeed C 0:49[1] 164976 0.85 Apollo 3000 UNIX C 5:39 232372 0.93 VAX 11/780 Dec C 15:48 aprox 165000 1.23 HP900 (500) UNIX C 17:08 222769 2.13 Macintosh SE Lightspeed C 2:15[1] 164976 2.40 IBM PC AT Turbo C 6:18 180208 2.58 IBM PC AT Microsoft C 36:45 197190 2.58 Table C - Systems Compared and Configuration Machine Oper. Sys. Memory Compiler (Version) DEC Vax 11/780 VMS (4.5) 4 M DEC C (2.2) HP9000 Series 500 HP-UX (5.0) 4.75 M standard UNIX C SUN 3/75 UNIX (4.2bsd) 4 M standard UNIX C Apollo 3000 UNIX (4.2bsd) 4 M standard UNIX C Macintosh SE MacOS 1 M Lightspeed C (2.13) Macintosh II MacOS 1 M Lightspeed C (2.13) IBM PC AT MS-DOS 3.2 640K Microsoft C (4.0) Turbo C (1.0) Compaq 386 MS-DOS 3.2 2 M Microsoft C (4.0) Turbo C (1.0) ------------------------------------ Jack, Sorry about the transmission errors and wrap-around. I can FTP you a clean copy if you like. Just let me know. .