Below are the benchmark programs and the results of the benchmark runs for the projects turned in last Thursday and Friday. All were run on the same machine with the Sun JVM. All times are measured in seconds. The groups have been sorted randomly and renamed to protect privacy. The first three tests (beginning with "correct_") were to test compiler correctness and were not timed. "ok" indicates that the test passed. correct_bounds and correct_null should result in a runtime error. Many of the test programs were complex and tested a large number features of the compiler, so relatively few group compilers were able to handle them.
| Group | O | P | Q | R | S* | T | U | V | W | X | Y | Z | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Group size | 4 | 4 | 3 | 4 | 3 | 3 | 4 | 4 | 2 | 4 | 3 | 4 | |
| Compiler size (tokens) | 90k | 75k | 76k | 79k | 61k | 56k | 88k | 35k | 64k | 98k | 55k | 74k | |
| correct_bounds.mod | Compile | (690s+) | 4.93 | 3.83 | 4.93 | 3.57 | 3.38 | 3.96 | 3.06 | 5.33 | 5.66 | 3.43 | 3.54 |
| Run | - | failed | asm | ok | asm | ok | asm | asm | ok | ok | asm | ok | |
| .obj size | - | 3561 | - | 3133 | - | 2681 | - | - | 2880 | 3217 | - | 3077 | |
| correct_null.mod | Compile | rx | 4.29 | 4.21 | 5.16 | 2.17 | 3.50 | 3.92 | rx | 5.50 | 4.42 | 3.38 | 3.95 |
| Run | - | ok | asm | ok | asm | ok | asm | - | ok | ok | asm | ok | |
| .obj size | - | 2958 | - | 4271 | - | 2713 | - | - | 2401 | 3032 | - | 3115 | |
| correct_call.mod | Compile | sem | 9.89 | cmp | 5.99 | sem | 4.97 | sem | rx | 8.58 | 20.08 | 7.67 | 3.94 |
| Run | - | 0.06 | - | 0.06 | - | 0.04 | - | - | 0.05 | 0.06 | 0.05 | 0.54 | |
| .obj size | - | 15259 | - | 16284 | - | 12618 | - | - | 15394 | 18374 | 20058 | 13707 | |
| argsdump.mod | Compile | 12.42 | 3.98 | 4.53 | 5.09 | 6.34 | 7.70 | 5.00 | 2.76 | 6.19 | 4.65 | 3.04 | 3.37 |
| Run | 0.06 | 0.05 | 0.05 | 0.05 | 0.06 | 0.06 | 0.04 | asm | 0.05 | 0.05 | failed | 0.04 | |
| .obj size | 2729 | 3083 | 2548 | 2975 | 3583 | 2610 | 2210 | - | 2453 | 2710 | 3007 | 3016 | |
| fold1000.mod | Compile | 4.01 | 4.56 | 3.75 | 4.20 | 2.74 | 3.81 | 3.86 | 8.44 | 6.53 | 4.85 | 3.36 | 3.90 |
| Run | failed | 0.05 | 0.05 | 0.06 | failed | 0.05 | 0.05 | link | 0.07 | 0.05 | 0.06 | 0.05 | |
| .obj size | 1959 | 2044 | 1729 | 3116 | 2018 | 2066 | 2042 | 28753 | 1758 | 2374 | 2146 | 1775 | |
| GA.mod, random.in | Compile | rx | 63.90 | 36.06 | 11.09 | sem | 10.33 | sem | rx | 22.10 | cmp | rx | 5.20 |
| Run | - | 88.12 | asm | crash | - | 107.55 | - | - | 130.26 | - | - | 69.71 | |
| .obj size | - | 69873 | - | 40490 | - | 45559 | - | - | 51087 | - | - | 40502 | |
| gps.mod | Compile | 3.92 | 5.71 | 4.10 | 5.25 | 2.93 | 4.11 | 3.73 | 2.63 | 6.29 | 6.69 | 3.45 | 3.93 |
| Run | asm | 0.05 | asm | 0.05 | asm | 0.05 | asm | asm | 0.08 | failed | asm | 0.06 | |
| .obj size | - | 5900 | - | 6873 | - | 5763 | - | - | 5482 | 5806 | - | 5103 | |
| Collection.int, Collection.mod, hash.mod | Compile | sem/sem | rx/sem | cmp/sem | rx/cmp | sem/rx | 6.3/sem | sem/sem | rx/rx | 7.4/sem | 17.2/rx | sem/sem | 4.5/sem |
| Run | - | - | - | - | - | - | - | - | - | - | - | - | |
| .obj size | - | - | - | - | - | 17696 | - | - | 18642 | 18106 | - | 14877 | |
| kmp_test.mod, kmp.int, kmp.mod, kmp.in | Compile | rx/(185s+) | 7.5/9.7 | 5.0/7.0 | 5.0/5.8 | rx/sem | 5.7/5.6 | sem/4.9 | rx/rx | 7.0/6.8 | 8.1/9.2 | 4.41/5.34 | 4.2/4.4 |
| Run | - | 5.75 | 4.54 | 2.58 | - | (301s+) | - | - | crash | 4.51 | link | 4.57 | |
| .obj size | - | 7k/12k | 5k/10k | 6k/11k | - | 5k/12k | -/8k | - | 6k/12k | 6k/8k | 7k/15k | 6k/10k | |
| ListSearch.mod, random.in | Compile | rx | 17.54 | 11.97 | 7.33 | rx | 7.16 | sem | rx | 18.72 | 32.78 | rx | 4.44 |
| Run | - | 8.49 | asm | crash | - | 24.78 | - | - | 9.07 | 10.22 | - | 6.79 | |
| .obj size | - | 22657 | - | 21283 | - | 17658 | - | - | 19911 | 22423 | - | 18678 | |
| maze_test.mod, maze.int, maze.mod | Compile | sem/(165s+) | 18.6/sem | cmp/cmp | rx/sem | sem/sem | 4.8/rx | sem/sem | rx/rx | sem/10.3 | 14.3/cmp | 7.02/rx | sem/5.6 |
| Run | - | - | - | - | - | - | - | - | - | - | - | - | |
| .obj size | - | 20446/- | - | - | - | 15129/- | - | - | -/41417 | 11111/- | - | -/45304 | |
| mdb_test.mod | Compile | rx | 16.42 | 23.01 | 5.75 | sem | 5.18 | 5.14 | rx | 10.74 | 14.41 | 6.68 | 4.96 |
| Run | - | 5.22 | failed | failed | - | 4.53 | failed | - | 5.39 | 5.02 | 6.62 | 5.19 | |
| .obj size | - | 26165 | 17504 | 15942 | - | 16692 | 9027 | - | 17322 | 12265 | 23460 | 13234 | |
| mmm.mod | Compile | 95.27 | 59.94 | cmp | cmp | sem | 8.98 | 8.40 | 50.10 | 9.78 | cmp | 13.82 | 4.96 |
| Run | crash | 1.53 | - | - | - | 1.52 | 3.67 | asm | 1.83 | - | 3.71 | 1.77 | |
| .obj size | 47047 | 33389 | - | - | - | 21507 | 14660 | - | 24629 | - | 32892 | 19711 | |
| sieve.mod | Compile | 5.82 | 8.89 | 5.51 | 5.01 | 5.07 | 3.56 | 4.06 | rx | 9.16 | 7.37 | 4.64 | 3.78 |
| Run | crash | 2.81 | 3.60 | 2.75 | 3.35 | 2.69 | 8.73 | - | 3.11 | 3.03 | crash | 2.80 | |
| .obj size | 6554 | 6189 | 4501 | 4870 | 9094 | 4750 | 3726 | - | 5219 | 4709 | 6429 | 4934 | |
* Group S used jview instead of Sun java to run their tests because of class file format errors. Times are inconsistent with the other groups.
rx = RuntimeException during compile
sem = semantic error during compile
cmp = other compile-time error (parse error, tiling problem, assertion
check failure)
link = link error (usually undefined symbol)
asm = incorrect assembly code
crash = run-time crash
failed = incorrect output or other run-time error
(Xs+) = killed after X seconds because run was taking too
long