 |
||
 |
 |
|
| HOME | CONTACT |  |
|
 |
||
    |
||
 |
||
 |
 |
 |
 |
|
|
 |
|
|
7 Benchmarking Myths Jump to Myth: 1 2 3 4 5 6 7  The first myth : its easy to do.Facts: Benchmarking is no place for amateurs - not if you want to make a decision based on the results. If you are "fooling around" and investigating things for yourself, you're very likely to get very different results even by altering what appears to be very minor factors. This is especially true in embedded benchmarking, where variable such as board bus speed, clock speed, memory speed, memory set-up, memory wait-states, types of memory, compiler used, compiler flags used, link map defined, incorrect programming of the on-chip timer, and literally dozens of other factors can make a huge difference in performance. Back To Top The second myth: "Just send me the binary or the source code and I can get the same results as you do."Facts: Well, we wish this were true. Sadly, its not that simple. Because of the First myth, you can see that re-creating the benchmark environment can be difficult, time-consuming, and exacting. If you do not have the exact configuration, exact set of the same software tools (including the same version!), and so on, it is usually very difficult to get the same scores as someone else. Back To Top The third myth: "Benchmarks will tell me what to buy."Facts: We wish this were true - it would make our mission in life all the more important. However, over-stating benchmark scores and drawing unwarranted conclusions based on simplistic benchmarks is more "bench-marketing" than benchmarking. Dhrystone is an excellent example. Its an integer only, synthetic, tiny benchmark that was written long ago by our friend, Dr. Reinhold Weicker (of Siemens-Nixdorf). Reinhold himself has disavowed Dhrystone (which explains why he was and is a key member of SPEC , a reputable workstation and server benchmark organization). Dhrystone is easily defeated by even primitive compilers, can be defeated easily if the rules are not followed (for example, if you in-line the code, the compiler can easily see all of the code and optimized most of it away!), and is an integrer-only benchmark. It is so small it does not take into account memory subsystems, nor floating point, nor SIMD instructions like AltiVec, 3DNow, MMX , and so on and so forth. Back To Top The fourth myth: MIPS or MOPS means something.Facts: Here's a hint - if someone is quoting benchmark scores in "MIPS" (millions of instructions per second, or "meaningless indicator of performance!) or "MOPS" (millions of operations per second), ask youself this: what kind of instructions? How "wide" are the instructions? And who certified the results to prevent cheating? Back To Top The fifth myth: "Only an industry-standard group has good benchmarks. Wrong! Good benchmarks are designed to tell you something about a processor, OS, compiler, memory subsystem, or other element in a computational system. Good benchmarks can come from industry groups, customers, internal engineering groups, or from private companies. The key is to understand - deeply - what the benchmarks tell you. We use benchmarks derived from real application algorithms as well as trusted synthetic benchmarks and even full applications.
Back To TopThe sixth myth: embedded benchmarking means writing the code in assembly language. Only that way can you compare apples to apples.Facts: Back in the old days, engineers and software programmers did write most DSP , code, and indeed most embedded code, in Assembly language. Compilers were inefficient, often emitted bad code, applications were simpler, and processors were frankly not all that powerful. The developer had to get the most from the processor, so Assembly language was the way to do it. Well, these days modern C compilers are generating, for the most part, very good code. In fact, many companies are extremely proud of the fact that you can get a fair fraction of the maximum performance from a processor just using their C compiler and other software tools (commonly called the Software Development Kit, or SDK). As applications have increased in complexity, it makes more sense to focus on good software engineering principles (to enable re-use, to allow for collaborative team development, and to improve quality) and to use the SDK's (software tools) to do so. Besides, the shear size of high-end embedded applications makes the thought of coding it all in assembly language painful to even think! Face it - you may be a great programmer, but as the embedded world has expanded to take up MOST of computing (its true!), the number of hot-shot, bare-metal, code for 100 hours straight to bare silicon programmers is not infinite. And besides - those folks are now CTO 's of companies, and don't have time to code themselves. Productivity depends upon quality software tools to help the programmer extract performance from the part. Modern DSP architectures, especially VLIW and mixed RISC / VLIW architectures, are tremendously complicated. Having the C compiler do the "dirty work" makes sense - you can always tune the algorithm in the hotspots in assembly language (or just wait for a slightly faster clock speed of the same part and save the months of pain). This situation will likely increase as DSP 's are blended with microcontrollers in modern SoC ("system on a chip") or SiP ("system in package") designs. If you want to use configurable/scalable IP (intellectual property) cores for your next SoC, you're not doing it in Assembler - the tools themselves emit the architecture, the instruction set, the Verilog or RTL, and so on. And Java? Well, embedded Java is taking off, too. That requires software tools that emit Java bytecodes (usually), which means the programmer is writing in Java, not Assembler. The revolution on archtitecture in the late 1980's and 1990's was this: compiler is part of archtitecture. RISC (reduced instruction set computers) and VLIW 's demand excellent compilers to do more of the work. Benchmarks can also help compare one compiler with another, just as you compare one processor with another! Other companies may write the benchmarks themselves in an architecture's assembly language, and then benchmarks it. How do you know that the programmer on one architecture is as good as that from another architecture? The seventh myth: companies don't want to compete based on real numbers. They all want to cheat.Facts: Well, maybe some. We have found that some companies join industry-standard groups to say, "hey, we support fair benchmarking. But groups like EEMBC PREVENT companies from comparing their scores against others! It has become an "old boys club", protecting companies from the realities of the market. Synchromesh Computing's benchmark studies give companies fair, honest, repeatable, certified benchmark scores, so they can compete fairly for customer's business. Back To Top |
 |
|
|
 |
 |
 |
 |
