User Tools
appendix:guidebook_authoring:benchmarks
This is an old revision of the document!
Table of Contents
Guidebook Benchmarks
Standards
5 Is the Number
- Each test case should be run 5 times, and averaged.
- 5 may seem somewhat arbitrary because it is.
- We needed a standard, so we selected one.
- 5 tests can be easily run by hand, and averaged by hand.
- Automation scripts will create many additional files to be maintained to ensure completeness.
- Only display the average.
Present Results in Seconds
Make a Change -> Rerun All Tests
Make All Efforts to Only Test Your Target
- Keep all test cases as simple as possible.
- Consider your tests carefully, and make efforts to only test the desired language feature.
- For our standard input benchmarks we timed our entire test using the bash
timebuiltin.- In all cases we stored the results of a read, but we did not maintain them between iterations.
- E.g., We did not want to consider the cost of
listgrowth outside ofstdin.readlines()where storing an entire file is unavoidable.
- For our standard output benchmarks we decided to read the same data as our standard input benchmarks to maintain consistency, but did not want to account for the cost of reads.
- Some level of judgment on the part of the author at the time is necessary to ensure that these efforts are made correctly.
appendix/guidebook_authoring/benchmarks.1534369654.txt.gz · Last modified: 2018/08/15 16:47 by jguerin
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International
