FuzzBench: hybrid_all report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
jigsaw 98.40
z3 97.02
fuzzolic 95.56
aflplusplus 95.13
symqemu 94.52
honggfuzz 94.50
symcc 93.75
eclipser 91.89
entropic 91.47
aflsmart 89.41
afl 89.09
mopt 87.31
libfuzzer 86.44
By avg. rank
average rank
fuzzer
jigsaw 3.97
z3 3.97
fuzzolic 4.56
symqemu 5.35
aflplusplus 5.41
symcc 5.62
honggfuzz 7.15
entropic 7.53
eclipser 8.29
afl 8.47
aflsmart 8.91
mopt 9.53
libfuzzer 10.47
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    jigsaw z3 fuzzolic aflplusplus honggfuzz symqemu symcc eclipser entropic aflsmart afl mopt libfuzzer
    FuzzerMedian 96.86 97.62 96.86 95.53 96.71 96.80 95.81 93.79 92.78 94.82 94.71 94.64 86.75
    FuzzerMean 93.94 92.63 91.28 91.00 90.40 90.29 89.51 87.90 87.45 85.49 85.17 83.43 82.54
    curl_curl_fuzzer_http 98.82 98.68 96.86 98.78 98.94 96.67 96.98 97.06 92.78 96.90 96.99 96.59 88.86
    freetype2-2017 90.89 90.59 85.52 90.66 93.71 77.92 82.69 73.37 74.89 66.66 66.93 66.85 62.48
    harfbuzz-1.3.2 95.02 97.80 92.95 91.99 94.24 92.59 92.59 91.52 93.08 91.16 91.27 91.08 90.97
    jsoncpp_jsoncpp_fuzzer 99.53 99.61 nan 99.53 99.84 nan nan 99.53 100.00 99.53 99.53 99.69 100.00
    lcms-2017-03-21 95.42 70.82 92.90 84.92 68.14 90.09 91.21 64.56 83.87 64.53 64.53 60.21 78.70
    libjpeg-turbo-07-2017 99.06 98.89 99.19 98.78 97.74 99.32 99.10 97.79 99.06 97.82 98.40 98.04 97.61
    libpng-1.2.56 96.86 96.68 97.70 94.70 98.02 97.84 87.14 81.27 91.49 73.55 68.73 68.73 90.87
    libxml2-v2.9.2 97.28 97.62 90.26 92.94 83.33 92.01 82.97 80.35 57.49 81.48 81.32 64.90 48.47
    libxslt_xpath 99.40 99.30 96.44 98.78 98.49 96.85 96.77 98.19 85.19 98.29 98.27 97.20 82.38
    mbedtls_fuzz_dtlsclient 71.37 71.48 71.12 70.99 70.37 70.92 71.05 70.91 70.50 70.91 71.10 71.00 69.64
    openssl_x509 99.86 99.93 99.86 99.94 99.60 99.93 99.93 99.93 99.87 99.94 99.93 99.93 99.67
    openthread-2019-12-23 77.10 77.08 63.31 62.51 62.37 62.78 63.56 62.69 63.15 62.37 62.45 62.72 63.05
    proj4-2017-08-14 87.14 85.45 73.51 70.07 96.71 66.93 67.57 89.62 98.10 62.71 60.96 54.47 96.81
    re2-2014-12-09 98.30 98.73 99.04 98.27 97.93 98.32 98.39 98.27 99.33 98.21 98.55 98.18 98.18
    sqlite3_ossfuzz 96.59 96.91 97.01 99.10 82.07 96.80 95.81 93.79 84.72 94.82 94.71 94.64 64.73
    vorbis-2017-12-11 99.52 99.38 99.45 99.45 98.74 99.45 99.43 99.33 99.31 99.38 99.36 99.40 83.97
    woff2-2016-05-06 94.84 95.74 98.85 95.53 96.63 98.72 98.72 96.12 93.77 95.10 94.90 94.74 86.75
    zlib_zlib_uncompress_fuzzer nan nan 97.77 nan nan 97.87 97.77 nan nan nan nan nan nan
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 17965.10 74.632362 17824.0 17920.75 17977.0 18000.75 18108.0
    jigsaw 82800 20.0 17954.55 110.614777 17743.0 17889.50 17956.5 18009.75 18158.0
    aflplusplus 82800 20.0 17959.10 106.579795 17794.0 17892.50 17948.5 18011.00 18170.0
    z3 82800 20.0 17927.15 120.755200 17728.0 17827.25 17931.0 18048.50 18108.0
    eclipser 82800 20.0 17628.65 73.138135 17503.0 17606.00 17636.0 17650.00 17840.0
    afl 82800 20.0 17622.05 58.351453 17521.0 17595.75 17623.0 17654.00 17755.0
    symcc 82800 20.0 17626.75 63.351879 17464.0 17602.25 17622.0 17663.25 17732.0
    aflsmart 82800 20.0 17624.15 70.812038 17534.0 17581.25 17607.0 17645.50 17796.0
    fuzzolic 82800 20.0 17610.50 39.907129 17555.0 17577.00 17599.5 17640.25 17698.0
    symqemu 82800 20.0 17561.70 73.968770 17357.0 17550.75 17564.5 17586.75 17714.0
    mopt 82800 20.0 17544.35 53.320555 17392.0 17527.50 17551.0 17572.50 17614.0
    entropic 82800 20.0 16896.35 550.903304 15772.0 16638.75 16859.0 17435.25 17656.0
    libfuzzer 82800 20.0 16098.60 905.373805 14277.0 15875.50 16145.0 16594.00 17452.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2-2017 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 28798.400000 497.285092 27453.0 28423.25 28895.5 29100.00 29772.0
    jigsaw 82800 18.0 28288.944444 1058.337235 26159.0 27916.50 28026.0 28234.75 30835.0
    aflplusplus 82800 20.0 27881.950000 1331.397781 25562.0 27377.25 27956.5 28307.50 30145.0
    z3 82800 20.0 28032.050000 936.381865 26285.0 27748.50 27932.5 28197.75 30486.0
    fuzzolic 82800 19.0 26480.157895 557.185413 25242.0 26170.50 26371.0 26989.50 27275.0
    symcc 82800 20.0 25448.550000 538.330702 23564.0 25337.25 25496.0 25750.75 26033.0
    symqemu 82800 20.0 24090.150000 546.996899 22806.0 23800.00 24028.0 24286.50 25330.0
    entropic 82800 20.0 23005.650000 1147.494950 20512.0 22598.50 23093.0 23620.50 25274.0
    eclipser 82800 20.0 22806.400000 668.781248 21825.0 22500.25 22623.5 22866.00 24576.0
    afl 82800 20.0 20599.800000 301.860128 19471.0 20588.75 20639.0 20692.00 20931.0
    mopt 82800 20.0 20524.050000 357.815224 19483.0 20469.25 20614.5 20694.75 20923.0
    aflsmart 82800 20.0 20509.600000 344.140027 19604.0 20471.50 20554.0 20680.75 20909.0
    libfuzzer 82800 20.0 19401.700000 1060.361413 17071.0 18812.50 19267.0 20064.00 21653.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz-1.3.2 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    z3 82800 20.0 8875.40 245.244540 8481.0 8658.75 8959.0 9081.50 9161.0
    jigsaw 82800 20.0 8748.10 219.030616 8462.0 8517.75 8705.0 8914.50 9108.0
    honggfuzz 82800 20.0 8642.00 83.504176 8492.0 8620.25 8633.0 8658.25 8914.0
    entropic 82800 20.0 8521.85 100.352681 8365.0 8432.50 8527.0 8611.25 8672.0
    fuzzolic 82800 20.0 8541.45 115.423009 8407.0 8437.50 8515.0 8645.25 8730.0
    symcc 82800 20.0 8577.00 204.458206 8420.0 8428.00 8482.5 8666.75 9135.0
    symqemu 82800 20.0 8551.10 107.523755 8441.0 8453.25 8482.5 8660.25 8704.0
    aflplusplus 82800 20.0 8495.60 144.377211 8367.0 8411.75 8427.0 8515.50 8820.0
    eclipser 82800 20.0 8414.15 106.771411 8269.0 8324.00 8384.5 8492.50 8598.0
    afl 82800 20.0 8402.95 86.791629 8314.0 8339.50 8361.0 8498.25 8560.0
    aflsmart 82800 20.0 8386.80 83.909475 8302.0 8343.75 8351.5 8381.75 8578.0
    mopt 82800 20.0 8363.70 62.238167 8287.0 8322.75 8343.5 8385.50 8498.0
    libfuzzer 82800 20.0 8351.30 93.435089 8112.0 8319.25 8333.5 8371.75 8547.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 641.00 0.000000 641.0 641.0 641.0 641.0 641.0
    libfuzzer 82800 20.0 641.00 0.000000 641.0 641.0 641.0 641.0 641.0
    honggfuzz 82800 20.0 640.20 0.410391 640.0 640.0 640.0 640.0 641.0
    mopt 82800 20.0 638.60 0.502625 638.0 638.0 639.0 639.0 639.0
    z3 82800 20.0 638.50 0.512989 638.0 638.0 638.5 639.0 639.0
    afl 82800 20.0 638.30 0.470162 638.0 638.0 638.0 639.0 639.0
    aflplusplus 82800 20.0 638.40 0.502625 638.0 638.0 638.0 639.0 639.0
    aflsmart 82800 20.0 638.40 0.502625 638.0 638.0 638.0 639.0 639.0
    eclipser 82800 20.0 638.35 0.489360 638.0 638.0 638.0 639.0 639.0
    jigsaw 82800 20.0 638.20 0.410391 638.0 638.0 638.0 638.0 639.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms-2017-03-21 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    jigsaw 82800 20.0 3845.90 102.799191 3695.0 3740.50 3872.0 3905.75 4058.0
    fuzzolic 82800 20.0 3740.40 94.020938 3535.0 3679.00 3770.0 3797.00 3887.0
    symcc 82800 20.0 3677.15 173.210026 3233.0 3651.00 3701.5 3774.75 3895.0
    symqemu 82800 20.0 3686.05 136.799690 3438.0 3605.75 3656.0 3764.75 3944.0
    aflplusplus 82800 20.0 3214.15 401.521485 2499.0 2783.75 3446.0 3514.50 3707.0
    entropic 82800 20.0 3407.90 124.575447 3208.0 3314.50 3403.5 3514.50 3587.0
    libfuzzer 82800 20.0 3190.65 138.313250 2920.0 3111.50 3193.5 3289.25 3394.0
    z3 82800 20.0 3123.80 400.802301 2643.0 2770.00 2874.0 3557.75 3618.0
    honggfuzz 82800 20.0 2997.25 454.180103 2559.0 2579.00 2765.0 3445.00 3776.0
    eclipser 82800 20.0 2545.05 95.750484 2430.0 2443.00 2620.0 2626.25 2655.0
    afl 82800 20.0 2495.10 264.741761 1444.0 2434.75 2618.5 2624.50 2659.0
    aflsmart 82800 20.0 2540.10 107.336608 2311.0 2444.50 2618.5 2624.75 2661.0
    mopt 82800 20.0 2509.05 100.323936 2431.0 2435.00 2443.5 2629.75 2680.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo-07-2017 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symqemu 82800 20.0 3814.300000 16.403145 3777.0 3801.00 3819.0 3825.00 3842.0
    fuzzolic 82800 20.0 3814.950000 19.261292 3779.0 3803.25 3814.0 3829.00 3844.0
    symcc 82800 20.0 3795.900000 72.646875 3494.0 3803.50 3810.5 3821.25 3841.0
    entropic 82800 19.0 3811.526316 11.379164 3785.0 3805.00 3809.0 3815.50 3833.0
    jigsaw 82800 19.0 3813.631579 20.486338 3765.0 3803.50 3809.0 3826.00 3845.0
    z3 82800 20.0 3802.050000 22.201174 3752.0 3794.00 3802.5 3821.00 3842.0
    aflplusplus 82800 20.0 3797.150000 19.762471 3754.0 3783.75 3798.0 3816.00 3824.0
    afl 82800 20.0 3735.500000 119.489044 3453.0 3758.75 3783.5 3800.00 3820.0
    mopt 82800 20.0 3764.150000 40.766698 3637.0 3757.75 3769.5 3784.50 3806.0
    aflsmart 82800 20.0 3645.650000 168.960374 3405.0 3457.75 3761.0 3784.00 3807.0
    eclipser 82800 20.0 3658.200000 152.117199 3443.0 3472.25 3760.0 3779.25 3811.0
    honggfuzz 82800 20.0 3671.000000 159.270706 3430.0 3439.75 3758.0 3779.00 3802.0
    libfuzzer 82800 20.0 3633.800000 225.909763 3186.0 3651.75 3753.0 3755.75 3790.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng-1.2.56 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 2149.50 22.765394 2103.0 2145.75 2153.5 2158.25 2197.0
    symqemu 82800 20.0 2142.45 18.420455 2108.0 2127.00 2149.5 2156.00 2166.0
    fuzzolic 82800 20.0 2138.15 23.454940 2097.0 2112.50 2146.5 2153.25 2174.0
    jigsaw 82800 20.0 2140.45 27.130241 2108.0 2124.75 2128.0 2159.00 2195.0
    z3 82800 20.0 2130.10 22.952812 2106.0 2113.75 2124.0 2137.25 2190.0
    aflplusplus 82800 20.0 2080.45 16.433198 2056.0 2071.75 2080.5 2088.00 2135.0
    entropic 82800 20.0 2015.85 16.255444 2003.0 2008.00 2010.0 2014.50 2067.0
    libfuzzer 82800 20.0 1987.70 21.491981 1943.0 1993.50 1996.5 2000.00 2002.0
    symcc 82800 20.0 1928.75 28.728997 1900.0 1909.00 1914.5 1940.50 2008.0
    eclipser 82800 20.0 1772.30 58.041818 1526.0 1783.50 1785.5 1787.00 1793.0
    aflsmart 82800 20.0 1617.70 6.641972 1614.0 1615.75 1616.0 1617.00 1645.0
    afl 82800 20.0 1510.85 3.166851 1508.0 1510.00 1510.0 1510.00 1520.0
    mopt 82800 20.0 1509.85 0.366348 1509.0 1510.00 1510.0 1510.00 1510.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2-v2.9.2 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: symcc.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    z3 82800 20.0 12921.850000 392.262704 11901.0 12650.50 13056.0 13275.75 13374.0
    jigsaw 82800 20.0 12923.300000 308.475897 12091.0 12724.25 13010.5 13157.25 13363.0
    aflplusplus 82800 20.0 11947.500000 1428.663582 7521.0 11818.25 12429.5 12716.50 13189.0
    symqemu 82800 19.0 12203.210526 470.727992 11295.0 11796.50 12305.0 12574.00 12910.0
    fuzzolic 82800 19.0 11936.631579 984.269228 8527.0 11773.50 12072.0 12558.50 12937.0
    honggfuzz 82800 20.0 11019.750000 1006.297742 9128.0 10324.00 11144.0 11576.25 12878.0
    symcc 82800 1.0 11097.000000 NaN 11097.0 11097.00 11097.0 11097.00 11097.0
    aflsmart 82800 20.0 10297.600000 1493.633274 6839.0 9998.50 10897.5 11146.75 12143.0
    afl 82800 20.0 10405.200000 1374.587195 6835.0 10319.25 10875.5 11207.50 11761.0
    eclipser 82800 20.0 10030.950000 1682.728157 6674.0 9722.00 10745.5 11320.75 11912.0
    mopt 82800 20.0 8871.600000 1136.880237 6860.0 8187.75 8679.5 9777.75 10844.0
    entropic 82800 20.0 7616.000000 629.329635 6493.0 7431.25 7688.5 8013.00 9232.0
    libfuzzer 82800 20.0 6880.250000 550.392671 6418.0 6460.00 6482.5 7570.50 7622.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    jigsaw 82800 20.0 19100.85 58.426089 19016.0 19061.50 19083.5 19142.25 19198.0
    z3 82800 20.0 19041.55 98.359959 18854.0 18980.00 19064.5 19102.25 19172.0
    aflplusplus 82800 20.0 18957.00 128.081800 18736.0 18888.75 18963.5 19056.50 19178.0
    honggfuzz 82800 20.0 18901.05 143.575572 18685.0 18778.25 18908.0 18996.75 19179.0
    aflsmart 82800 20.0 18880.30 60.963406 18765.0 18830.00 18870.5 18923.50 18977.0
    afl 82800 20.0 18850.15 71.677843 18686.0 18822.25 18865.5 18898.00 18976.0
    eclipser 82800 20.0 18850.15 65.984268 18748.0 18797.00 18850.0 18894.75 19022.0
    mopt 82800 20.0 18652.60 69.051011 18515.0 18615.25 18661.0 18694.25 18760.0
    symqemu 82800 18.0 18560.50 78.628875 18394.0 18508.50 18592.5 18618.50 18663.0
    symcc 82800 20.0 18566.75 101.998388 18409.0 18480.50 18577.0 18621.75 18789.0
    fuzzolic 82800 20.0 18516.50 96.095621 18303.0 18452.25 18515.0 18561.00 18712.0
    entropic 82800 20.0 16407.50 178.394448 16151.0 16287.75 16355.5 16539.50 16767.0
    libfuzzer 82800 20.0 15849.00 123.141767 15700.0 15754.25 15815.5 15917.25 16110.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    z3 82800 20.0 8556.600000 781.168854 8220.0 8291.75 8310.0 8336.25 10913.0
    jigsaw 82800 20.0 8398.150000 473.942066 8202.0 8290.75 8297.0 8309.50 10407.0
    fuzzolic 82800 20.0 8263.500000 49.827598 8151.0 8243.50 8268.0 8277.00 8396.0
    afl 82800 20.0 8595.350000 1016.725460 8205.0 8233.50 8265.0 8313.25 11625.0
    symcc 82800 17.0 8400.588235 676.300789 8143.0 8193.00 8260.0 8279.00 11016.0
    mopt 82800 20.0 8582.800000 979.570883 8215.0 8230.75 8253.5 8326.00 11503.0
    aflplusplus 82800 20.0 8239.900000 46.769198 8118.0 8211.75 8252.5 8277.75 8291.0
    symqemu 82800 20.0 8464.400000 689.521909 8159.0 8215.50 8244.5 8274.00 10525.0
    aflsmart 82800 20.0 8259.900000 46.873739 8204.0 8227.00 8243.5 8278.25 8356.0
    eclipser 82800 20.0 8395.100000 673.758031 8206.0 8220.50 8243.5 8267.75 11255.0
    entropic 82800 20.0 8198.200000 43.124788 8141.0 8166.75 8196.0 8215.00 8302.0
    honggfuzz 82800 20.0 8215.200000 228.845294 8091.0 8122.25 8181.0 8198.75 9168.0
    libfuzzer 82800 20.0 7932.700000 239.825616 7602.0 7702.00 8095.5 8137.25 8212.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 13774.95 8.369588 13752.0 13775.25 13779.0 13779.00 13785.0
    aflsmart 82800 20.0 13778.50 1.933091 13775.0 13777.00 13779.0 13779.00 13783.0
    z3 82800 20.0 13774.25 10.789249 13752.0 13777.00 13778.0 13779.00 13787.0
    afl 82800 20.0 13777.35 1.348488 13775.0 13777.00 13777.0 13779.00 13779.0
    eclipser 82800 20.0 13778.20 3.396593 13775.0 13775.00 13777.0 13779.00 13785.0
    mopt 82800 20.0 13775.50 5.907177 13751.0 13775.00 13777.0 13777.00 13779.0
    symcc 82800 20.0 13773.60 6.003508 13759.0 13773.25 13777.0 13777.00 13779.0
    symqemu 82800 20.0 13774.90 4.700504 13759.0 13775.00 13777.0 13777.00 13779.0
    entropic 82800 20.0 13766.60 11.962221 13745.0 13755.00 13769.5 13778.00 13779.0
    jigsaw 82800 20.0 13766.35 12.398111 13750.0 13752.00 13768.0 13779.00 13779.0
    fuzzolic 82800 20.0 13769.20 9.294028 13753.0 13761.50 13767.5 13778.00 13780.0
    libfuzzer 82800 20.0 13744.00 6.316228 13724.0 13742.00 13742.0 13750.00 13751.0
    honggfuzz 82800 20.0 13738.40 11.690752 13727.0 13729.00 13732.5 13752.25 13758.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread-2019-12-23 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    jigsaw 82800 20.0 7318.35 610.625243 5748.0 7181.00 7199.5 7220.75 8619.0
    z3 82800 20.0 7385.40 794.069757 5837.0 7176.25 7197.5 7475.50 9338.0
    symcc 82800 20.0 6071.95 378.441746 5850.0 5900.75 5935.0 5950.50 7054.0
    fuzzolic 82800 20.0 5909.10 32.884007 5856.0 5890.25 5912.0 5933.25 5949.0
    entropic 82800 20.0 5892.90 14.472842 5850.0 5890.00 5897.0 5901.00 5916.0
    libfuzzer 82800 20.0 5891.15 8.875009 5877.0 5886.00 5887.5 5895.50 5910.0
    symqemu 82800 20.0 5903.30 200.671268 5816.0 5843.25 5862.5 5881.25 6750.0
    mopt 82800 20.0 5815.90 140.108191 5244.0 5826.25 5857.0 5871.00 5897.0
    eclipser 82800 20.0 5847.95 37.276315 5736.0 5838.00 5854.0 5882.25 5889.0
    aflplusplus 82800 20.0 5841.95 23.493504 5822.0 5827.75 5837.5 5840.25 5921.0
    afl 82800 20.0 5829.55 39.662358 5716.0 5814.75 5832.0 5854.00 5911.0
    aflsmart 82800 20.0 5782.10 132.662572 5250.0 5789.25 5824.5 5836.75 5856.0
    honggfuzz 82800 20.0 5828.40 49.508532 5720.0 5812.75 5824.5 5872.75 5888.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4-2017-08-14 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 7793.10 78.750706 7697.0 7727.25 7789.0 7834.50 7940.0
    libfuzzer 82800 20.0 7688.10 111.489863 7406.0 7624.00 7687.0 7773.75 7875.0
    honggfuzz 82800 20.0 7695.05 84.786093 7542.0 7638.75 7678.5 7715.00 7864.0
    eclipser 82800 20.0 7073.20 165.101755 6603.0 7038.25 7115.5 7160.50 7289.0
    jigsaw 82800 20.0 6925.80 62.634781 6797.0 6891.75 6919.0 6971.00 7048.0
    z3 82800 20.0 6762.80 157.454956 6512.0 6658.75 6785.0 6815.25 7110.0
    fuzzolic 82800 20.0 5843.40 126.113316 5625.0 5786.00 5836.5 5902.50 6154.0
    aflplusplus 82800 20.0 5602.85 120.008015 5452.0 5509.00 5563.5 5686.50 5861.0
    symcc 82800 20.0 5374.25 157.883660 5149.0 5252.00 5365.0 5496.00 5620.0
    symqemu 82800 20.0 5355.75 154.432024 5178.0 5229.25 5314.0 5435.00 5657.0
    aflsmart 82800 20.0 4581.90 721.583194 3154.0 4303.25 4979.0 5105.25 5267.0
    afl 82800 20.0 4541.60 682.244315 3133.0 4198.25 4840.5 5043.75 5158.0
    mopt 82800 20.0 4221.75 655.483339 3094.0 3729.25 4325.0 4707.25 5066.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2-2014-12-09 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 3542.10 18.951184 3516.0 3523.00 3555.0 3558.00 3564.0
    fuzzolic 82800 20.0 3535.30 18.719818 3510.0 3517.75 3544.5 3552.50 3559.0
    z3 82800 20.0 3533.40 20.734918 3504.0 3516.00 3533.5 3553.25 3558.0
    afl 82800 20.0 3527.45 19.030377 3503.0 3509.75 3527.0 3545.25 3555.0
    symcc 82800 20.0 3531.25 17.905821 3507.0 3517.25 3521.5 3548.75 3554.0
    symqemu 82800 20.0 3525.80 15.746679 3507.0 3517.00 3519.0 3528.50 3555.0
    jigsaw 82800 20.0 3527.25 16.970175 3510.0 3516.00 3518.0 3546.50 3558.0
    aflplusplus 82800 20.0 3524.15 15.267009 3508.0 3514.00 3517.0 3540.75 3553.0
    eclipser 82800 20.0 3527.55 21.421767 3507.0 3511.00 3517.0 3549.00 3579.0
    aflsmart 82800 20.0 3522.30 16.157449 3506.0 3511.00 3515.0 3541.25 3550.0
    libfuzzer 82800 20.0 3512.10 15.231201 3474.0 3511.50 3514.0 3517.00 3552.0
    mopt 82800 20.0 3522.65 21.492410 3472.0 3508.50 3514.0 3544.00 3547.0
    honggfuzz 82800 20.0 3510.15 12.807379 3500.0 3504.00 3505.0 3506.75 3545.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: symcc.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 36684.80 193.007663 36163.0 36604.00 36699.0 36793.25 37031.0
    fuzzolic 82800 20.0 35783.20 476.576011 34425.0 35524.25 35922.5 36062.75 36451.0
    z3 82800 20.0 35836.05 270.917248 35232.0 35618.00 35886.5 36030.50 36239.0
    symqemu 82800 20.0 35836.40 362.761253 35046.0 35621.75 35845.5 36134.25 36344.0
    jigsaw 82800 20.0 35744.25 335.216956 35169.0 35477.25 35767.0 35896.25 36451.0
    symcc 82800 4.0 35471.50 367.079011 35088.0 35196.75 35478.5 35753.25 35841.0
    aflsmart 82800 20.0 35095.70 336.804504 34459.0 34949.25 35114.5 35303.25 35724.0
    afl 82800 20.0 34877.70 909.362825 32344.0 34776.25 35071.0 35434.25 35723.0
    mopt 82800 20.0 35012.80 469.740869 34109.0 34805.50 35045.0 35328.75 35670.0
    eclipser 82800 20.0 34675.45 408.497951 33871.0 34381.00 34733.0 34996.50 35434.0
    entropic 82800 20.0 31296.45 654.333367 30130.0 30875.75 31372.5 31766.00 32352.0
    honggfuzz 82800 20.0 30340.25 604.276677 28827.0 30082.50 30390.5 30602.25 31677.0
    libfuzzer 82800 20.0 23851.30 1056.883856 22099.0 23139.50 23970.0 24521.50 25417.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis-2017-12-11 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    jigsaw 82800 20.0 2169.65 2.323224 2166.0 2168.00 2169.5 2171.00 2175.0
    aflplusplus 82800 20.0 2168.40 2.909151 2165.0 2167.00 2168.0 2169.00 2177.0
    fuzzolic 82800 20.0 2168.80 4.323741 2162.0 2166.00 2168.0 2170.25 2178.0
    symqemu 82800 20.0 2168.35 3.602265 2164.0 2167.00 2168.0 2169.00 2180.0
    symcc 82800 20.0 2168.25 4.024595 2163.0 2165.75 2167.5 2170.00 2180.0
    mopt 82800 20.0 2167.95 5.889107 2158.0 2164.50 2167.0 2173.25 2178.0
    aflsmart 82800 20.0 2166.85 4.934039 2159.0 2163.00 2166.5 2168.25 2177.0
    z3 82800 20.0 2166.30 1.750188 2163.0 2165.00 2166.5 2167.25 2169.0
    afl 82800 20.0 2165.70 5.694873 2152.0 2163.75 2166.0 2167.00 2179.0
    eclipser 82800 20.0 2165.20 4.237676 2157.0 2162.00 2165.5 2167.00 2176.0
    entropic 82800 20.0 2131.70 103.035559 1830.0 2163.00 2165.0 2165.25 2172.0
    honggfuzz 82800 20.0 2150.75 7.765477 2135.0 2150.00 2152.5 2156.00 2160.0
    libfuzzer 82800 20.0 1976.15 165.904943 1829.0 1830.00 1830.5 2155.50 2159.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2-2016-05-06 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fuzzolic 82800 20.0 1932.60 18.636128 1871.0 1926.00 1936.5 1942.50 1959.0
    symcc 82800 20.0 1933.10 17.873532 1898.0 1920.00 1934.0 1950.25 1956.0
    symqemu 82800 20.0 1927.70 21.006515 1890.0 1906.75 1934.0 1947.75 1953.0
    honggfuzz 82800 20.0 1888.30 21.940350 1836.0 1873.50 1893.0 1907.25 1915.0
    eclipser 82800 20.0 1880.70 23.468007 1849.0 1858.50 1883.0 1894.00 1929.0
    z3 82800 20.0 1866.70 18.646574 1829.0 1848.75 1875.5 1880.00 1889.0
    aflplusplus 82800 20.0 1869.10 16.631929 1836.0 1864.25 1871.5 1881.00 1890.0
    aflsmart 82800 20.0 1861.45 24.191071 1789.0 1846.75 1863.0 1883.00 1890.0
    afl 82800 20.0 1861.30 18.067418 1839.0 1843.75 1859.0 1876.50 1887.0
    jigsaw 82800 20.0 1859.10 15.589808 1838.0 1844.50 1858.0 1870.00 1887.0
    mopt 82800 20.0 1860.75 16.764232 1829.0 1851.25 1856.0 1876.00 1891.0
    entropic 82800 20.0 1841.65 87.216171 1698.0 1757.00 1837.0 1922.75 1959.0
    libfuzzer 82800 20.0 1697.10 55.169500 1599.0 1654.00 1699.5 1745.00 1799.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symqemu 82800 20.0 964.30 7.671547 946.0 962.25 964.0 965.0 983.0
    fuzzolic 82800 20.0 965.60 8.100682 960.0 961.00 963.0 964.0 985.0
    symcc 82800 20.0 962.75 1.970172 960.0 961.00 963.0 964.0 966.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

from cached data