Paper 21 (Research track)

Enhancing Fuzzing with Test Case Prioritization Techniques

Author(s): Gen Zhang

Full text: submitted version

Abstract: Fuzzing is an efficient testing technique to catch bugs early, before they turn into vulnerabilities. Without complex program analysis, fuzzers extended with coverage information can generates interesting results and find potential bugs in programs. However, we observed that previous coverage-based fuzzers, such as American Fuzzy Lop (AFL) , fail to realize the importance of the order of input test cases or they are unable to adopt useful coverage information, so some of them suffer from dramatically poor performance. Meanwhile, the main idea of test case prioritization (TCP) in the field of software testing is to rank the test cases according to a certain rule and leverage coverage information between source code and test cases, to help identify bugs and vulnerabilities. Thus our work concentrates on complementing fuzzing techniques with the characteristics of TCP and improving the effectiveness and efficiency of traditional fuzzers.

In this paper, we present a combined fuzzing technique, integrated with useful coverage information and prioritization properties commonly used in TCP, which enhances the process of creating new test cases and finding bugs. In order to achieve expected results, we implement our method by extending state-of-the-art fuzzer AFL with TCP techniques and evaluate it on 6 widely-used programs from GNU. We conduct experiments on 6 target programs to illustrate our performance both on bug detection and time cost. On all of these experiments, improvement of our method is witnessed and significantly better outcomes are generated.

Keywords: fuzzing; test case prioritization; coverage information; security

Decision: reject

Review 1 (by Alessandro Margara)

(RELEVANCE TO ESWC) Not relevant.
(NOVELTY OF THE PROPOSED SOLUTION) Cannot judge, since the paper is out of the scope of the conference.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) Cannot judge, since the paper is out of the scope of the conference.
(EVALUATION OF THE STATE-OF-THE-ART) Cannot judge, since the paper is out of the scope of the conference.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Cannot judge, since the paper is out of the scope of the conference.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Cannot judge, since the paper is out of the scope of the conference.
(OVERALL SCORE) The paper is clearly out of scope for ESWC, since it discusses a new technique
for software testing. Proper venues for this paper are conferences on software
testing (ISSTA, ICST) or more in general on software engineering (ICSE, FSE).


Review 2 (by Lavdim Halilaj)

(RELEVANCE TO ESWC) Not relevant.
(NOVELTY OF THE PROPOSED SOLUTION) Not relevant.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) Not relevant.
(EVALUATION OF THE STATE-OF-THE-ART) Not relevant.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Not relevant.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Not relevant.
(OVERALL SCORE) The paper presents a technique that integrates coverage information and prioritization properties to enhance the process of creating new test cases and finding bugs.
Although the paper is well-written, unfortunately, according to my opinion it doesn't fit with any topic of the Benchmarking and Evaluation track or topics of the conference in general.
Moreover, the paper is not written according to the required conference format.


Review 3 (by Serena Villata)

(RELEVANCE TO ESWC) The main issue with the paper is the lack of connections with the Semantic Web. It is not clear how the present paper makes a contribution in the Semantic Web area. The author should consider submitting the paper to another conference.
(NOVELTY OF THE PROPOSED SOLUTION) I don't feel enough expert in the domain of the paper to assess its novelty.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) I don't feel enough expert in the domain of the paper to assess its correctness and completeness.
(EVALUATION OF THE STATE-OF-THE-ART) I don't feel enough expert in the domain of the paper to assess whether the comparison with the state-of-the-art is appropriate.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) These are the main positive points:
- the paper evaluates the proposed approach on 6 real-world applications
- the experimental results show an improvement in performance thanks to the proposed approach
- the paper presents an extension of the state-of-the-art American Fuzzy Lop
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) I don't feel enough expert in the domain of the paper to assess reproducibility and generality of the experimental study.
(OVERALL SCORE) The paper proposes a combined fuzzing technique, integrated with useful coverage information and prioritization properties commonly used in TCP. Fuzzing is an efficient testing technique to catch bugs early. The approach is evaluated on 6 real-world applications.
Positive points:
- the paper evaluates the proposed approach on 6 real-world applications
- the experimental results show an improvement in performance thanks to the proposed approach
- the paper presents an extension of the state-of-the-art American Fuzzy Lop
Negative points:
- the main issue with the paper is the lack of connections with the Semantic Web. It is not clear how the present paper makes a contribution in the Semantic Web area. 
- the paper is very technical and difficult to follow for non-experts as I am
- examples are missing


Metareview by Emanuele Dellavalle

The paper is out of the scope of the conference.


Share on

Leave a Reply

Your email address will not be published. Required fields are marked *