Survey Questions

Thanks for taking part in this anonymous survey. Our goal is to understand the characteristics of source code recommender systems that are important for developers.
With source code recommender systems, we refer to approaches that can be used to automatically suggest code to developers while they are writing code.
The classic example in this context is the code completion feature implemented in several IDEs. However, some tools go beyond the classic code completion task and recommend longer pieces of code to developers to autocomplete a task they perform (see, e.g., https://copilot.github.com/).

In the following, we use the term "code recommender" to refer both to standard code completion tools as well as to more advanced tools recently proposed. Also, we use the term "recommended code" to refer to the code automatically generated by the code completion tool. The survey is designed to take no more than 15 minutes.

Q1: Which role better describes your current job position

Q2: How many years of experience do you have?

040

Q3: Which programming language(s) do you mostly use?

Q4: Which tool(s) or plugin(s) do you use to get code recommendations?

Q5: When typing code, how often do you exploit code recommendations?

Q6: Please rate the code recommender capabilities of the tools/IDEs you use accordingly to the following dimensions:

Low Medium High Not Sure
Coverage of the recommendation: how different are the coding scenarios in which the code recommender is able to provide support. For example, a code completion tool able to recommend variable names and method calls has a higher coverage as compared to a technique only able to recommend method calls.
Accuracy of the recommendation: how likely it is that the code recommended is correct.
Usability of the recommendation: how friendly the user interface is.

Q7: Concerning the three previous dimensions, what improvements would you like to see, such as code recommendation features of the tool/IDE?

Coverage of the recommendation: how different are the coding scenarios in which the code recommender is able to provide support. For example, a code completion tool able to recommend variable names and method calls has a higher coverage as compared to a technique only able to recommend method calls.

Accuracy of the recommendation: how likely it is that the code recommended is correct.

Usability of the recommendation: how friendly the user interface is.

Q8: Which are the most important characteristics of the code recommendation that strongly impact its usefulness (and, as a consequence, the likelihood that you will accept and use the recommendation)?
Please list at least three of them sorted by importance, accompanying each one with a short explanation. Just a note: Use "Correctness" if you want to refer to the fact that the recommended code must be bug-free, use "Accuracy" if you want to indicate that the recommendations received are usually precise and match what the developer wants to write.

Q9: Concerning the accuracy of the recommendation. Let’s assume that the code recommender tool you use provides — for the code you are writing — a list of possible recommendations sorted by likelihood of being relevant (i.e., the most relevant on top). How many of these recommendations would you be willing to read in order to find the right one?
For example, TOP 3 means that the correct recommendation is within these three recommendations. Otherwise, you believe it’s faster just manually to type the code.

Q10: Which of the following characteristics of a code recommendation would strongly impact its usefulness (and, as a consequence, the likelihood that you will accept and use the recommendation)? Select the most relevant categories for you.