Developer Perspectives on Software Testing: A Study
Software testing is a critical phase in the software development lifecycle, ensuring the reliability and quality of software products. However, anecdotal evidence suggests that developers' attitudes towards testing are varied. This paper presents findings from a Prolific study involving 100 software professionals to investigate their views on software testing. The study explores the time allocation for testing, perceived value, personal enjoyment, test suite sufficiency, and team responsibilities related to testing. The results provide quantitative insights into developer perceptions, highlighting both the recognized importance of testing and areas for potential improvement in developer engagement and testing practices.
Introduction
Software testing is indispensable for delivering robust and dependable software. It helps identify defects early in the development process, reduces the risk of production failures, and contributes to overall software quality 1. Despite its acknowledged importance, the practical application and developer perception of software testing remain areas of ongoing discussion within the software engineering community. Understanding how developers view testing is crucial for optimizing development workflows, promoting better testing practices, and ultimately improving software quality. This study aims to empirically investigate these perceptions by surveying software professionals on their attitudes and experiences with software testing in their daily work.
Methodology
To understand developer perspectives on software testing, we conducted a survey-based study on Prolific, a platform for online research. We recruited 100 participants who are software professionals with experience in software development and testing. The survey consisted of six key questions designed to capture different facets of their views on software testing:
- Time Allocation: How much time do you spend on software testing versus building? (writing & maintaining tests, running tests, reviewing failures, etc.) - Categorical responses were provided (e.g., 1-15%, 15-25%, 25-35%, 35-50%, > 50% testing).
- Perceived Value: Do you consider testing valuable when building software? - Responses: Yes, Somewhat, No.
- Personal Enjoyment: Do you personally like testing? - Responses: Yes, Somewhat, No.
- Test Suite Sufficiency: Is your test suite sufficient for the software project(s) you are currently working on? - Responses: Excellent, Good, Insufficient, No tests.
- Testing Responsibility: Who does the testing in your team today? - Responses: Developers write, maintain and run tests, Quality Assurance, Multiple roles, Product Manager (or other business role).
- Demographics: We also collected demographic data including age, sex, ethnicity, country of birth, country of residence, nationality, language, student status, and employment status to provide context for the responses.
The dataset provided includes responses from 50 participants. Data cleaning involved removing incomplete or timed-out responses, resulting in a final sample size of 49 usable responses for analysis. The following section presents the key results derived from this dataset.
Results
The analysis of the survey data reveals several interesting trends in developer perceptions of software testing.
Time Allocation for Testing
Participants were asked to estimate the percentage of their time spent on testing versus building features. The distribution is shown in Table 1.
Table 1: Time Allocation for Testing vs. Building
| Time Allocation | Percentage of Respondents |
|---|---|
| 1-15% testing, 75-99% building | 32.65% |
| 15-25% testing, 75-85% building | 36.73% |
| 25-35% testing, 65-75% building | 18.37% |
| 35-50% testing, 50-65% building | 10.20% |
| > 50% testing | 2.04% |
| 0%, I don't test | 0.00% |
The majority of respondents (36.73%) reported spending 15-25% of their time on testing, with a significant proportion (32.65%) spending even less (1-15%). This suggests that while testing is part of the development process, it often occupies a smaller portion of developers' time compared to feature building.
Perceived Value of Testing
The overwhelming majority of participants recognize the value of software testing, as shown in Table 2.
Table 2: Perceived Value of Software Testing
| Perceived Value | Percentage of Respondents |
|---|---|
| Yes, you cannot build software without testing | 91.84% |
| Somewhat, testing is useful but not necessary | 6.12% |
| Somewhat, I see it the same as building features | 2.04% |
| No, I prefer building features (not testing them) | 0.00% |
Approximately 91.84% of respondents believe testing is essential ("Yes, you cannot build software without testing"). A small percentage (6.12%) consider it useful but not strictly necessary, while a very small minority view it as similar to feature building or prefer building features over testing. No participants indicated a complete lack of value in testing.
Personal Enjoyment of Testing
Developer enjoyment of testing is more varied compared to their perception of its value, as presented in Table 3.
Table 3: Personal Enjoyment of Software Testing
| Personal Enjoyment | Percentage of Respondents |
|---|---|
| Yes, I enjoy writing, maintaining and running tests | 32.65% |
| Somewhat, I see it the same as building features | 32.65% |
| No, I prefer building features (not testing them) | 34.69% |
The responses are almost evenly distributed. While 32.65% enjoy testing, a similar percentage (32.65%) see it neutrally, and a slightly larger group (34.69%) express a preference for feature building over testing. This highlights a potential challenge in fostering enthusiasm for testing activities among developers.
Test Suite Sufficiency
Participants assessed the sufficiency of their current project's test suite, with results shown in Table 4.
Table 4: Test Suite Sufficiency
| Test Suite Sufficiency | Percentage of Respondents |
|---|---|
| Insufficient: There are a few tests but they are not complete and/or up to date | 44.90% |
| Good: We have good but not perfect test coverage | 42.86% |
| Excellent: Our testing covers all critical aspects of our application | 12.24% |
| No tests: There are no or very few tests | 0.00% |
A significant majority of respondents indicated that their test suites are not fully sufficient. Approximately 44.90% reported their test suites as "Insufficient," and 42.86% as "Good but not perfect." Only 12.24% considered their test coverage to be "Excellent." This suggests a widespread awareness of the need for improved testing practices and test coverage within software teams.
Responsibility for Testing
The study also investigated who is primarily responsible for testing within the participants' teams, as shown in Table 5.
Table 5: Responsibility for Testing
| Testing Responsibility | Percentage of Respondents |
|---|---|
| Developers write, maintain and run tests | 59.18% |
| Quality Assurance | 20.41% |
| Multiple roles | 18.37% |
| Product Manager (or other business role) | 2.04% |
In most teams (59.18%), developers are primarily responsible for writing, maintaining, and running tests. Quality Assurance (QA) specialists are responsible in 20.41% of cases, and a shared responsibility model ("Multiple roles") exists in 18.37% of teams. Product managers or other business roles are rarely involved directly in testing.
Discussion
The findings of this study reveal a nuanced perspective on software testing among software professionals. While there is a strong consensus on the value and necessity of testing, developers generally perceive it as a secondary activity in terms of time allocation compared to feature development. Interestingly, despite recognizing the importance of testing, personal enjoyment of testing is not universally shared, with a considerable portion of developers preferring feature building.
The perceived insufficiency of test suites in a significant number of projects is a notable concern. This gap between recognizing the value of testing and having adequate test coverage suggests potential challenges in implementing effective testing practices. These challenges could stem from time constraints, lack of resources, or insufficient tooling and automation.
The dominant model of developers being primarily responsible for testing aligns with modern software development practices like Agile and DevOps, which advocate for developer ownership of quality. However, the mixed levels of enjoyment and perceived test sufficiency indicate a potential need for strategies to improve developer engagement with testing and provide them with better tools and support.
Conclusion
This study provides empirical data on developer views on software testing, highlighting a blend of recognition of its importance and existing challenges in practice. The findings underscore the need for continued efforts to:
- Improve developer experience with testing: Making testing more enjoyable and integrated into the development workflow could increase developer engagement and ownership.
- Enhance testing efficiency and automation: Tools and techniques that reduce the time burden of testing without compromising quality are crucial, as indicated by the time allocation data.
- Promote a culture of quality: While value is recognized, translating this into consistently sufficient test coverage requires ongoing focus on quality as a shared responsibility and priority.