Context of V&V Technology
1. What is your role in your organization?
Response Count Software Tester 5 Quality Assurance 4 Verification and Validation 5 Software Developer 28 Project Manager 5 Researcher 9 Academic 4 Other 0
2. How many years of experience do you have with V&V?
Less than one year 1-2 years 3-5 years 6 years or more 2 5 8 18
3. What is the scope of verification and validation that you perform?
- Ad hoc
Response Count Unit 29 Integration 23 System 18 Acceptance 9 Other 0
- Accessibility
- model checking
4. What programming languages do you use?
Years of experience with Java:
6, 3, 2, 6, 8, 9, 6, 8, 7, 7, 10, 8, 9, 7, 10, 8, 10, 8, 10, 7, 2, 8, 9, 10, 5, 5, 6, 8, 6, 8, 6, 12
Current version of Java used:
1.5, 5.0, 1.4.2, 1.5, 5, 5, Java SE 6 beta2, Mustang, 1.4, JDK 5, Mustang b76, 1.5, 1.6, 5, 5.0, 1.5, 6.0, 1.4, 5.0, 5, 1.5, jdk1.5, 1.5, 1.4.2, 1, 5, 1.4.2, 1.5, 1.4, jdk 5
Years of experience with C++:
1, 6, 15, 2, 15, 11, 3, 10, 10, 18, 1, 2, 23, 3, 10, 8, 2, 10+, 5, 4, 10, 15 (company), 10, 1
Current version of C++ used:
Gnu 4.0, gcc 4; MSVC 6, Initial C++ spec (10 years ago), n/a, ?, gcc 3.4.5, 1998, 1998, ?
Years of experience with Ada:
10, <1, 1, 20 (company)
Current version of Ada used:
not current, n/a, ?
Other programmling languages used: - Haskell (12 years) Esterel (5 years) C# (2 years) F# (1 year) Pascal (12 years) BCPL (6 years) C (15 years) BASIC (7 years)
- C COBOL PL/1
- Python, Tcl, C, and Basic.
- C 23 years, Sun C 5.2/gcc 4.0 Pascal 15 years, Encore ISO Pascal Fortran 24 years, Encore Fortran Modula-2, Modula-3, 7 years PL/1, PL/N (PL/1-like proprietary language) 3 years Myriad assembly languages spanning ~25 years, notably: x86 Assembler 3 years (wrote an embedded OS w. preemtive "thread" scheduler & priority interrupt handling that supported interpretive "application programs"
- Perl, 4, 5.8.5
- Perl, ~10, 5.8+ Lisp, since 1980s, n/a Fortran, since 1970s, n/a Haskell, <1, recent GHC and Hugs etc.
- C, 11, ? Progress,10,9
- Objective-C : 7 years C : 3 years (~40% of ObjC dev) Smalltalk: 3 years
- C, 15 Yrs., I don't use it any more. I've used many other languages over the years, but not professionally.
- Prolog: 2 years C: 2 years Pascal: years
- C (gcc 4.0), 15 years exp.
- Perl, shell scripting, C, Pascal - 10 years
- Fortran 1 year 77
- Various assembly languages, 10 years experience C#, 3 years
- C, Perl, Pascal
- x86 Assembly 2 years IBM 370 assembly 2 years C 3 years
- Java, Lisp, C.
- Python 2 2.4
- C Pascal
- Fortran 2 fortran77 XML 8 2.0(?) beanshell 1
5. What concurrency defects are (or have been) an issue in your work?
- Correctness
Response Count Deadlock 31 Interference 27 Starvation 17 Other 6
- livelock, missed wakeups
- priority inversion, live lock, defective synchronization
- livelock
- race conditions
- race conditions (synonym for interference?)
- lock contention (although not a true defect.. is can cause serious problems), and not enough locking in databases (or too much locking in databases)
- Data races (interference)
6. a) What V&V technologies do you use?
- Jtest
Response Count Code inspection/code walkthrough 32 Pair programming 7 Automated static analysis tools 18 Coverage 15 Capture/Playback 5 Model-checker 7 Development testing tools (JUnit) 29 Other 3
- manual flow analysis on Whiteboard
- Home grown JSP checker based on regular expressions and Tag EI classes
- System automated test
6. b) If you marked code inspection/code walkthrough, please choose the style you most often apply:
individual-systematic individual-ad hoc group-systematic group-ad hoc 6 17 5 4
7. a) Is any of the V&V technology you use open source?
Yes No 21 12
7. b) If yes, then please list the open source tools you use:
Zing SMV ZChaff
junit
junit, macker, ...
findbugs, checkstyle, PMD, clover, JUnit,
Spin
JUnit, FindBugs, EMMA
FindBugs
JUnit, FindBugs
FindBugs, PMD, manual techniques (inserting dummies/tracers/boobie traps etc.)
findbugs, junit
JUnit
i have built a tool that i intend to release in the near future
findbugs
JUnit, JUnitEE, Checkstyle, Selenium
SPIN
junit, findbugs, cobertura
XMC/Concurrency Factory from SUNY Stony Brook, NY, USA; Patrice Godefroid's Verisoft verification tool; PVS from SRI International, Menlo Park, CA, USA.
I do not have a list right now... sorry.
JUnit, FindBugs, PMD, Checkstyle
junit
Decision Process for V&V Use
8. For each of the following sources of information, indicate your level of agreement with the statement that the particular source of information is an important factor in determining what V&V technology to use.
Question SA A N D SD History of use in other companies 7 15 6 3 1 History of use in your company 12 14 5 1 0 Marketing information 0 5 16 4 7 Search Engine results on the V&V technology 1 11 14 6 1 Newgroups/Forums 6 15 9 3 0 Research Literature 9 14 7 3 0
Information from Case Studies
9. If you consider case studies or experience reports in your decisions to apply certain V&V technologies, indicate your level of agreement with the statement that the particular aspect of a case study is an important factor in determining what V&V technology to use.
Question SA A N D SD NA The empirical study took place in industry. 4 12 9 0 0 6 There was a large number of participants taking part in the study. 4 8 8 5 1 5 The study involved participants that applied the V&V technology. (i.e.the evaluation was not automated). 6 8 11 1 0 5 The defects in the study were real, as opposed to artificially seeded. 10 12 4 0 1 5 The case study was in a similar application domain to yours. 4 13 7 3 0 5
Other aspects of case studies that you consider important:
Who performed them
attacking "hard" problems (e.g. non-statically determined race condition detection) instead of solving problems that can often attributed to bad practice/bad APIs/language defects
Some form of measurable (or ideally quantifiable) result and the time taken to find such results. eg. number of bugs found, manual time spent setting up tool, time tool took to run. Sample output from tool.
How the tool or methodology scales to industrial scale (i.e. very large and rapidly evolving) software projects.
Scalability
10. Rate the following aspects of cost-effectiveness of V&V technology in terms of their relative importance. (1-highest importance, 2-middle importance, 3-lowest importance).
The V&V tool finds all faults of a certain type.
1 3 1 1 1 1 2 3 1 1 3 1 1 2 1 2 1 2 1 1 1 2 1 3 1 1 2 2 1 3 2 2 2 1 2
The V&V tool does not report any false positives (i.e. warnings that turn out not to be errors).
1 1 3 3 3 1 2 3 3 3 1 3 2 3 3 3 1 1 2 2 2 2 2 3 1 2 3 3 2 2 2 3 3 3 2
The cost of using the V&V tool relative to the number of errors detected.
1 2 1 2 2 1 1 1 2 2 2 2 3 1 2 1 1 1 1 3 3 1 3 2 1 1 1 3 1 1 1 1 3 2 3
Other aspects of cost-effectiveness of V&V technology that you consider important:
automation capabilities/need for manual conversion of code to evaluate cycle time for analysis
Ease of integration into the development cycle - if it cannot be executed in an automated build cycle it's not as effective.
"automated" operation is a plus (batch runs vs. interactive use), but (apparently fictional?) interactive systems driven by e.g. expert/rule systems with feedback cycles would be OK too as long as they help find/crack the "hard problems".
Really the number and quality of error reports, rather than "false alarm, real bug" is more important. Because it takes work to figure out whether an alarm is true, and then once figured out, you have to figure out how to fix it. Even if a tool only output true alarms, if those alarms were not helpful in finding and fixing the underlying bugs they wouldn't be very helpful.
Whether V&V can be automated - 1 Time taken to learn - 2 Ease of use of report output - 2
"Saleability" of results to the academic community; the goal of V&V in research papers is often to be able to convince the reader (who may be an expert in a field other than V&V) that the algorithm under test is actually correct. Model checking of models derived from source is therefore particularly useful.
The purchase cost of tool does not include the cost to implement; it's important that the tool itself be easy to set-up and train people with. If a tool is practically free (in terms of cost/time to implement and integrate) and only catches 5-10% of concurrency defects that other tools do not, it's very useful in my opinion.
The cost of using the tool is by far the most important. Note that cost includes such aspects as whether the software source code requires manual translation into a language that the tool understands. Tools that consume the source code directly and which scale to millions of lines of source code are the ones that industry could really use. The rest, which constitute the majority of v&v tools on the market, are merely academic curiosities.
The errors reported are likely to be high priority. (some tools find a lot of "real" errors, but they are not things likely to make a difference in practice)
Cost of maintenance also important.