A Quantitative Analysis of Survey Data for Software

Design Patterns

Cheng Zhang Futian Wang Rongbin Xu

Anhui University Anhui University Anhui University
No. 111 Jiulong Road No. 111 Jiulong Road No. 111 Jiulong Road
Hefei, China Hefei, China Hefei, China
cheng.zhang@ahu.edu.cn wft@ahu.edu.cn xurongbin910@gmail.com
Xuejun Li Yun Yang
Anhui University Anhui University
No. 111 Jiulong Road No. 111 Jiulong Road
Hefei, China Hefei, China
xjli@ahu.edu.cn yyang@swin.edu.au

ABSTRACT design has become quite extensive. Design patterns provide

Software design patterns are largely concerned with improv- access to design knowledge through the description of a set
ing the practices and products of software development. How- of successful solutions extracted from previous experiences
ever, there has been no systematic analysis how users’ pro- with respect to a certain type of problem.
files influence the effectiveness of design patterns. The aim From the concept of design patterns, it has two motivat-
of this paper is to investigate the links between the respon- ing elements in the definitions: ‘design patterns are suc-
dents’ demographic data from our previous online survey cessful solutions’ and ‘design patterns solve recurring prob-
and the design patterns. In this paper we employ a sta- lems’. For the first element ‘design patterns are successful
tistical approach to analyse the quantitative data collected solutions’, developers undoubtedly expect to improve their
from the respondents of our previous online survey. Through products successfully via reusing successful solutions of de-
analysing the demographic data from the 206 responses of sign patterns. However this ideal expectation is based on
the questionnaire, we find that the positive assessment per- effective use of design patterns. Frakes and Kang have indi-
centage of using patterns increases with greater experience cated that software quality and productivity are determined
with design patterns. The results show that the functions of by software reuse [5]. Therefore, as a reuse technique, de-
design patterns are influenced by users’ experiences rather sign patterns influence on software quality and productivity.
than users’ roles. However, we cannot be sure how effective design patterns
are, before a systematic investigation is conducted.
For the second element ‘design patterns solve recurring
Categories and Subject Descriptors problems’, design patterns are means of sharing and trans-
D.2.2 [Design Tools and Techniques]: Object-Oriented ferring experiences of software design between designers and
Design Methods learners. But the large number of researches is convinced
that design patterns are only suitable for use by experienced
General Terms developers. This made us doubt whether reusing design pat-
terns can be used to solve recurring problems under any con-
Measurement, Theory dition. Therefore it has motivated us to investigate people’s
experiences to see how far they are right.
Keywords For the two problems mentioned above, in the previous
design patterns, quantitative analysis, survey, empirical studies we performed a systematic literature review in the
form of a mapping study, conducted an online survey and
a follow-on survey. The systematic review was applied to
1. INTRODUCTION identify how extensively patterns have been studied and also
From twenty years ago, starting with the well-known text- provide the basis for both the organisation the online sur-
book by the ‘Gang of Four’ [6], often abbreviated to GoF, vey [17]. The online survey and the follow-on survey were
the literature on design patterns and their use in software conducted to investigate the specific patterns and their use-
fulness as perceived by researchers and developers, drawing
upon their experiences [18, 19]. All the previous studies are
Permission to make digital or hard copies of all or part of this work for personal or
classroom usetois make
granteddigital
withoutorfeehard
provided thatofcopies described in the next section.
Permission copies all orarepart
not made or work
of this distributed
for
for profit or commercial advantage and that copies bear this notice and the full citation
personal or classroom use is granted without fee provided that copies are The previous studies focus on the effectiveness of the spe-
on the first page. Copyrights for components of this work owned by others than ACM
not made
must or distributed
be honored. Abstractingforwith
profit or iscommercial
credit permitted. Toadvantage and that
copy otherwise, copies
or republish,
cific design patterns. However, we are also interested in the
to postthis
bear on notice
servers and
or tothe
redistribute to lists,
full citation onrequires
the firstprior specific
page. permission
To copy and/ortoa
otherwise, links between the respondents’ demographic roles and the
fee. Requesttopermissions
republish, from Permissions@acm.org.
post on servers or to redistribute to lists, requires prior specific general usage of all design patterns based on the quantita-
permissionMay
EAST’14, and/or
26, a2014,
fee. Nanjing, China tive analysis results. Although we provide some outcomes
EAST ’14, 2014
Copyright May 26,
ACM 2014, Nanjing, China
978-1-4503-2965-1/14/05...$15.00 which describe the links between the demographic roles and
http://dx.doi.org/10.1145/2627508.2627516
Copyright 2014 ACM 978-1-4503-2965-1/14/05 ...$15.00.

48
the design patterns in the previous studies, we decide to
analyse the demographic quantitative data in depth and in-
vestigate links between the respondents’ background and the
usage of design patterns. The response data are based on
the online survey. Therefore, our research question for this
study was
“What lessons can we learn from the links be-
tween the respondents’ demographic data and the
design patterns? ”
In the following section we provide a little more background
about the previous studies. Then Section 3 describes the
analysis method for the quantitative data of the online sur-
vey. In the rest of this paper we present our results and
discuss how far they answer the research questions .
2. BACKGROUND
In this paper our study is based on software design pat-
terns and some forms of the previous studies. The objectives
of this section are to introduce the software design patterns.
And it also describes the previous studies in this section.
2.1 Software Design Patterns
“Patterns are a means of documentation.” [14]
Design patterns are not specific methods as libraries and
frameworks. They provide something much more like guid-
ance for software design and maintenance. However libraries
and frameworks focus on the specific application on pro-
gramming, and they do not consider such design properties
as understandability, flexibility and maintainability. Design
patterns do not focus on coding. They aim to provide access
to successful solutions when software developers meet recur-
ring problems. The structure of design patterns focuses on
the relation of classes and objects rather than the specific
code.
Figure 1 describes how users employ design patterns. First
of all design patterns are the documentation for successful
software design in the minds of developers and maintainers.
They are not specific code for programming reuse. When
developers meet a design problem, they think about how
to choose suitable patterns under that situation. Then the
thinking of the patterns and their structures can be used in
the specific programming process. The application of design
patterns also make the implementation more maintainable.
Design patterns provide the common vocabulary to improve
the communication between designers and maintainers.
Creational
Structural Developer
Design Patterns Design
(Documentation) (Common Vocabulary)
Programming
Behavioral
Maintainer
Figure 1: The Working Process of Design Patterns
to Users
The most basic problem for design patterns is that design
patterns have not been investigated, evaluated and validated
49
systematically. Sometimes software reuse is regarded as a
panacea. Indeed, in some studies about real development,
the results indicate that design patterns do not always lead
to positive products [16, 15]. For some specific patterns, for
example, the Singleton pattern, even one of the authors of
the GoF has doubts about the benefit of it 1 . These conflict-
ing conditions reveal that there is no thorough investigation
on design patterns. Most people have no comprehensive cog-
nition for all design patterns. Therefore, a systematic inves-
tigation, survey for all design patterns are required. More
details about the systematic review and survey are discussed
as follows.
2.2 The Mapping Study
The mapping study is an evidence-based approach widely
used in software engineering research [11]. The concept of
evidence-based research was originally developed in the area
of clinical medicine. With its success in clinical medicine,
evidence-based research was adopted for use in other do-
mains, including software engineering. Evidence-based soft-
ware engineering aims “to provide the means by which cur-
rent best evidence from research can be integrated with prac-
tical experience and human values in the decision making
process regarding the development and maintenance of soft-
ware” [4].
Based on the mapping study method applied in the pre-
vious study, we conducted a very thorough systematic map-
ping study which searched the design patterns literature
from 1995 to 2009. 611 candidate papers were identified
after the mapping study. And then via the selection pro-
cess the mapping study found 10 papers which included 11
experimental studies and 7 experience studies [17]. In the
11 experimental studies and 7 experience papers 17 specific
design patterns were examined. The results of the mapping
study displayed an overview of the field of design patterns,
making clear where evidence is lacking and what needs to
be investigated. And also the outcomes helped to form a
clear framework about design patterns, they provided the
basis for both the organisation the online survey as well as
an input to the final results of this study.
2.3 The Online Survey
Because of the small number of papers containing empiri-
cal evidence about design patterns, and about the usefulness
of particular patterns, it was felt that it would be valuable
to perform a primary study to try to find out more about
which patterns from the 23 in the GoF text were more widely
used (and which ones were little used). Therefore, it was felt
that performing an online survey could provide a compre-
hensive overview of the ranking of all 23 GoF design pat-
terns, and improve and complement the analysis from the
mapping study.
The purpose of the online survey was therefore to investi-
gate how deeply design patterns influenced software develop-
ers’ experiences, which design patterns were most frequently
applied, the ones they were most reluctant to use, and the
reasons for their views. Therefore the title of the survey was
defined as “The survey of experience about design patterns”.
The survey form consists of five parts and 47 questions. Each
part has its own questions [18].
Part I is the Introduction part. Part II asks participants
to provide their profiles and offer their general assessments
1
http://www.informit.com/articles/article.aspx?p=1404056
for the 23 design patterns. Part III asks participants to
provide the experiences of the patterns (up to three) which
are considered as useful. Part IV asks participants to pro-
vide the experiences of the patterns (up to three) which are
considered as not useful. Part VI asks participants some
miscellaneous questions and thanks to their contributions.
The details about Part II is shown in Appendix A.
After finishing the survey form design, we performed a
pilot testing. The form was reviewed by a small team of two
assessors. Then the forms were sent out with the requests
after being modified based on the assessors’ comments.
Finally we sent 877 requests and reminders to the authors
of the collected papers in the mapping study. After removing
196 invalid email addresses, there were 681 valid addresses
in our database. We received 136 responses from the au-
thors, 53 responses from the people known to authors, and
38 responses from the LinkedIn groups. After excluding the
invalid responses we collected 128 responses from the au-
thors, 41 responses from the people known to authors, and
37 responses from the LinkedIn groups finally. Therefore the
response rate of the survey originally targeted (authors) is
128/681 or 19%. Because we do not know how many peo-
ple the authors passed the survey requests to, so we simply
added the 53 responses to the sampling frame and calculated
the response rate for the wider set as (128+41)/(681+53) or
23%. Because we cannot determine the valid population
of the LinkedIn groups, the responses from the LinkedIn
groups were not counted in any calculation of the response
rate. Whichever value is used as the final response rate, it
is well above the usual expected response rate for a survey
(10%).
In the following sections we focus on the profile of the
respondents in Part II of the online survey and their dis-
tributions. Therefore we analyse the composition of the re-
spondents by making use of the demographic questions in
the online survey, and explore the links between the demo-
graphic data and design pattern usage by applying statistical
method.
2.4 The Follow-on Survey
The previous online survey showed that software devel-
opers who are experienced with using design patterns hold
some conflicting opinions about three of the more popular
design patterns: Facade, Singleton and Visitor. Therefore,
we conducted a follow-on survey to investigate “What char-
acteristics of these three patterns cause developers to hold
widely differing views about their use?”
After finishing the follow-on survey, We received 46 usable
responses from a possible total of 188, with nearly 85% of
respondents having six or more years of experience with de-
sign patterns. Of these, 27 also provided comments and
descriptions of experiences about the patterns, which we
categorised. This follow-on survey performed a qualitative
analysis for the three patterns, Facade, Singleton and Vis-
itor. The outcomes displayed that all three patterns could
easily be misused and in each case, the consequences of mis-
use were regarded as being particularly significant.
3. THE ANALYSIS METHOD
Normally descriptive statistics can be used to describe and
analyse the data based on the tables and charts as above [12]
using frequencies, ratio, etc to describe the data patterns
[13]. But for our samples in the online survey we found that
50
Table 1: The Definition of the Data Used in the
Demographic Questions of the Online Survey
Question Continuity Category Method
Q2 Discrete Nominal Quantitative
Q3 Discrete Ordinal Quantitative
Q4 Continuous Interval Quantitative
Q5 Continuous Interval Quantitative
Q6 Discrete Nominal Quantitative
Q7 Discrete Ordinal Quantitative
the descriptive statistics only provided the basic profiles for
the data, and we could not analyse them deeply and sys-
tematically. Therefore inferential statistics were employed
to address this need. This section aims to introduce the sta-
tistical forms and analysis methods employed in analysing
our survey. These provide the basic and solid concepts for
our practical analysis.
3.1 Hypothesis Testing
Hypothesis testing is a statistical method used to evaluate
a hypothesis and make decisions by using sample data, it
is a commonly used method in inferential statistics [8, 9].
Following to the methods for hypothesis testing [3, 9, 10],
the process applied in our survey data was defined as below:
• Firstly, we defined an appropriate scientific hypoth-
esis about the causes we aimed to investigate. The
scientific hypothesis was then used to derive the sta-
tistical hypothesis, including the null hypothesis (H0 )
and alternative hypothesis (HA ). Normally the null
hypothesis represents the case that there is no differ-
ence or relation between the samples. If the statistical
result shows that there is no significant difference, it
means that the null hypothesis is accepted. Otherwise,
the null hypothesis is rejected and the alternative hy-
pothesis is accepted if the statistical result shows there
exists a significant difference.
• Secondly, we analysed whether the samples satisfied
the statistical assumptions or not, and distinguished
between the types of data. The condition used for de-
termining satisfaction can be employed to decide what
kind of testing is more suitable, parametric testing or
non-parametric testing [10].
Besides that, normally the form of the data can be clas-
sified as either discrete data, including nominal and or-
dinal data, or continuous data, including interval and
ratio data [2]. For better analysis of the survey we
classified the types of the data employed in the survey
according to this classification. The clear definition of
the data makes it possible to analyse the survey by
using SPSS appropriately and determine which statis-
tical test should be employed. For those nominal data
elements that have qualitative values we coded them
to give them a quantitative property. Table 1 displays
the definition of the data elements from our survey.
• Thirdly, we chose a suitable statistical test for data
analysis. The factor variable and the independent vari-
able were also defined. The data relying on others are
defined as the independent variable [2]. Conversely,
the data which are relied by other data are defined as
the factor variable.
 • Fourthly, we accepted or rejected the null hypothesis analysed the composition of the respondents by making use
based on the analysis results. If the statistical results of the demographic questions in the online survey question-
were not very significant we accepted the null hypothe- naire (Q2-Q7).
sis (H0 ). Otherwise, if the statistical results considered Analysis of the responses to Question 2 generated Table
to be statistically significant we rejected the null hy- 2, which displays the profile of the respondents based on
pothesis and accepted the alternative hypothesis (HA ). their primary roles. Table 2 shows that the majority of the
respondents were software developers and researchers. The
3.2 Parametric Test numbers for the software teachers and computing students
Parametric test is a statistics method which assumes the were smaller.
data is probability distributed and infer the distribution of
the parameters [7]. In our survey analysis, we employed
the parametric test to infer the distribution of the respon-
dents’ profile. From the definition of parametric test, it has Table 2: Profile of respondents: Primary Roles
to meet the assumptions. If it fails to meet the assump- Category Authors Snowball LinkedIn Total
tions, Non-Parametric test can be used for an alternative, Groups
and also can apply “Post Hoc” analysis method for multiple (#) (%)
compare between the variables. We applied the “Tamhane’s Developer 20 27 34 81 39.3
T2” analysing method of “Post Hoc” analysis in our para- Researcher 70 6 3 79 38.4
metric test. Parametric test contains a series of methods. Teacher 38 1 0 39 18.9
In our analysis we applied One-Way ANOVA and t-Test. Student 0 7 0 7 3.4
Total 128 41 37 206 100.0
• One-Way ANOVA

One-Way ANOVA is a parametric test to compare
means of three or more populations [1]. The applica-
tion of One-Way ANOVA in our survey analysis dealt
with comparing three groups of respondents’ profiles
(Hypothesis 1).
• t-Test
Using the same categories of respondents, Table 3 sum-
maries the distribution of the respondents in terms of their
education. Most of the respondents have a degree higher
than Bachelors. The respondents who have a PhD degree
dominate the majority of all respondents, with 124 respon-
dents forming 60.2% of the entire in the database.

The difference with One-Way ANOVA is that t-Test Table 3: Profile of Respondents: Education
compares the means of two populations [1]. The appli- Highest Authors Snowball LinkedIn Total
cation of t-Test in our survey analysis dealt with com- Degree Groups
paring two groups of respondents’ profiles (Hypothesis (#) (%)
2). Associate 0 1 0 1 0.5
Bachelor’s 1 12 20 33 16.0
3.3 Non-Parametric Test Master’s 13 18 14 45 21.8
Non-Parametric test is a distribution-free technique, it can PhD 114 7 3 124 60.2
be used under the condition of that data do not meet the Other 0 3 0 3 1.5
assumptions of Parametric test [1]. In our analysis we ap- Total 128 41 37 206 100.0
plied the Kruskal Wallis method and the Chi-Square Test
method of Non-Parametric test.
In Questions 4 and 5 of the questionnaire form we re-
• Kruskal Wallis spectively obtained the experiences of the participants about
Object-Oriented development and design patterns applica-
Kruskal Wallis is used to test ordinal data. The appli- tion. We summarised the two profiles for these in Table 4
cation of Kruskal Wallis in our survey analysis dealt and Table 5. Table 4 provides a profile of the experience of
with comparing three groups of respondents’ profiles the different groups of respondents with object-oriented de-
(Hypothesis 1). velopment in general. More than half had over ten years of
experience with OO development (54.9%). Table 5 provides
4. RESULTS a similar profile for experience with OO patterns. Again 58
In this section we present the results obtained from analysing (42.2%) respondents who had 6-10 years experiences of us-
the relations between the different variables of each ques- ing OO patterns dominated the majority of all respondents.
tion which were summarised in Table 1. A mix of descrip- The second group was the group of 11-15 years respondents,
tive statistics and inferential statistics were employed in our 52 respondents (25.2%). From the percentage of each group
analysis. The analysing process was based on demographic of the respondents in these two tables, we can see that most
part of the online survey. In the demographic part we ap- of the respondents have at least 3 years experience of OO
plied the hypothesis method to test the relation between the development and applying patterns. This reflects the expec-
respondents’ profiles and their pattern assessment. tation that most of the respondents were experts in the areas
of OO development and OO design patterns. The probabil-
4.1 Descriptive Analysis ity of the responses being influenced by novices is therefore
Before it is possible to perform any inferential statistical very small. So in the analysis process of the next step we
data analysis, it is necessary to be familiar with the pro- concentrated on those respondents whose experiences of OO
file of the respondents and their distributions. Therefore we development and using patterns are over three years.

51
 LinkedIn group will not be different in terms of their
Table 4: Profile of Respondents: Experience with experience of using design patterns.
OO Development
Length of Authors Snowball LinkedIn Total H0 : p > 0.05. The three groups will not have any
Experi- Groups significant differences.
ence
HA : p 6 0.05. The three groups will differ signifi-
(#) (%)
cantly.
<3 years 1 5 4 10 4.8
3-5 years 2 13 10 25 12.1 Statistical Tests Used: Kruskal Wallis Test, One-
6-10 years 35 10 13 58 28.2 way ANOVA.
11-15 years 34 4 5 43 20.9 Questions Used: Question 5.
>15 years 56 9 5 70 34.0

Table 5: Profile of Respondents: Experience with

OO Patterns Table 7: Statistics Result for Hypothesis 1
Length of Authors Snowball LinkedIn Total Test Statistics(a,b)
Experi- Groups The three groups would not have signifi-
ence cant differences on experience of using De-
(#) (%) sign Patterns
<3 years 3 13 14 30 14.6 Chi-Square 50.575
3-5 years 16 9 12 37 18.0 df 2
6-10 years 65 13 9 87 42.2 Asymp. Sig. 0.000
11-15 years 44 6 2 52 25.2 a. Kruskal Wallis Test
b. Grouping Variable: Group
Table 6 the results of Question 6 (“Have you written or Table 8: Multiple Comparisons Result for Hypoth-
rewritten any patterns?”). For this we expected to find that esis 1
there would be a substantial degree of experience with writ- Multiple Comparisons
ing patterns since the original sampling frame was authors Dependent Variable: How many years of
of pattern papers. Table 6 shows that 122 respondents had experience do you have with working with
pattern authoring experience, the ratio being 59.2%. The design patterns?
distributions in the authors and LinkedIn groups were sim- Tamhane
ilar, only for the snowball group was this more balanced. (I) (J) Mean Differ- Std. Error Sig.
Group Group ence (I-J)
1 2 .879(*) 0.180 0.000
Table 6: Pattern Authoring Experience
3 1.199(*) 0.166 0.000
Pattern Authors Snowball LinkedIn Total
2 1 -.879(*) 0.180 0.000
Author? Groups
3 0.320 0.227 0.414
(#) (%)
3 1 -1.199(*) 0.166 0.000
Yes 79 20 23 122 59.2
2 -0.320 0.227 0.414
No 49 21 14 84 40.8
*. The mean difference is significant at the .05 level.
1 = Author, 2 = Snowball, 3 = LinkedIn
4.2 Inferential Analysis
This part was designed to investigate the links between the Question 5 inquired about the experience year range of
respondents’ profiles and their summary assessment of the using design patterns. We compared the difference on us-
design patterns. Here we brought forward some hypotheses ing design patterns between the three respondents’ groups.
of the relation between the profiles and the pattern assess- Table 7 shows that the result has a significant difference (p
ment, and analysed the results. < 0.05). So the null hypothesis is rejected. The three re-
spondents’ groups do have significantly different experience
4.2.1 The Relation between the Respondents’ Groups with using design patterns. Table 8 also shows the post
As mentioned above, there were three groups of respon- comparison results by using the “Tamhane’s T2” analysing
dents in our database, namely, Authors, Snowball and LinkedIn method between the three respondents’ groups under the
Groups. Comparing the distributions from Table 2 to Ta- condition of One-Way ANOVA test do not meet the assump-
ble 5, shows that the profiles for each of these groups differs tions. From Table 8 we find there is no difference between
quite significantly. The Snowball group was much more like the Snowball group and the LinkedIn group. Therefore, we
the LinkedIn group than the original Author group. There- could compare the author group and the LinkedIn group to
fore it is necessary to check whether we should analyse the check the consistency.
response data from the three respondents’ groups separately. Therefore the results show that the profiles for each of
The experiences of using design patterns were employed as these groups differs significantly, especially between the au-
the variables for checking the hypothesis. thor group and the others. For more precise statistics we
analysed the author group separately in the following hy-
• Hypothesis 1: The respondents from the three groups, pothesis and compared between the Snowball group and the
the original author group, the snowball group, and the LinkedIn group.

52
 90.00% choices in Question 7 against the different design patterns
80.00% experience year range in Question 5. First of all, the num-
70.00%
bers for each assessment choice, such as very useful (VU);
useful (U); not very useful (NVU); not at all useful (NU);
60.00% Little or no experience of using this pattern (LE), were
50.00% Positive counted under the category of the different design patterns
40.00% Negative experience. Because the number of respondents in the differ-
LE ent experience groups were not the same, it is not possible
30.00%
to compare the count values directly. So we counted the
20.00% percentages of each assessment in each experience group.
10.00% Then the assessments of very useful (VU) and useful (U)
were aggregated into the positive assessment, and not very
0.00%
<3 3-5 6-10 11-15 useful (NVU); not at all useful (NU) were aggregated into
the negative assessment.
We compared the assessment percentages in both of the
Figure 2: The Comparison of Using Patterns Expe- author group and the merged group. Figures 2 and 3 com-
riences in the Author Group pared the assessment percentages in each year range for us-
ing patterns experience. From these two figures it is very
90.00%
obvious that the positive assessment percentage of using pat-
80.00% terns increases with greater experience with design patterns,
70.00% and the ‘less or no experience’ assessment percentage of us-
ing patterns decreases with design patterns’ experience. In
60.00%
the negative assessment there is no significant difference be-
50.00% Positive tween the different pattern experiences. To summarise, the
40.00% Negative degree of experience with using design patterns influences
30.00%
LE how people apply the design patterns. Greater experience
means that more patterns are considered useful. Note that
20.00%
we did not consider those respondents whose experience with
10.00% using design patterns was less than 3 years in the following
0.00% analysis.
<3 3-5 6-10 11-15
4.2.3 The Relation between the Primary Roles and
Figure 3: The Comparison of Using Patterns Expe-
the Pattern Usefulness
riences in the Merged Group We also looked at how the ‘votes’ were used by the respon-
dents according to their different roles (researchers, teachers
and developers). To do this, we compared the assessment
percentage in each role. The method is same as used for
• Hypothesis 2: The respondents from the Snowball the comparison of the different design patterns experiences
group and the LinkedIn group will have no difference in the previous section. Figures 4 and 5 show that there are
on the experience of using design patterns. no significant differences on the assessment of design pat-
H0 : p > 0.05. The two groups will not have significant terns between the different roles. Within the four primary
differences. roles just that of the student is slightly different from the
HA : p 6 0.05. The two groups will have significant other roles. The percentage of students in Figure 4 is 0%.
differences. That is there are not any student respondents in the Author
group. Figure 5 reflects the percentage of design patterns
Statistical Test Used: t Test. under ‘little or no experience’ is higher than the other roles,
Questions Used: Question 5. such as developer, researcher and teacher. With the results
We compared the difference on using design patterns be- of the previous section for the pattern experiences, it is clear
tween the two respondents’ groups, the Author group and that the students have less experience than the other roles.
the LinkedIn group. The result has no difference (p > 0.05). Therefore, from the two figures opinions about how to use
So the null hypothesis is accepted. The two respondents’ and assess the design patterns is not influenced by people’s
groups have not a significant different experience on using primary roles.
design patterns. Therefore, the profiles for each of these
groups have no a significant difference, and can be consid-
ered as being drawn from the same population. Therefore
5. THREATS TO VALIDITY
both of these two groups were merged into a single group. As with any empirical study, it is necessary to consider
We analyse the data from the author group and the merged the possible effects of any factors that could have biassed the
group separately in the following hypothesis. outcomes from the study. Here we can identify two threats
to validity for the online survey:
4.2.2 The Relation between Patterns Experience and
the Usefulness Profile • the effectiveness of the responses (internal)
In order to investigate the relations between the experi-
ences of using design patterns, we compared the profile of • the completeness of the responses (external)

53
 90.00% 5.2 External Validity
80.00%
During the process of collecting responses from the online
70.00% survey, we found there were only few responses to the open
60.00% questions. Most of the answers to the open questions were
50.00% also very short and simple. Only a few of the respondents
Positive
40.00% provided any details of their experiences about using pat-
Negative terns. For this study we focused on the close questions in
30.00%
20.00%
LE the demographic part. Therefore, our results would not be
10.00%
influenced by this problem.
To the length of the survey form, we checked the responses
0.00%
Commercial Software Software Student of
including the ‘excluded’ responses. Some respondents only
software researcher teacher computing answered the “Demographic” part before exiting. But most
developer respondents completed it, they were not influenced by the
form length. So overall, the rate of completion suggests that
Figure 4: The Comparison of Different Roles in the we got the length of the survey about right.
Author Group
6. SUMMARY
90.00%
Since the publication of the milestone GoF textbook, some
80.00% of the 23 design patterns in this book have been widely em-
70.00% ployed in the OO development and maintenance process.
60.00% The concept of a design pattern indicates that design pat-
50.00% terns are a reuse means of documentation for successful so-
Positive lutions, which have been abstracted from experts’ successful
40.00%
30.00%
Negative experiences. Theoretically design patterns can make our de-
LE signs easier, and also help to create systems with properties
20.00%
such as flexibility and maintainability. The use of a design
10.00%
pattern can make a system more understandable for both
0.00% designers and maintainers.
Commercial Software Software Student of
software researcher teacher computing
But actually design patterns are not a universal solution
developer for more effective software development, and the application
of design patterns is not so easy and perfect as the expec-
tation. Our study has investigated the links between the
Figure 5: The Comparison of Different Roles in the
users’ background and the effectiveness of design patterns
Merged Group
by analysing quantitative data. This study has observed
and analysed the functions of design patterns are influenced
by users’ experiences rather than users’ roles.

5.1 Internal Validity 7. ACKNOWLEDGMENTS

To collect the responses in the online survey, we used the
online database in the professional survey website
“www.surveymonkey.com”. The responses from the partici-
pants were stored in its database automatically. Before col-
lecting these responses we sent our requests to the authors
of the papers in our mapping study. But sometimes one au-
thor’s name can appear in more than one paper. To avoid
missing any authors, we sent the requests to the authors’
different addresses. It might have led to some authors re-
ceiving the requests more than one time. Hence they could
answer the questionnaire twice, creating the duplicated an-
swers. Due to the design function issues of the SurveyMon-
key website, we cannot prevent this happening or identify
the duplications.
But the website provides the functions to record the iden-
tities of the respondents, such as IP address, response’ time.
These can be used to distinguish the duplications. And also
we designed the first question in the survey form to ask the
respondents provide their names and email addresses for dis-
tinguishing. We checked every response and compared with
our author’s list to check for any duplications. From the
checking result we did not find any duplications of the re-
sponses. So through these thorough checking process we are
quite confident about the effectiveness of the responses.
This work was supported by an award from Anhui Provin-
cial Natural Science Foundation (1408085QF108). We would
like to acknowledge the help received from Professor David
Budgen, and also acknowledge all those respondents who
participated in our survey.
8. REFERENCES
[1] R. Bernstein and S. Bernstein. Schaum’s Outline of
Elements of Statistics II: Inferential Statistics.
McGraw-Hill, 1999.
[2] S. Bernstein and R. Bernstein. Schaum’s Outline of
Elements of Statistics I: Descriptive Statistics and
Probability. McGraw-Hill, 1st edition, 1998.
[3] G. Brue. Six Sigma for Managers: 24 Lessons to
Understand and Apply Six Sigma Principles in Any
Organization (The McGraw-Hill Professional
Education Series). McGraw-Hill, 1st edition, 2005.
[4] T. Dybå, B. Kitchenham, and M. Jørgensen.
Evidence-based software engineering for practitioners.
IEEE Software, 22(1):58–65, 2005.
[5] W. Frakes and K. Kang. Software reuse research:
status and future. Software Engineering, IEEE
Transactions on, 31(7):529–536, July 2005.

54
 [6] E. Gamma, R. Helm, R. Johnson, and J. Vlissides. APPENDIX
Design Patterns: Elements of Reusable
Object-Oriented Software. Addison-Wesley, 1995. A. THE ONLINE SURVEY (PART II)
[7] S. Geisser and W. Johnson. Modes of Parametric
Statistical Inference (Wiley Series in Probability and 1. Please note you will not be identified in any report that
Statistics). Wiley-Interscience, 1st edition, 2006. is produced using the information you have provided in
this questionnaire.
[8] F. Gravetter and L. Wallnau. Essentials of Statistics
Your name: Email address:
for the Behavioral Science. Wadsworth Publishing, 6th
2. Which of the following best describes your primary role
edition, 2007.
during software development?
[9] D. Howell. Fundamental Statistics for the Behavioral
Sciences (with CD-ROM). Duxbury Resource Center, (a) Commercial software developer
4th edition, 1998. (b) Software researcher
[10] R. Kirk. Experimental Design: Procedures for (c) Software teacher
Behavioral Sciences (Psychology). Wadsworth
Publishing, third edition, 1994. (d) Student of computing
[11] B. A. Kitchenham, D. Budgen, and O. P. Brereton. 3. Highest degree you have earned?
Using mapping studies as the basis for further research
(a) Associate degree
- a participant-observer case study. Information and
Software Technology, 53(6):638 – 651, 2011. (b) Bachelor’s degree
[12] P. S. Mann. Introductory Statistics. Wiley, 6 edition, (c) Masters
2006. (d) Ph.D. or equivalent
[13] S. Ross. Introduction to Probability and Statistics for (e) Other
Engineers and Scientists. Academic Press, 2nd edition,
1999. 4. How many years of experience do you have with Object-
[14] H. van Vliet. Software Engineering: Principles and Oriented development?
Practice. Wiley, 2 edition, 2000. (a) Less than 3 years
[15] M. Vokáč, W. F. Tichy, D. I. K. Sjøberg, E. Arisolm, (b) 3 to 5 years
and M. Aldrin. A controlled experiment comparing
the maintainability of programs designed with and (c) 6 to 10 years
without design patterns—a replication in a real (d) 11 to 15 years
programming environment. Empirical Software (e) Over 15 years
Engineering, 9:149–195, 2004.
5. How many years of experience do you have with work-
[16] P. Wendorff. Assessment of design patterns during
ing with design patterns?
software reengineering: Lessons learned from a large
commercial project. In Proceedings of 5th European (a) Less than 3 years
Conference on Software Maintenance and (b) 3 to 5 years
Reengineering (CSMR’01), pages 77–84. IEEE
(c) 6 to 10 years
Computer Society Press, 2001.
[17] C. Zhang and D. Budgen. What do we know about (d) 11 to 15 years
the effectiveness of software design patterns? IEEE 6. Have you written any patterns (or rewritten any exist-
Transactions on Software Engineering, ing patterns)?
38(5):1212–1231, September/October 2012.
(a) Yes
[18] C. Zhang and D. Budgen. A survey of experienced
user perceptions about software design patterns. (b) No
Information and Software Technology, 55(5):822 – 835, 7. In this section you are asked to provide us with your
2013. assessment of the usefulness of each pattern in the book
[19] C. Zhang, D. Budgen, and S. Drummond. Using a “Design Patterns: Elements of Reusable Object-Oriented
follow-on survey to investigate why use of the visitor, Software” by Gamma et al., based upon your experi-
singleton amp; facade patterns is controversial. In ences with using that pattern.
Empirical Software Engineering and Measurement (For each pattern identify as: Very Useful ; Useful ; Not
(ESEM), 2012 ACM-IEEE International Symposium Very Useful ; Not at all Useful ; Little or no Experience
on, pages 79–88, Sept 2012. of using this pattern)

55