Professional Documents
Culture Documents
Software Testing
C o n fe re n c e
EuroSTAR
Software Testing
Community
Abstract
This eBook contains a number of chapters from Albert Witteveens book entitled Performance testing, a practical guide and approach. This book was written to give people a start in performance testing. It should help people to understand the basics of how to do performance testing and hopefully how to do a good job. Another target audience is anyone dealing with performance demands or problems. How can you tell if the testers are going to deliver a result that gives tangible insight in the performance? The book aims to give you some handle with which you can judge the approach and see if you get your moneys worth.
since some words we use often are defined in many other ways. Which definition is right is not the point, but is important that we define here what definition we use.
2.1 Introduction
What should you learn first? To navigate or to sail? If you learned how to navigate perfectly and then discover that you dont last on a sailing boat for longer than 10 minutes, those navigation skills are a bit useless. Thats why in this book we first get into the actual performance testing itself before discussing planning, modeling and so-forth. The basics will teach you how to simulate many users with tools and what you need to know before you can judge if you created a good test script.
When we are using the term Load we are talking about workload put on a system. Typically this is the load provided by the multiple users that we expect when a system is in production. When you a have a web application of which you expect that under normal circumstances it is used by 200 simultaneous users, those users provide the load. What we test for is the performance of software in terms of response times, throughput etc. at the given load. So when we are discussing load testing, we discuss testing on the performance of the application in terms of response times, throughput etc. when we apply a load that is the same as what we expect during production.
PAGE
way beyond expected load and apply load until the system cant handle this anymore. This is done for a few reasons: we want to know when a system breaks. But most of all it is to determine what happens at that moment. Is there a full system outage? Do we lose transactions? Are there data synchronization issues? Additionally we can monitor to see which part of the system and resources is the bottleneck. That will give a clear indication on what to improve or expand if we have to. Stress testing is therefore about finding out when it breaks and what happens when it breaks.
PAGE
to use your browser and perform the actions as the real user would do. All the traffic between the browser and the server is recorded. Then the tool will turn the recorded traffic into a test script that it can rerun to playback the exact same request as were performed during the recording.
tells you that the script is passed. We however still dont know if it really passes. We know that we send the same requests, but we dont if the answers that we get are the correct ones. Take the following example of a web application. You have created a test script that tests the login procedure of the application. The script sends the requests. But when the login is requested the server, due to a parametrization omission doesnt answer with a page that youre logged in, but it answers instead: login denied. On the server side this means it probably does a lot less than when the login is successful, giving you different results than what you should get. Therefore in any test script you need to build content checks. For instance if the screen of a successful login shows the text: welcome user Username1, build in a check that tests that the response of the server contains the words welcome user. Also it can be necessary to check in systems that should be updated by your test that the updates are done.
2.3.4 Parametrization
Now what would happen if you recorded the creation of a new user account? During recording you would for instance create the user: testuser1. When you rerun the exact same request, you would try to create the user testuser1 one again. This time however the server would reply something like user already exists. All the coming requests would subsequently receive error messages, effectively making sure that the load on the server is not what you wanted to generate. And this is for the simple test of making a new user. For changing data, for most tests, data needs to be used that is available and does not conflict with certain business rules. There is more. Much of the requests that a browser makes are based on returning information the browser received from that server such as cookies and session-ids. Many applications even have their own information going back and forth the keep track on requests. For this a script nearly always needs parametrization before it can be rerun, especially before it can be rerun under load. Parametrization is one of the most difficult and time consuming part of load and stress testing.
PAGE
issues that indeed exist is much easier with just a few virtual users. It also prevents you from burning your test data.
PAGE
Collect test data for the parametrized items that need to be filled with data. Build in checks to see if the server gave the correct responds. Verify the parametrization by running a test with a few virtual users to detect parameter property errors. There is much more to tell about creating test scripts. But these are the basic steps.
bit too much just to be certain. In all other cases, keep it limited to what you would look at before detecting an issue in the first place. When you do detect an issue that needs further investigation, rerun the test with added monitors.
test and to whom to report. All these items will now be explained in more detail.
2.5 Reporting
When everything worked out and we finished a test, it needs to be reported. As always with reporting, what you report depends greatly on who you report to. You may have to report to an accepting party, sometimes a software development project needs a report to show it has met its targets, often you are involved in testing to reproduce performance issues on production and verify that a solution will provide improvements. All of these different reasons for testing require a different focus in your report. Obviously there are also overlaps and similarities, but reporting is important and is equally important to report the right thing. Also reporting issues needs to be done right, although in many ways similar to reporting regular issues or findings, performance testing puts some strong demands on what you include in the issue report. We will explain much more in the chapter on reporting.
PAGE
2.6 Conclusion
The basics were explained here to provide an overview of the activities. Performance testing can mean load testing, stress testing or endurance testing. To perform a test, a tester needs to perform quite a few steps. Recording is just one step, the scripts needs to be customized to handle different data, server responses and check for real results. To provide insightful reports we need proper monitoring. This needs to be set up. When a test is done, the results need to be reported and reporting on performance depends on the purpose of the
5. Get in Line
5.1 Introduction
Computers behave in a very simple way. They stand still or they run at full speed. Yet we often consider the systems not to be running at full capacity. The systems are quite often waiting for something. To understand performance and to describe the performance affecting items queuing theory is a good approach. At minimum a performance tester and or engineer should understand queuing in our systems. Even better is if the tester applies queuing models and monitors all components. That way the behavior gets documented much better and allows us to optimize performance. This chapter explains the basics, but we encourage you to learn more about queuing theory. The book called Analyzing Computer Systems Performance with Perl::PDQ by Neil J. Gunther does this very well.
The customer that is being served is spending time on unloading the cart, paying the cashier and packing up, the service time. The other customers in the line are waiting. The total time it takes you from arrival to departure is called residence time. We all are familiar with queues in supermarket
PAGE
waiting line. You have customers arriving and Algorithm 5.1 Residence time served = total waiting The customer that is being is spending t time + service time. packing up, the service time. The other custome from arrival to departure is called residence tim Algorithm 5.1Residence time = total waiting 5.3.1 Small shop with one checkout lane
5.3.2Multiple lanes
5.3.2 Multiple lanes
Any larger sized supermarket there will be mor Any larger sized supermarket there will be more You do this based on the number of other custo checkout lines. When you arrive you pick your cannot influence your service time [B] [B]to ke lane. You do this based on the number of other very notably slower or faster than the others, b customers in the lines and how full their carts depends on number of customers of you are. You cannot influence your service ahead time [B] This out [B] tomodel keep itturns simple welike will this: ignore that some
cashiers are very notably slower or faster than the others, but you can influence your waiting time. And this depends on number of customers ahead of you and their service time.
Any larger sized supermarket there will be more checkout lines. When you arrive you pick You do this based on the number of other customers in the lines and how full their carts are cannot influence service time [B] [B]to keep it simple we will ignore that some cashi Performance testing, a practicalyour guide and approach very notably slower or faster than the others, but you can influence your waiting time. And depends on number of customers ahead of you and their service time. Other processes will have to wait. Anyone This model turns out like this: This model turns out like this: that encoded a video file on older PCs will
Figure 5.2 Multiple checkout lanes Figure 5.2 Multiple checkout lanes
model of 5.1 As soon as a processor gets a task 5.3.3 One lane, multiple cashiers Other have to wait. Anyone that 5.4.2 processes Web server with multiple And in the larger super markets the cheese sections often usewill a system where you get a num soon as the encoding starts, And until in theyour larger super markets the good cheese processes wait number is up. The thingeffect. is thatAs you do not need to assesprocess for yourself w sections often use a system where you get a operating systems dois have to all will be the fastest for you. Rather than having three queues there one some queuemethods that is handle number and wait until your number is up. The If we look at a higher level at a system with multiple people. The model would look likeservice this: time of the processes is small you may good thing is that you do not need to asses for multiple web servers serving to clients, the residence time ends uppages being small as well
yourself which line will be the fastest for you. Rather than having three queues there is one queue that is handled by multiple people. The model would look like this:
5.3.3One lane, multiple cashiers On a computer system with one (single core)
understand the effect. As soon as the encoding process starts, anything else will hardly get The queues in the supermarket can be compare through. Although operating systems do have few common examples with the models from th some methods to allow for processes to get some CPU time. Now if the service time of the processes is small you may hardly notice. The wait time gets to be so small that the residence time ends up being small as well.
5.4Queues in a server e
the model will resemble the supermarket with multiple checkout lanes. If there are multiple web servers, some load balancing service is in place that will receive the request and give it to one of the servers. Like with the customer If we look at a higher level at a system with mu deciding on the number of preceding customers will resemble the supermarket with multiple ch and the content of their shopping cart, the load load balancing service is in place that will rec balancing service will also make a decision to deciding on that the number with the customer which server to hand it and basically means, of pr cart, the loadit balancing in what queue ends up. service will also make
PAGE
Figure 5.4Three webservers Figure 5.4 Three webservers Interesting to see above is that in this very simp locations can occur that cause w Interestingwhere to see queues above is that in this very
simple model you can already see that there are multiple locations where queues can occur that cause wait time.
And the last model can be seen as well. If there 5.4.3sure One queue, multiple make that such requests are dealt with by m processes software (such as Apache and Microsofts IIS) worker processes available the queue may be s And the last model can be seen as well. If there arrive. It should be noted though that if a single is one service that can handle multiple requests system level will be queuing atwith for instan and make sure there that such requests are dealt
by multiple worker processes, such as most web serving software (such as Apache and Microsofts IIS), you will something similar. If the software has many worker processes available the queue may be small, but there is a central queue where the requests arrive. It should be noted though that if a single server is setup to have many worker processes, on the system level there will be queuing at for instance the CPUs or for memory.
However as the waiting line is larger, the residence time for a request becomes large. Even though the processor did, when it got around to it, calculates just as fast when it was less busy. The service time stays the same. As a result, when the CPU is starting to get more requests, the chance that requests of other different processes are there increases resulting in that processes will find queues at the CPU. There will however still be moments that the CPU is idle until the load is so high that it is continuously occupied. At that time the waiting time is usually large and the overall residence time at the CPU is large as well reducing response times. Overall it feels as if the CPU is doing its calculations slower, whilst really we just have to wait longer.
Like we said before, a computer component basically runs at full speed or does nothing. If Server systems usually have more than one CPU. youre CPU monitoring tool tells you the CPU And modern CPUs often have multiple cores. is running at 25% it does not mean that it runs Multiple cores effectively work as multiple 25% of its possible speed. It means that for 25% CPUs. I once had a colleague report that a certain process was taking almost 100% CPU time. The of the measured time it was running at 100% administrator dryly responded: good then were using the CPU efficiently. He was right, yet for good and for 75% it was doing nothing, or as this is the CPU or at the CPU core level, the reasons if a server reports 100% CPU time, we often At view this as level, a problem. usually idle. a computer component basicallyqueuing center does not nothing. change. If We just have Like we called: said before, runs at full speed or does youre multiple centers. Like in the supermarket CPU monitoring tool tells you the CPU is running at 25% it does not mean that it runs 25% of itswhere If we draw a queuing center for a CPU it looks open checkout lanes and do not make possible speed. It means that for 25% of the measuredmore time it was running at 100% for 75% it the was doing nothing, or as this is usually called: idle. service time itself shorter. But the wait time is like this: If we draw a queuing center for a CPU it looks like this: shorter. In a system where we add equally fast CPUs, the residence time at the CPU will decrease if there are enough processes running.
PAGE
There is a catch though. Not every process Figure 5.5 Queueing center a single Figure 5.5 Queueing center for for a single core core CPU CPU will divide its workload over the extra CPUs. For most processes this waiting line hardly has any impact. The you service time most requests is so Suppose have a for fully loaded shopping small that the waitingthis linewaiting is hardly noticeable. Even CPU intensive processes do not send one large For most processes line hardly has cart in the supermarket and there are six open request, but many small ones. As a result, even if the CPU is busy, processes can get their requests in. any impact. The service time for most requests is checkout lanes, your residence time would not However as the larger,noticeable. the residence time for a request becomes large. Even though so small that thewaiting waitingline line is is hardly be shorter if you were the only one in the shop. the processor did, when it got around to it, calculates just as fast when it was less busy. The service Even CPU intensive processes do not send one In that case six open lanes will be just as fast as time stays the same. large request, but many small ones. As a result, one lane. Unless you find some way to divide As a result, the CPU is starting to get even if the when CPU is busy, processes can getmore their requests, the chance that requests of other different your cart in six smaller carts and processes are there increases resulting in that processes will find queues at the CPU. There will requests in. however still be moments that the CPU is idle until the load is so high that it is continuously occupied. At that time the waiting time is usually large and the overall residence time at the CPU is large as
10
much slower if you have many cores, but you checkout in six lanes. In computer systems this will have little impact on the other processes. is the same. A process will be handled by one processor or core unless the application was build in such a way that the workload is divided 5.6 Queuing on higher levels over multiple processes or threads. For most processes you will not end up in the situation The principals described work at higher is div that one CPU is core very busy whereas the others was processor or unless the application build in such a way here that the workload level in the same way. You can look at a system are idle. But on the process level the queuing multiple processes or threads. For most processes you will not end up in the situation that on infrastructure level, describing the servers, center for single threaded applications look like very busy whereas the others are idle. But on the process level the queuing center for sing network and thea same principal applies. If is divi this: processor or look core like unless the application was build in such way that the workload applications this: you for instance have a end couple multiple processes or threads. For most processes you will not up of inweb the servers, situation that database servers and a load balancer, if you very busy whereas the others are idle. But on the process level the queuing center for sing apply load you should be able to detect queuing applications look like this: somewhere in the system. Even though possibly none of the systems are stressed in any way, if Figure 5.6 Single thread or process you analyze the processes as they go through Figure 5.6 Single thread or process If a process ends up in a lane it will be in that and handled by thatto core. the queue entire system it mustfully be possible find Multithreaded programs get this center: out the residence times and service times. In If a process ends up in a lane it will be in Figure 5.6 Single thread or process that queue and handled fully by that core. fact there is queuing occurring at least in one If a process ends up in athis lane it will be in that queue and handled fully by If that core. Multithreaded programs get center: location even if it is hard to detect. there was Multithreaded programs get this center: no queuing anywhere, each process would have finished in the blink of an eye.
Figure 5.7Multithreaded yousituations find a queuing center at a higher in smalle The process will end up using both CPUs. If For where a server haslevel, many many it makes sense go down a level at inte there is not much difference. There is however acases big difference if to we have large CPU Figure Multithreaded the queue to find In the example Figure 5.75.7 Multithreaded processes. Multithreaded processes will use as much of allout of more. the CPUs as they we can. You used we could or instance see queuing in the smalle The process will end up using both CPUs. For situations where a server has many compare this with unloading your cart divided over all the checkout lanes. If you are the o The process will enddifference. up using both CPUs. For web a servers. That would we would have there is not much There is however big time difference ifmean we smallest. have large CPU inte shop thatwhere is the a most effective andsmaller overall residence will be the However i situations server has many toas look into the monitoring of the web servers. processes. Multithreaded processes will use much of all of the CPUs as they can. You a supermarket other customers willThere not likeThere this. And sometimes in computer systems we processes, there is not much difference. we all would have to find queuing in for compare this with unloading your cart divided over the checkout lanes. If you are the is however a big difference if we have large CPU this either. When we for instance have a maintenance batch job running on the tell server thato instance the CPU or memory. This would us shop thatprocesses. is the most effective and overall time will While be the smallest. However if intensive Multithreaded processes intensive this will lead to a degradation ofresidence the other services. all CPUs ch much about where the most making time is used and will use as much of all ofcustomers the CPUs aswill they not a supermarket other like this. And sometimes in computer systems d shorten the time of the batch job, thecan. degradation of performance may be a problem. Ifwe you possibly where it can be improved. You can compare this with unloading your cart this either. When we for instance have a maintenance batch processes, job running on the server the batch job is not multithreaded or divided over multiple you will make that surei divided over all the checkout lanes. If you are intensive this will lead to degradation of theslower other services. While making all CPUs chi only one of a that CPU. Itawill run much if you have many cores, but you will the only oncore the shop is the mostthen effective shorten the time oftime the batch the degradation performance may be a problem. If you and overall residence will be job, the smallest. 5.7 Isof there always a queue? impact on the other processes. the batch not multithreaded divided over multiple processes, you will make sure However ifjob youis are in a supermarketor other only one core oflike a CPU. It will then run if you have manythat cores, you will customers will not this. And sometimes in much Weslower once tested a batch process was but taking computer systems we dont like this either. When impact on the other processes. a long time to complete. We started to run this
we for instance have a maintenance batch job running on the server that is CPU intensive this will lead to a degradation of the other services. While making all CPUs chip in will shorten the time of the batch job, the degradation of performance may be a problem. If you make sure the batch job is not multithreaded or divided over multiple processes, you will make sure it will use only one core of a CPU. It will then run process in parallel on one server and we noticed that if we ran three processes instead of one it did indeed finish nearly three times as fast. This is not weird on a multi core CPU or indeed a multi CPU system, but we were not seeing high CPU utilization. Now if there had been another queuing center than the CPU, we suspected not to see almost the same increase in performance
PAGE
as the amount of processes we ran. So we did a deeper monitoring of the CPU. What we then noticed is that the process had many locks. The program had internally a loop with a sleep in it. As this program was called through the batch process in sequence, although it required a small service time, the sleep time meant that it waited until completing. As a result, we got a throttled process which took a long time to complete but was not continuously stressing the CPU. The solution to increasing the performance was laying in getting the sleep time out of the program. What is important to remember is that you should always find a queuing center. If there doesnt seem to be one, continue investigating until you do. Once found, you may find an option to optimize the software. At that moment it is running at full speed. So if you can, especially reporting to technical teams, report where the queuing center is or where you found out that the software is not performing at its maximum performance. You may enable the team to greatly improve the performance. Of course in the case of the batch process, the throttling may be the desired behavior. Considering that it was doing its job without putting much load on the CPU, it also didnt hinder other functionality. If you do use maximum performance, the batch process would end earlier but would have led to other users getting a much slower system.
be much smaller if it has to wait to be serviced. Waiting is done in queues. When load and stress testing using at the very least the concept of queuing helps us find where the most time is spent. Of reporting to technical team, it makes sense to make a model and show where the queuing takes place. If you cant find any component that has its resources fully utilized, you should investigate further until the cause is found, especially if you do see decreasing performance with added load.
11
PAGE
5.8 Conclusion
Similar to queues in day to day life, computer systems behave in a way that mimics these queues. Computer components dont perform at a certain percentage of their capacity. They are performing at a certain percentage of time. The total time it takes a process to complete is called residence time, but the service time may
10.1 Introduction
If you are a typical (test) manager you probably skipped the rest of the book and went straight for this chapter. That is OK; this chapter is here to outline what you should look for in a performance team. We want to give you some handles on how to see if you are getting good work from your team without getting in their way.
4. Experienced performance testers have lessons learned or interesting anecdotes. Eager and enthusiastic testers will often talk about them even when not being asked.
10.2 Intake
If you are involved in the selection of the team, the vision and technical experience of the team are very important. There are some questions you can ask to help you evaluate this even if you are not an expert yourself: 1. Ask for their vision. A team or tester that will repeat they had the expensive training with the expensive tool and years of experience of using it shows no vision. They only show that they can show up for class and work. 2. If they show: an approach, mention how they usually derive test cases, their approach on monitoring, report, requirements from other teams and expertise desired, etc. they show better understanding of performance testing and not just their tool of choice. 3. Another interesting question is what load testing tool they prefer. There is not best tool for every situation. It does make sense that they have a preference for a tool with which they have a lot of experience, but for some application specialized tools may exist. Sometimes a competitor supports the protocol you need much better than the one you are familiar with. Hire a
12
PAGE
testing. Also remember that specialist may be called in to help such as DBAs. Check if they have taken this into account arranged for this support.
the end that performance is not good enough when even the first test already showed issues. 4. Checks: The results can be very deceiving; the tools may report excellent behavior, only to find out later that nothing really happened. See if they checked that the tests really performed the full business process they set out to. Do not accept the line: just wait until the report.
13
PAGE
3. Modesty: Performance testing cannot give exact results for how the application will behave in production. The report should make the limitations clear. A good indicator of a serious team is that there is modesty in conclusions and statements.
Biography
14
Albert Witteveen has been working both as an operations manager and a professional tester for nearly two decades now. The combination of setting up complex server environments and professional testing almost automatically led to a specialisation in load and stress testing. He wrote a practical guide to load and stress testing which is available at Amazon. Lately his focus has been on performance of applications hosted with Cloud technology which led to insight on the changing needs in performance testing for Cloud based applications.
PAGE
Become a fan of EuroSTAR on Facebook Join our LinkedIn Group Add us to your circles Contribute to the Blog Check out our free Webinar Archive Download our latest eBook
w w w. e u r o s t a r c o n f e r e n c e s . c o m