Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!
Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!
Do you want to transition¬†from waterfall to agile? Like so many other companies, you may also seek to replace waterfall with agile in a quest to shorten the time-to-market and deliver high-quality software faster, more frequently and at a lower cost. The road to agile, however, can be a rocky one.¬†That‚Äôs why we have put together a few lessons¬†and tips that will help you in succeeding in¬†agile software development.#¬†1: Embrace the¬†change in mindset and culture.
More than anything, agile requires a change in mindset and culture throughout the entire company. The key to success is that everyone involved knows what to expect, is patient and, most importantly, embraces change. Agile is about adjusting. It‚Äôs about continuously assessing if requirements and goals are still valid. This new learning culture is¬†a big challenge.#¬†2: Make the transition to agile a joint company process.
You won‚Äôt be able to reap the rewards of agile without engaging both the management and the team. If not everyone involved is committed and participates effectively, you‚Äôll unavoidably run into problems. That’s why you have to make sure that the transition to agile is a joint activity. Communication, effective contribution by everyone involved and continuous planning is required to get all interest groups on board, and avoid tension between how the teams operate and how management runs the company.# 3: Roles and responsibilities change throughout the company.
Undoubtedly, the shift to agile affects roles and responsibilities within a company. A¬†top-down command-and-control culture should be replaced with horizontal conversations. Management helps to remove impediments, encourages teams and ensures the business alignment of projects. It has become a facilitator and is in charge of creating an environment in which teams can grow. The team itself is a group of experts, who know best themselves what should be done by who and how. They are self-organizing and self-reliant.#¬†4: Encourage transparency and exchange between teams.
In an agile environment, small teams that communicate and work together efficiently are important. Consequently, transparency is key to success. Be open about failures, discuss impediments and work on a solution together. This will create unity in the team and build towards a better product. At the same time, you need to encourage exchange between those highly productive units so as to avoid isolation of individual teams.#¬†5: Agile is not a linear process.
In waterfall projects, teams were used to working in a sequential manner: requirements, design, implementation, then testing and maintenance. Everything was strictly planned and scheduled. In contrast, software is developed and delivered in small chunks in agile. To ensure it meets actual market needs, the software is continuously adjusted according to changing requirements as well as to feedback from customers and stakeholders. Business managers, developers and testers have to work together closely and stay in sync to evaluate the situation and determine the right next steps.#¬†6: The team is responsible for quality.
The team¬†needs to collaborate right from the beginning: when it comes to defining requirements, the scope of projects and what is needed to assure the quality of the software. As software quality is a continuous process that requires commitment from everyone involved, the whole team is in charge of testing.#¬†7: Test early and often.
As software is developed incrementally, code is constantly changing. Every time you check in new code, you have to ensure that it meets its requirements and does not break existing functionality. The frequency of code commitment requires fast feedback from tests that can only be achieved with automation. When you integrate tests into your continuous integration process, each code change is automatically tested. The goal is a near real-time feedback, which helps you in reacting to potential errors as soon as possible.#¬†8: Automation is a necessity, not a necessary evil in agile software development.
Agile software development will not succeed without test automation. Manual testing is slow, labor intensive, inconsistent and prone to errors. If you want to reduce the time-to market and release small increments of software in frequent iterations, agile test automation has to be an integral part of the project from the beginning. Well-designed automated tests are faster, provide continuous feedback on software quality, increase risk-coverage and will give you confidence in your application.#¬†9: Early feedback from automated tests is important.
The earlier you find bugs in your software, the easier and cheaper they are to fix. The key is to understand what to automate and when, in order to get quality feedback from your automated tests. If you want to get early feedback on the quality of your code, you need to shift testing left and perform more automated testing on a unit and mid-tier level. The iterative feedback increases transparency and enables you to react to potential problems faster, decreasing the risk of release delays and software failure.# 10: Test automation tools have to enable team collaboration.
In agile software development, it‚Äôs all about continuous testing and team collaboration¬†on agile test automation projects. When you choose a test automation tool for your team, make sure that it fosters this collaborative dynamic in cross-functional teams, enables you to start with test automation early in the development cycle, supports¬†reusability of tests and encourages communication and feedback. Always keep in mind that after all, your team is one of your greatest assets.
We‚Äôre excited to announce that Ranorex 6.2 is ready for you to download! In this release, we‚Äôve focused on providing you with an advanced technology support for desktop and mobile applications as well as on making it easier for teams with mixed skills to collaborate on test automation projects.Advanced Technology Support
We‚Äôve made the support of innovative technologies a priority and now enable testing of Chromium-based frameworks in desktop apps and WKWebView objects in iOS apps.Chromium Embedded Framework (CEF) support
CEF is one of the most frequently used frameworks for embedding web content in desktop applications. Next to Google‚Äôs native CEF implementation, we now also support testing of these Chromium-based frameworks:
The WKWebView class enables you to embed interactive web content in your iOS mobile applications. Using Ranorex 6.2, you can now test your embedded WKWebView objects.User Code Library
Sometimes, you need to add further functionality to your tests with user code. We understand that in a team with mixed skills, not everyone wants to code. That‚Äôs why we‚Äôve introduced the user code library with Ranorex 6.2. It gives you direct access to the user code methods your colleagues have previously created in a test automation solution, so you can add additional functionality to your tests without writing a single line of code.A quick overview of the workflow:
If you‚Äôre a developer or tester with programming skills, you can start by filling the user code library with user code methods. You can logically organize methods that relate to a specific workflow or application area. A tip: Add a description to each method. This makes it easier for your colleagues to find the right user code method¬†.
As a tester in your team, you can directly access the library from¬†the action table and select a method from there to use it as an action. This way, you can add further functionality to your tests without having to dip into code!
We hope you have as much fun using this update as we‚Äôve had creating it!
Our Ranorex-NeoLoad webinar was a great success and we were delighted to see how many of you are interested in the Ranorex-NeoLoad integration. In this webinar, we’ve showed you in detail why it makes sense to combine automated functional and load tests, how you can set up the Ranorex-NeoLoad integration, and what you can do with it.¬†For those of you who’ve missed the webinar, we’ve recorded it and you can view it anytime here: Ranorex-NeoLoad webinar.
Thank you again for all the questions you’ve asked during this webinar. As there wasn’t enough time to answer all questions during the webinar, we’ve taken this opportunity to answer a few of them, that we believe are interesting to all of you, here:Q1: Can I test my desktop app, which is connected to a server, with NeoLoad and Ranorex?
Yes, that‚Äôs possible. If the communication is based on the HTTP protocol, the NeoLoad recorder captures the traffic utilizing the HTTP transport layer. If the communication is not based on HTTP, you can use custom actions. Ranorex is then able to determine performance values of the desktop app and transmit these values back to NeoLoad.Q2: I have already created Ranorex web tests. Can I transform functional test sequences from these tests into NeoLoad load tests?
Yes. Simply start a NeoLoad recording manually, and then start the Ranorex test. NeoLoad will record all steps that are being carried out by Ranorex, so that the Ranorex test sequence is available in NeoLoad. You can now put a load on your server with NeoLoad and see if your functional test sequence is still successful when under stress.Q3: Where should I¬†install Ranorex and NeoLoad? Can everything run on the same machine or should we use separate systems?
In theory, you can install the NeoLoad controller on the same machine as Ranorex is installed on, but that’s¬†not recommended. Load generation can cause a high CPU usage and the controller needs a lot of memory. That‚Äôs why it makes sense to install both the NeoLoad controller as well as the load generator on a different machine than Ranorex is installed on. The modules, which are included in the Ranorex-NeoLoad ¬†NuGet package, enable communication between Ranorex and Neoload regardless of where Ranorex and Neoload are installed. However, if NeoLoad triggers a Ranorex test, a NeoLoad load generator must be installed on the same machine as the Ranorex test runs on.Q4: Are virtual users in NeoLoad carrying out the same actions as the Ranorex automated test?
This completely depends on your setup. The ‚Äútest sequences‚ÄĚ are not linked together in any way. You‚Äôre free to set up the same sequence or use a different one.Q5: When should I start my Ranorex test with NeoLoad, and when does it make sense to start my NeoLoad test with Ranorex?
If you run a functional test session and want to make sure¬†that individual functional test sequences still succeed when the system is under load, it makes sense to trigger and control the NeoLoad tests from Ranorex. The other way round, if you‚Äôre running load testing scenarios and want to know if key functional uses cases still work, it makes more sense to trigger the respective Ranorex tests from NeoLoad.Q6: Is there a limit on virtual users, which Ranorex can add to a NeoLoad test?
The number of virtual users is not limited by Ranorex in any way. You can add as many users as your NeoLoad license provides.Q7: Do we need a Ranorex Runtime License for each virtual user in NeoLoad?
No. You can add as many users as provided in your NeoLoad license with a single Ranorex Runtime License. The Ranorex license does not influence the NeoLoad license in any way and vice versa.Q8: Is there a free version of Ranorex and NeoLoad so I can try the integration?
You could also run multiple Ranorex tests in parallel on multiple virtual machines. However, Ranorex is only able to run a single test per machine, as it is a functional test automation tool that automates real mouse and keyboard events. That‚Äôs why it makes sense to use NeoLoad to create load.Q10: What are the system requirements for Ranorex and NeoLoad to enable¬†this integration?
Yes, that‚Äôs no problem at all. Ranorex creates standalone executable files, which carry out the test. These files have to get triggered by the CI system. You can find instructions¬†on how to integrate Ranorex tests in your CI system in our blog “Integrate Automated Testing into Any Continuous Integration Process“. NeoLoad also offers command line execution of load tests. You simply have to start the ‚ÄėNeoLoadCmd.exe‚Äô file, as described in detail on this page.Further Resources:
A moment of sudden insight. An idea for a feature so amazing, it‚Äôd transform your testing task with Ranorex into a moment of enjoyable, reliable, automated testing bliss and save you a good few hours of work. As you‚Äôre not part of the Ranorex crew and simply won‚Äôt get to code this feature into our software yourself, your incredible idea will go to waste. At best, it‚Äôd make for a great morning coffee conversation with your colleague.
What if we told you that we found a place where ideas count, and counts turn into actual features on our product roadmap? At Ranorex, we refer to this magical place as Ranorex User Voice. Using this platform, you can be part of our journey to creating the best test automation software. After all, we share the same goal: Align Ranorex to your actual automated testing needs.Ranorex User Voice ‚Äď the right place for your ideas on Ranorex
We believe in providing a space where your ideas count. Let us know how we can make your test automation experience with Ranorex even better. Now, you can directly interact with our product management on the Ranorex User Voice platform¬†by
They certainly do. We greatly appreciate your feedback. The more you contribute on this platform by submitting feature requests and voting for ideas, the more likely it is that your idea will make it on our product roadmap.
We‚Äôre looking forward to reading your feature requests soon!
Android Nougat is Google’s big refresh of its phone and tablet operating system – split-screen mode, quick reply to notifications, revamped settings and toggle menus all make your phone easier and more friendly to use.¬†We are happy to announce that with the¬†release Ranorex 6.1.1, we now support Android Nougat.
In addition to this update, some minor bugs have been fixed.¬†Check out the release notes for more details about the changes in this release.
Let's be honest: We rarely test the product functionality under load. But how can we be sure¬†our end product works when our customers are using it? As we've described in our previous blog post "Combining Automated Functional and Load Testing", it often makes sense to combine functional and non-functional tests.¬†A functional test, which works fine in idle conditions, might fail when the back-end server is under load. Just like simply stressing a back-end system may not reveal functional issues, which can¬†only be found by an automated functional test. If we want to find those errors that only occur under load, we have to combine automated functional tests and automated load tests.
We're happy to announce that you can now combine Ranorex and NeoLoad tests!
In this blog, we want to show you how you can set up the Ranorex-NeoLoad integration and¬†what you can do with it. But first, let's quickly cover the basics:What is NeoLoad?
NeoLoad is an automated load and performance testing tool from Neotys.
NeoLoad offers a full-fledged REST API to either remote control the execution of a NeoLoad test or transmit timing¬†values to NeoLoad. To enable integration with Ranorex, the REST API calls are wrapped with Ranorex functions and packaged into a NuGet package for easy deployment.What do I need to enable the Ranorex-NeoLoad¬†integration?
Now that you're all set, we¬†want to show you in detail how you can:
First, we¬†need to set up the integration:
This will automatically add the necessary libraries to the Ranorex project. The following code modules will now appear in the module browser:
Step 2: Extending the 'app.config' file
To ensure the Ranorex project is compiled correctly, you need to extend the 'runtime' section in the 'app.config' file of the Ranorex project.
To do so, open the NEOLOAD_README.txt file, which was automatically added to your Ranorex project with the Ranorex ‚Äď NeoLoad NuGet package.
Copy the assembly binding information from this file, and paste it into the 'runtime' section of your local 'app.config' file.
Important: Copy <assemblyBinding> and its child-tags only.
Before pasting the copied code, your 'app.config' file should look like this:
After pasting the code section from the NEOLOAD_README.txt file, your 'app.config' file should look like this:
You can now¬†use the modules, which are included in the NuGet package, freely within the Ranorex test automation project.Modules included in the Ranorex-NeoLoad NuGet package
The following modules, and their individual variables,¬†are included in the Ranorex-NeoLoad NuGet package:
This module establishes a connection to the NeoLoad runtime API. This API is used to remote control a running NeoLoad test. It must be initialized before using the following modules: Start/StopNeoLoadTest and Add/RemoveVirtualUsers.Show the variables available for this module RuntimeApiUri: The Uniform Resource Identifier (URI) of the NeoLoad REST service.
Select Edit > Preference to access these variables in NeoLoad.
This module establishes a connection to the NeoLoad data-exchange API. This API is used to transmit external data to a running¬†NeoLoad test (it is not active if no test is running). This module¬†must be initialized before the modules¬†Start/StopNeoLoadTest and Add/RemoveVirtualUsers.Show the variables available for this module The following three values provide meta information for a running test and can be used in the "filter" functionality within NeoLoad test results.
Select Edit > Preference to access the last two variables in NeoLoad.
This module starts¬†a NeoLoad test scenario. You need to define the scenario in NeoLoad before.Show the variables available for this module Scenario: The scenario, as defined within the NeoLoad test, that should be started.
Interval: The time interval¬†(in hh:mm:ss) after which Ranorex¬†retries to¬†start a specific test (recommended value: 00:00:10).
Important: Please make sure to add¬†a leading 0 before a single digit number when entering the timeout and interval values.StopNeoLoadTest
This module stops¬†the currently running NeoLoad test.Show the variables available for this module Timeout: The maximum amount of time (in hh:mm:ss) given to Ranorex to start a specific test (recommended value: 00:01:00).
Important: Please make sure to add¬†a leading 0 before a single digit number when entering the timeout and interval values.AddVirtualUsers
This module adds¬†virtual users to a population, defined in a NeoLoad test scenario. This module can only be used when a test is already running.Show the variables available for this module Population: The population, as defined in the NeoLoad test scenario, virtual users will be added to.
This module removes¬†virtual users from a population, which is defined in a NeoLoad test scenario. This module can only be used when a test is already running.Show the variables available for this module Population: The population, as defined in the NeoLoad test, virtual users will be removed from.
Opening a website is related to a certain latency. This latency depends on various factors, such as the network connection or the browser used. It¬†can be measured with the "Navigation Timing" API, which is offered by all browsers. If you evaluate¬†these timing values, especially when the website is under load, you can¬†localize potential bottlenecks. Eliminating the identified bottlenecks will ultimately improve the user experience.
The NuGet package offers a mechanism to calculate these timing values and transmit the results to NeoLoad. You can find a more detailed description of the navigation timing¬†here.¬†The timing values¬†are calculated by the Ranorex/NeoLoad Nuget package:
Highlighted in green, you can see the timing values that are calculated by Ranorex and submitted to NeoLoad.
To transmit the timing values, you need¬†to drag and drop the repository root element, which represents the¬†DOM of the website under test, into the action table in Ranorex Studio. Once the NuGet package is added to the Ranorex project,¬†the additional entry "SendNeoLoadTimingValues" will appear in the "Dynamic Actions" list.
Please note: This entry only appears if the root element was created after the NuGet package was added to the Ranorex project. You can find a description of how to enable the NeoLoad "capability" in an existing repository here.
The "SendNeoLoadTimingValues" action accepts a "transaction name" as an argument. We¬†recommend using the current page as a transaction name in the Ranorex action table. As soon as NeoLoad receives the timing¬†values of this transaction, a¬†tree with the root node containing the¬†Ranorex test suite is¬†automatically created. Another subfolder is automatically created for the respective transaction name. This folder contains the timing values transmitted from Ranorex.
Important: ¬†Please make sure to initialize the module "ConnectToDataExchangeApi" before you use the module¬†"SendNeoLoadTimingValues". Otherwise, an error is thrown.¬†
You can drag the¬†data series into the graph board in NeoLoad to visualize it. If you've provided meta-information, such as "Hardware", "Software" or "Location" in the "ConnectToDataExchangeApi", you can now use this information to filter timing values transmitted from Ranorex.Update meta-information in cross-browser tests
If you execute the test in multiple browsers, you have to update the filter options in NeoLoad by calling the "ConnectToDataExchangeApi" module again. To do so, bind the data column, which specifies the browsers, with¬†the "Software" argument from the "ConnectToDataExchangeApi" module. You can now compare timing values from different browsers.
Exemplary Ranorex project structure
In the screenshot below you can see an example of how you can use the modules provided in the NuGet package within a Ranorex test project:
As you can see, a connection to the runtime API is established in the global setup section. The login information take the form of global parameters.
At the very beginning,¬†"StartNeoLoadTest" starts the NeoLoad test scenario. The following¬†test case is data driven and provides the number of virtual users that will be added to the test. These values are provided¬†in "AddVirtualUsers".¬†The inner loop is a cross-browser loop. It defines the browsers in which the test will be executed.
Please note: The module "ConnectToDataExchangeApi" can be called multiple times to update the current browser with the filter feature in NeoLoad.Upgrade an existing Ranorex project with the Ranorex/NeoLoad NuGet package
If you add the NuGet package to an existing Ranorex project, which already contains a Ranorex Object Repository with repository elements, the modules provided by the NuGet package are automatically available in the module browser. In this case, the "SendNeoLoadTimingValues" option won't be available in the "Dynamic Actions" list for the already existing repository items. Perform the following steps to enable this option:
1. Open the RanoreXPath editor
2. Switch to "Browser & Results"
3. Drag and¬†drop the root element from the Ranorex Spy to the matching root element in the Ranorex Object Repository.
Now, the "SendNeoLoadTimingValues" will be available in the Dynamic Actions list for the repository root element that describes the website DOM.Conclusion
In this blog, you've learned how you can combine Ranorex and NeoLoad tests. You've seen the modules and variables that are available with this integration and how you can transmit timing values to a NeoLoad performance tests. Now, you will be able to validate critical business cases under load conditions to ensure system stability under real usage conditions and identify potential bottle-necks across technology borders.Further Resources
At conferences, we¬†often get asked how you can¬†increase risk coverage and ensure highest user experience. Fact is that budget constraints and time pressure force testers to achieve maximum results using a minimum of resources. In reality, there’ll always be those¬†problems¬†that occur once the product is released. These are difficult to predict, but can greatly¬†impede the user experience. In this blog we¬†want to show you why it often¬†makes sense to combine functional testing and load¬†testing.Why performance and functionality of a system are important
First and foremost, an app or website has to work correctly. In an online shop, the user has to be able to open the site, select the desired product, add it to the shopping cart and complete¬†the order. If one of these steps doesn’t work, the user cannot buy the product. This leads to poor user experience (UX) and worst case, results¬†in a lost customer. But also the reliability of a system influences the UX. A site has to work! But it has to work over and over again, even under varying conditions. It has to work when the traffic increases, different browsers are used, but also if new functionalities are added. If a website does not offer reliable performance, customers will shy away from using it. Fact is that functionality and performance are important to the success of your website.
Ideal case: you test every possible scenario. But tests take time and no one‚Äôs got time to spare. We need to decide which tests to automate, when and how often. A risk assessment is crucial to pick the right test cases.Assess risk contribution of potential issues
When creating a test strategy with maximum risk coverage, you have to estimate the probability that certain issues occur and assess their impact on the system. Some functional issues only surface¬†if the system is put under stress. These are particularly hard to predict. However unlikely these issues¬†may be, their impact¬†may just not be worth the risk. Just think about the consequences if a ‚ÄėBuy‚Äô button in the shopping cart doesn‚Äôt appear on Cyber Monday, when thousands of people access the site at the same time.Separated automated functional and load testing
Typically, testing types are clearly separated. This is also true for automated functional and load¬†testing. Automated functional testing¬†enables you to¬†automatically and reliably verify the system‚Äôs functionality. Load testing verifies the system‚Äôs behavior under peak load. A typical sequence of tests¬†could look like this:
With functional automated testing, you first validate if your website or app works¬†as expected. Next, the team¬†tries to manually find bugs while exploring the software. These¬†tests are performed while the system is idle. As soon as they succeed, a performance¬†test verifies the system’s behavior in real usage conditions¬†– varying traffic, multiple users, increased network load, etc. Finally, the team manually validates the system using¬†simulated load conditions.Downsides of separating load from functional testing
First of all, this testing scenario includes two sections of manual testing. Naturally, manual testing takes¬†a lot of¬†time, is brittle and reduces test coverage. Additionally, your automated functional tests will only provide you with feedback on how your system works in idle conditions. You won’t be able to evaluate the load test results for functional correctness. It is not until the very end of the testing process that you will find out how your system works¬†under stress, in real usage conditions.Benefits of combining automated functional and load¬†testing
In reality, there will always be those bugs that occur if certain conditions correlate: the link breaks if the site traffic increases or the site content only loads after a refresh. To increase risk coverage and find those functional¬†issues¬†that manifest under load, we need to include¬†load in our automated functional tests. A testing process could now look like this:
You can now incorporate a sustained load in your automated functional tests. This approach to testing enables you to reliably test key functionalities that are most valuable to your business under realistic usage condition. You will be able to identify those issues that only manifest under load early in the testing process, which makes it easier to fix them and will ultimately save you a lot of time. But more importantly, you will finally get tremendously important insights on how end users experience your website. Ultimately, it is up to you to assess the potential impact certain issues may have on your system and if combining these two testing types makes sense in your individual case.
We¬†would like¬†to announce that you can now combine Ranorex functional tests with NeoLoad performance¬†tests!Further Resources