Introduction
Test automation development is software development. Hence all the metrics would apply and the architecture, standards and procedures have to be well in place. Obviously automation suite should not have more lines of code than the application code. From the viewpoint of the automated test application, every aspect of the underlying application under test is data. Similarly, every Testcase shall be treaded as a feature
Benefits
1. Save Time
2. Reduce Human Error
3. Track of what is done
4. Compatibility Testing [eg: Browser compatibility]
5. Performance Benchmarking
6. Regression with little effort
7. ROI in the long run
Framework Proposal For MaasS360
The proposed framework is oriented towards encompassing
Keyword for test instructions [Action, Command etc]
Component [UI components – Unique property as in id or name etc for Window, CheckBox, TextBox, Image, Link, etc,]
Content – [Text, Media]
Data
Support as well as test Libraries [Routines - Re-usuable functions, pre-requisites etc]
Keywords for Test Instructions
This will require the development of data tables and actions and commands, which shall be used to execute the tests on the application-under-test and the data.
Component
Different possible types of components have to be identified from the application under test. They should shall be identified by any of their respective property that is unique[like id, name etc]. Example of such components can be Window, CheckBox, TextBox, Image, Link, etc,
Data Handling
Here the test input and output values are read from data files (ODBC sources, CVS files, Excel files, or any other objects) that are loaded or scripted in test scripts. Hence navigation through the program, reading of data files and logging test-status with related information shall be coded in the test-script. This has crtain advantages like the following
1. It will allow us to develop scripts while application development is still in progress
2. Reduces the redundant effort as it will be very much modular with the input or the validation data in files
3. Incorporating changes will not require a huge effort
4. Since data input/output are stored in files they are easily maintainable
5. Functions shall return either ‘true’ or ‘false’ to the calling script for effective error handling to increase the robustness of the testscripts
Libraries
The test library architecture can be put in place by grouping the application-under-test into procedures and functions (or objects and methods depending on the implementation language) instead of scripts. This will require to create a set of library files (SQABasic libraries, APIs, DLLs etc) that represent modules, sections, and functions of the application-under-test. These library files shall be called directly from the test case script. This will help in modularization that will reduce the risk in maintaining the tests.
The Support Libraries are the general-purpose routines and utilities that let the overall automation framework do what it needs to do. They are the modules that provide services like, File Handling , String Handling, Buffer Handling, Variable Handling, Database Access, Logging Utilities, System\Environment Handling, Application Mapping Functions, System Messaging or System API Enhancements and Wrappers
Applications/Features Considered
The applications-workflows for automation shall be considered only if the following are satisfied
1. Manually tested over a period of time
2. Stable
3. Repetitive workflows that are time consuming
4. Core features that requires regression every-time there is a bug-fix, release or an incremental release
5. That requires thorough last minute regression
UMS
Where does this module get data from? How many fields are input boxes [note the rules for each component], how many fields take values from other modules[dependencies]?
Example
Create User
Except the User details and the password under the General Information tab every other tab or fields require just a selection of a value that comes from the other modules. These values will determined around the following items
Customer Type
1. Normal
2. Re-seller
3. Privileged
4. Regular
User Type based on Domain
1. Authenticated user
2. Non-Authenticated user
Field Dependencies
The field values that are dependent on other modules/workflows have to be determined. There are certain rules that has to incorporated as conditions and validation points
Access Rights
Various access rights for various users. So, the rules would apply.
Field Validation
Input fields under the general information tab. Rules would apply
Tests Considered for Automation
1. All tests that are independent
2. Similar tasks with different data should be grouped. So that the same test can be executed multiple number of times with different data
3. Tests that are well tested manually.
4. A near stable application under test
Creation of Dataset
Overview
1. For the existing applications, a dataset shall be created which will be pristine dataset.
2. Every time a change happens, while it is being manually tested, the data for such tests will be added to the persistant dataset which shall be included into the pristine set.
3. For automation purpose, mirror images [Testgroups] can be used. There shall be a scheduler to refresh the testgroups at a specified interval of time.
Analyzing Data
Data
Example
Customer Data Characteristics & Categorization
Customer Static, Pre-Defined Data
Roles – Access Rights Static, Pre-Defined Data
Customer Setup Custom(seed) Data, Predefined Set
State & Requests Dynamic
Time Stamp Dynamic output data
Product Data User Defined
Billing Data Internally Generated
Performance Volume
Creation of Test Groups
It is mandatory that test-groups are created. Mirror Images of pristine, persistent dataset - minimum two test-groups shall be accommodated, say, TG1, TG2
Script template
Browser Selection, Data pool selection (test group), Pre-requisites
1. Teststeps of the testcase that is considered for automation should be incorporated in the script as comments
2. Start a new Browser and launch the app.
3. A reusable function to login with arguments as username, password
a. Login(‘username’, ‘password’)
4. A reusable function to Close all the popup like Alerts etc…
a. ClosePopup
5. Script for the test
6. A reusable function to Clear Browser Cookie, cache etc...
a. ClearMaas
7. A Reusable function to Close or Kill the Browser, session. This shall be the last line of the script
a. EndMaas
The above template shall be complemented with the standards and procedures document
Automation Environment
1. There shall be a separate environment known as STABLE environment, which will always be a mirror of the production environment, except before a new elevation/release.
2. As the name says, this shall be more stable than staging and hence shall be accommodated between staging and production.
3. Test automation shall be mostly done on this and mostly run on the same. Though regression using the automated test suite, shall be done on staging and QA.
4. This shall be linked to the test-groups [Mirror Images of Persistent dataset say, TG1, TG2]
5. Every day, a scheduler can be put in place to run a data refresh, so that TG1 and 4. TG2 will always have the same data as the Persistent dataset.
6. A wrapper can be built around the suite/batch to make it run on any browser and to use either TG1 or TG2.
7. Else to cut it short the STAGING can be used as the automation environment, provided there is a plan in place, so that when the QA happens on staging, the automation team is not affected. Though the test groups shall be in place.
Thought Points
1. Generating log-entry any time the program reaches an unexpected state, makes an unexpected state transition, manages memory, stack space, or other resources in an unexpected way, or does anything else that is an indicator of one of the types of errors under investigation.
2. Simulation – Hitting the application hard with huge volume of data example – state transitions etc… 1. Preparing our Platform application for Automation [Not an exhaustive list though...]
3. Numbering/Naming Platform apps in terms of application, reports, data
4. Enabling a FastPath
5. Single page shift to different Browsers/Browser Flavors/OS
6. -Test/Requirements Coverage [Tree, Traceability Matrix]
7. Test Suite/Testbed organization
8. Test Reporting/Analysis
Test automation development is software development. Hence all the metrics would apply and the architecture, standards and procedures have to be well in place. Obviously automation suite should not have more lines of code than the application code. From the viewpoint of the automated test application, every aspect of the underlying application under test is data. Similarly, every Testcase shall be treaded as a feature
Benefits
1. Save Time
2. Reduce Human Error
3. Track of what is done
4. Compatibility Testing [eg: Browser compatibility]
5. Performance Benchmarking
6. Regression with little effort
7. ROI in the long run
Framework Proposal For MaasS360
The proposed framework is oriented towards encompassing
Keyword for test instructions [Action, Command etc]
Component [UI components – Unique property as in id or name etc for Window, CheckBox, TextBox, Image, Link, etc,]
Content – [Text, Media]
Data
Support as well as test Libraries [Routines - Re-usuable functions, pre-requisites etc]
Keywords for Test Instructions
This will require the development of data tables and actions and commands, which shall be used to execute the tests on the application-under-test and the data.
Component
Different possible types of components have to be identified from the application under test. They should shall be identified by any of their respective property that is unique[like id, name etc]. Example of such components can be Window, CheckBox, TextBox, Image, Link, etc,
Data Handling
Here the test input and output values are read from data files (ODBC sources, CVS files, Excel files, or any other objects) that are loaded or scripted in test scripts. Hence navigation through the program, reading of data files and logging test-status with related information shall be coded in the test-script. This has crtain advantages like the following
1. It will allow us to develop scripts while application development is still in progress
2. Reduces the redundant effort as it will be very much modular with the input or the validation data in files
3. Incorporating changes will not require a huge effort
4. Since data input/output are stored in files they are easily maintainable
5. Functions shall return either ‘true’ or ‘false’ to the calling script for effective error handling to increase the robustness of the testscripts
Libraries
The test library architecture can be put in place by grouping the application-under-test into procedures and functions (or objects and methods depending on the implementation language) instead of scripts. This will require to create a set of library files (SQABasic libraries, APIs, DLLs etc) that represent modules, sections, and functions of the application-under-test. These library files shall be called directly from the test case script. This will help in modularization that will reduce the risk in maintaining the tests.
The Support Libraries are the general-purpose routines and utilities that let the overall automation framework do what it needs to do. They are the modules that provide services like, File Handling , String Handling, Buffer Handling, Variable Handling, Database Access, Logging Utilities, System\Environment Handling, Application Mapping Functions, System Messaging or System API Enhancements and Wrappers
Applications/Features Considered
The applications-workflows for automation shall be considered only if the following are satisfied
1. Manually tested over a period of time
2. Stable
3. Repetitive workflows that are time consuming
4. Core features that requires regression every-time there is a bug-fix, release or an incremental release
5. That requires thorough last minute regression
UMS
Where does this module get data from? How many fields are input boxes [note the rules for each component], how many fields take values from other modules[dependencies]?
Example
Create User
Except the User details and the password under the General Information tab every other tab or fields require just a selection of a value that comes from the other modules. These values will determined around the following items
Customer Type
1. Normal
2. Re-seller
3. Privileged
4. Regular
User Type based on Domain
1. Authenticated user
2. Non-Authenticated user
Field Dependencies
The field values that are dependent on other modules/workflows have to be determined. There are certain rules that has to incorporated as conditions and validation points
Access Rights
Various access rights for various users. So, the rules would apply.
Field Validation
Input fields under the general information tab. Rules would apply
Tests Considered for Automation
1. All tests that are independent
2. Similar tasks with different data should be grouped. So that the same test can be executed multiple number of times with different data
3. Tests that are well tested manually.
4. A near stable application under test
Creation of Dataset
Overview
1. For the existing applications, a dataset shall be created which will be pristine dataset.
2. Every time a change happens, while it is being manually tested, the data for such tests will be added to the persistant dataset which shall be included into the pristine set.
3. For automation purpose, mirror images [Testgroups] can be used. There shall be a scheduler to refresh the testgroups at a specified interval of time.
Analyzing Data
Data
Example
Customer Data Characteristics & Categorization
Customer Static, Pre-Defined Data
Roles – Access Rights Static, Pre-Defined Data
Customer Setup Custom(seed) Data, Predefined Set
State & Requests Dynamic
Time Stamp Dynamic output data
Product Data User Defined
Billing Data Internally Generated
Performance Volume
Creation of Test Groups
It is mandatory that test-groups are created. Mirror Images of pristine, persistent dataset - minimum two test-groups shall be accommodated, say, TG1, TG2
Script template
Browser Selection, Data pool selection (test group), Pre-requisites
1. Teststeps of the testcase that is considered for automation should be incorporated in the script as comments
2. Start a new Browser and launch the app.
3. A reusable function to login with arguments as username, password
a. Login(‘username’, ‘password’)
4. A reusable function to Close all the popup like Alerts etc…
a. ClosePopup
5. Script for the test
6. A reusable function to Clear Browser Cookie, cache etc...
a. ClearMaas
7. A Reusable function to Close or Kill the Browser, session. This shall be the last line of the script
a. EndMaas
The above template shall be complemented with the standards and procedures document
Automation Environment
1. There shall be a separate environment known as STABLE environment, which will always be a mirror of the production environment, except before a new elevation/release.
2. As the name says, this shall be more stable than staging and hence shall be accommodated between staging and production.
3. Test automation shall be mostly done on this and mostly run on the same. Though regression using the automated test suite, shall be done on staging and QA.
4. This shall be linked to the test-groups [Mirror Images of Persistent dataset say, TG1, TG2]
5. Every day, a scheduler can be put in place to run a data refresh, so that TG1 and 4. TG2 will always have the same data as the Persistent dataset.
6. A wrapper can be built around the suite/batch to make it run on any browser and to use either TG1 or TG2.
7. Else to cut it short the STAGING can be used as the automation environment, provided there is a plan in place, so that when the QA happens on staging, the automation team is not affected. Though the test groups shall be in place.
Thought Points
1. Generating log-entry any time the program reaches an unexpected state, makes an unexpected state transition, manages memory, stack space, or other resources in an unexpected way, or does anything else that is an indicator of one of the types of errors under investigation.
2. Simulation – Hitting the application hard with huge volume of data example – state transitions etc… 1. Preparing our Platform application for Automation [Not an exhaustive list though...]
3. Numbering/Naming Platform apps in terms of application, reports, data
4. Enabling a FastPath
5. Single page shift to different Browsers/Browser Flavors/OS
6. -Test/Requirements Coverage [Tree, Traceability Matrix]
7. Test Suite/Testbed organization
8. Test Reporting/Analysis
No comments:
Post a Comment