Mobile website Performance Testing - Akamai Mobitest

Came across this one, for Mobile website performance testing.

Akamai Mobitest is a free tool created to raise awareness to Mobile Web Performance.
Your page is loaded on a real mobile device, and you'll receive rich detail about how long it took to load, including waterfall charts and video recording of the page load.
The Mobitest agent has been open-sourced, so you can install it on your own devices, connected to your own WebPageTest private instance.
The site provides an option to compare the history of tests.

Go this url http://mobitest.akamai.com/m/index.cgi
Enter your mobile website in the space provided for website url
Choose your device (iOS 5.0, andrid 2.3)
Specify how many runs
Choose Location
Run Performance Test and know the results for your mobile website!

Calabash Cucumber for iOS

Calabash Cucumber for iOS - Required Tools
Calabash library and framework
Cucumber tool - To execute the feature
Gherkin – to write steps [Behavior driven, domain specific language]
Jenkins/maven for Reports
Ruby gems – All of the above require ruby, as they are written in ruby

Calabash iOS
Automation and test library for iOS
Behavior Driven Development for testing
Consists of 2 parts
Client library : written in Ruby
Calabash.framework: server framework written in objective C ( have to be linked to our test build)

Calabash Architecture
The server framework will start an HTTP server inside our app that listens for requests from the client library (Courtesy: Lesspainful.com)



Accessibility Labels
One of the ways calabash interacts with the UI elements in your app via accessibility labels.
That’s how it knows which button to tap when a command like Then I touch "reply“ is issued
Enable this from Settings->General->Accessibility->Accessibility Inspector (ON)

Writing a test
You’ll need two files
xxx.feature (steps are written using Gherkin)
Xxx.rb (steps are implemented using ruby)
There are a lot of predefined steps that come when you install calabash cucumber iOS
Custom steps can be written using ruby

Sample Feature File
Feature: Send Email Logs from Workplace
   As an iOS tester
   I want to  go to My device tab
   so I can send email logs
Scenario: Send Email logs
   Given I am on my device
   Then I touch “Email Logs”
Sample Step definition
Given /^I am on my device$/ do
  if view_with_mark_exists(“Email Logs”)
  macro ‘I should see a “Email Logs” button’
  end
end

Running on simulator
Bind your app with calabash.framework and create a testbuild and launch it in simulator
The following will be there on Xcode console
Creating the server:
Started LPHTTP server on port 37265
Bonjour Service Published: domain(local.) type(_http._tcp.) name(Calabash Server)
 
Command to run on simulator
$DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber features/send_email.feature
Run Group of tests
Tag the scenarios with @
@negative_test
Scenario: Send email button pushed down
DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber --tags @negative_test
Exclude some tests by ~
DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber --tags ~@negative_test

Running it on device
The device and the machines that runs xcode has to be on the same wifi.
DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber features/send_email.feature DEVICE_ENDPOINT=http://your.device.ip.address:37265

Calabash-ios Console
You can get into the interactive ruby console
$calabash-ios console
Another way to find out UI elements is thro console.
$query "tableViewCell index:0“
This can be used to write custom step definitions.

Record and Playback from Calabash-ios Console 
Record and playback
$record_begin
$Record_end “xxx”
Creates a .base64 file with the above name
$playback “xxx”

Download links
https://github.com/calabash/calabash-ios

iOS Automation

I finally decided to write on iOS automation after evaluating couple of tools.
The tools that I tried are
1. Fone Monkey (I tried it when it was fone monkey)
2. Test Studio
3. UIAutomation (from Xcode Instruments)
4. Calabash Cucumber framework.

Fone Monkey, was easy to setup and the device had to be on the same wifi as the desktop/laptop, where the fonemonkey is integrated with the build.
As it works via the JSON string commands. the difficulty was the same script passed and failed based on the speed of connectivity.

Test Studio, the scripts are easy to comprehend. Easy for demo purposes. But to edit and insert your own code will be difficult. also, not sure if the latest version supports the hybrid app. coz html links were not working. Also, the scripts once written had to be exported. And the scripts have to be recorded for each device stack.

UIAutomation from Xcode Instruments. It is a viable tool where the scripts can be written using javascript. You can automate both native and hybrid apps. It is an amazing tool, where the UIAutomation ref pdf from apple is up to the mark. But you may need to to maintain the code.

Calabash Cucumber, has a long list of setup instructions which will guide you to install ruby version manager (rvm) and ruby gems, calabash ios, cucumber etc. But they are to the point, including the possible errors that one might encounter. Once you set that up, there are predefined functions which are called predefined steps. They are all included in the resources file. All you need to do is start writing code in natural language in a file that has an extension as .feature. Viola, the script runs flawlessly! You have a specific requirement, you can write your own custom steps. Also, record and playback is easier, not only that, you can save it as a .base64 file and can call from your script using the playback function!. Since it is in natural language, you dont need an automation specialist to write your scripts.
The only drawback is, if you have a backend platform (portal) to which the app talks, those steps cannot be automated using this, except for the workflows that you can execute with curl commands, or the ones, where you can make a function call. But for an app only application, that does not require end to end (from portal to app), this is a blessing. Support wise there are a lot of blogs threads, where you can get prompt answers.

Jailbreak iOS 6.1.2 with Evasi0n

Today I did jail break an iOS 6.1.2 device with evasi0n (http://evasi0n.com/).
I should admit, that it was a pleasant experience in a long time!
I have been continuously jail-breaking every release of iOS from 4.0 for testing purposes.
Things were easier with just Jailbreakme.com. But with the later versions of iOS, the number of steps were up with number of tools to be downloaded, like, setting your device in recovery mode, redsn0w, tiny umbrella and the number of device reboots etc.
Among these green poison absinthe was easier to explain to others.
Also, there were problems like the jailbreak not working on device reboot, device freeze etc.
But today I tried with the Evasi0n and I pretty much did not do anything apart from downloading Evasion and clicking on Jailbreak.
Abracadabra!! the device is jailbroken and cydia was installed.
However - there is one thing that has to be taken care of: Disable the password for backup and recovery of the iPhone

Defect Triggers

A trigger is any event that allowed a fault to become a failure.
Multiple triggers may uncover the same fault.
The same trigger may be used at any point in the software lifecycle.
There are 3 families of triggers: review/inspection, function test, and system test.

Review/Inspection Triggers

DESIGN CONFORMANCE: Comparing the implemented design against a reference –design document, pattern, or guideline.
LOGIC/FLOW: Checking for correctness or flaws using knowledge of the practice.
BACKWARD COMPATIBILITY: Examining compatibility with prior version of the product.
LATERAL COMPATIBILITY: Examining for compatibility with other products and platforms that need to work with this release.
CONCURRENCY: Serialization, shared resources, multi-threaded tasks, timing, etc.
INTERNAL DOCUMENT: Inconsistencies in prologs, and sections in the same work product.
LANGUAGE DEPENDENCY: Programming standards, specific implementation considerations, environment restrictions, execution modes, etc.
SIDE EFFECTS: Usage behavior beyond design, but relevant in context.  Do A; B happens.
RARE SITUATION: Unusual issues related to idiosyncrasy of environment, hardware, or software.

Function Test Triggers

DESIGN CONFORMANCE: Comparing the implemented design against a reference –design document, pattern, or guideline.
LOGIC/FLOW: Checking for correctness or flaws using knowledge of the practice.
BACKWARD COMPATIBILITY: Examining compatibility with prior version of the product.
LATERAL COMPATIBILITY: Examining for compatibility with other products and platforms that need to work with this release.
CONCURRENCY: Serialization, shared resources, multi-threaded tasks, timing, etc.
INTERNAL DOCUMENT: Inconsistencies in prologs, and sections in the same work product.
LANGUAGE DEPENDENCY: Programming standards, specific implementation considerations, environment restrictions, execution modes, etc.
SIDE EFFECTS: Usage behavior beyond design, but relevant in context.  Do A; B happens.
RARE SITUATION: Unusual issues related to idiosyncrasy of environment, hardware, or software.

Systems Test Triggers

WORKLOAD STRESS: Pushing the limits of performance, resources, users, queues, traffic, etc.
RECOVERY: Invoke exception handling, recovery, termination, error percolation, etc.
STARTUP/RESTART: Major events of turning on, off, or changing the degree of service availability.
HARDWARE CONFIGURATION: Issues surfaced as a consequence of changes in hardware setup.
SOFTWARE CONFIGURATION: Issues surfaced as a consequence of changes in software setup.
BLOCKED TEST/NORMAL MODE: Test could not run during System Test, or customer found nonspecific trigger.  Look for additional trigger.

Invited Talk - NCSE, SHC, TPT

I was invited to give a talk on Software Quality Assurance at the National Conference for Software Engineering 2012, Sacred Heart College, Tirupattur. The crowd, mostly research scholars and computer science were very eager to know, why I chose Quality Assurance as a profession and what are the possible research directions that they could under take.

I began the talk by explaining what Quality is, its relativity, characteristics and importance. It was more of a QA awareness campaign. Where a quality professional should know, what goes into the code, the wholesome factor - end to end and with customer-end user context . Ideally, it should be taken as a specialization on the base course. (For example, specialization in gynae after the base medicine degree)

And the gist of the talk can be found below.

Systems
Safety
Health critical
Transaction critical
Embedded
Computation/Math/Data/Prediction
Real Time/Signals/Mobile
Virtual
ERP/CRM etc

What goes into the production code?
 
Various Engagement Models
Consulting (on Prem/ off prem)
Independent Testing
Crowd Sourcing
Factory Method
School of Testing/Heuristics
Business Validation/User Acceptance
Transitioning from testing requirements to Testing ideas
Automatic build deployment and verification etc

Research Directions

•       Measuring software Quality (Quantitative/Qualitative)

–       Metrics / Cost of Quality

•       Quality Methodologies / Test Design Techniques

•       Quality Processes/Agile methodology/Quality Frameworks and Standards

•       Test Processes / Test life cycle / Test Phases / Test strategies and methods

•       Complexity Analysis for testing

–       Test Estimation / Function Point analysis

•       Code/Feature/Test Coverage

•       Defect Prediction / Root Cause Analysis / Orthogonal defect classification

•       Test Generation from Formal Specification / Model Based Testing

•       Design of Experiments

–       Combinatorial/ pairwise / orthogonal array / Fractional factorial design

•       Performance/Scalability/Reliability (PSR)

–        Measuring / Predicting / Certifying

•       Testing on cloud / Testing applications on Cloud

•       Test Automation Framework

–        Ad Hoc/Data Driven/Keyword Driven/Model Driven/Query Driven

•       Integration and interoperability

•       Test Optimization / Prioritization/Operation Readiness

•       Test Organization/Test Maturity Model

•       Test Assessment/Domain Maturity/Analysis & reporting/Process Improvements

•       Data Mining and Inferences / Localization and Internationalization

      •       Message Testing/Data Mining