Archive for the ‘Manual Testing’ Category

ERP Testing Saps Resource at Marstons

When Judy Doust at Marston’s told us about her pain points with testing SAP applications, it was obvious that our unique approach to testing would be beneficial. So we put our developers to the task of coming up with an SAP solution. We’re pleased to say they’ve come up trumps and have helped the independent UK brewer and pub retailer to ease the increasing burden of SAP testing without the need for employing additional technical resources…

Historically Marston’s had only been using two SAP modules: Financials and Sales & Distribution, yet it would take over six months to plan and execute the testing of each upgrade. Now the company is using additional modules within the SAP ERP Central Component solution, CRM and SRM within the SAP Business Suite, SAP BI, bespoke applications and many web applications. This increased complexity had become a growing concern for Marston’s IT department. Continuous manual testing was not a feasible option moving forward, especially with the way SAP was planning updates in its latest release. Instead of one big upgrade, patches were to be delivered in regular packages throughout the year. For Marston’s the scope and complexity of testing SAP updates has dramatically increased.

In addition to the long testing cycles, Marston’s also felt it was constrained by a lack of choice in the SAP automated testing market and required the solution it eventually purchased to fit in with how Marston’s ran its business. Marston’s has always utilised the business users as its testers, as it is they who really know how the applications are supposed to work. The business users have no IT skills, but have a more holistic view of the business processes under test. As such, the solution had to be easy to use without the need for any technical knowledge such as programming.

One choice would have been HP, given its monopoly-like relationship with SAP for testing. But HP was not a realistic choice for Marston’s.

Judy Doust, Test Manager at Marston’s, explains: “Our SAP testing is business process driven and as such requires people with knowledge of the business process to actually design and execute tests. Although the integration exists between HP and SAP, I really didn’t like feeling as though I was being tied into something that was not right for our business. HP is a technical solution and I didn’t want to employ another couple of people to do the coding and maintenance, so I went looking for a viable alternative.”

“Our prime objective was not to employ a huge test department. Our end users are the business users – we’ve always used them as testers and it works well for us – it saves on re-work. A system that requires IT specialism would force us to employ a team of specialist testers. We’d still need to utilise the users for the business process knowledge and there would be crossover and duplicated effort. Why perform all that work if there was another option out there?” continued Judy.

When Marston’s learnt about Original Software and realised how easy the solution was in terms of usability, it immediately saw the benefits it could provide the business with its SAP testing. Judy added: “When I approached Original Software to investigate the solution suite’s suitability for SAP testing, their display of willingness and intent gave us confidence.”

“We have now implemented their manual and automated testing solution and are in the process of training users. The beauty of TestDrive-Assist is the non-intrusive way it records activity while the user is testing. This allows us to capture the day-to-day processes of the business users so that their knowledge is retained and shared within our organisation. We will be able to leverage that business process knowledge in order to build automated test regression packs, providing us with huge time savings each time we re-run the tests.”

Marston’s next objective is upgrade regression pack building. “We want to be able to automate and make upgrades less painful, but quality really is at the heart of everything we are doing. We want to improve the quality of our solutions, while at the same time maintain the same level of service for users. Our overall goal is that the business continues to believe in IT and what we are doing – all the way from the business process owners and end users, right up to board director level,” Judy concluded.

Drilling deeper into the World Quality Report

Some good headlines about emerging industry trends came out of the recent World Quality Report produced by HP, Sogeti and Capgemini, but as I read deeper, some things started niggling at me, so much so that I decided they were worthy of a blog post.

For instance:

‘Only 4% of IT professionals agree that their ALM investments are fully paying off, and only slightly more than a third said that half or more of their solutions have been fully implemented and are helping to improve application quality.’ (Pg.8)

HP and Sogeti state that ‘QA organizations need to focus on increasing the adoption rates of their purchased technologies’. This is of course great advice, but both companies have a vested interest in these users throwing more and more money at consultancy, training, and in HP’s case, overpriced and difficult to implement products, (that consequently need a lot of ‘services’ cash thrown at them).

In my opinion it would be more prudent to look in more detail at why these companies are not getting a return on their investments or managing to fully implement their products.

The answers seem to fall into three categories. Company issues, Resource issues and Technology issues, with technology resoundingly winning the prize for biggest stumbling block.

20% failed due to lack of an internal process or support from management. The question does beg to be asked – how on earth did they ever manage to purchase their ALM technologies without some form of internal support? To make ALM successful, it must touch more than just the QA team: support needs to be gained, processes mapped out and business goals and requirements defined way before you make decisions about what technology to buy. It’s really not surprising that these projects failed.

26% stated that not enough resources were invested into the adoption of the technologies. Now I could have classed this under company issues, but I think you really need to look deeper than just writing this off as a staffing issue. It could equally be a technology issue. Was it that not enough staff were trained? Was the project badly planned? Was the technology too complicated for users outside the dev/test team to adopt?

In terms of technology, a whopping 41% struggled because their ALM investment was the wrong choice of technology; it failed to integrate with other technologies or was too complicated and required specialist skills that were thin on the ground. I’ve lost count of the amount of times we hear stories like that when meeting companies across the globe. To truly embed a solution in your company, you need to empower all stakeholders. Unless everyone involved in the delivery of IT projects can collaborate using the solution, it’s just not going to work. When choosing a solution, you need to think about how easily management, business analysts, business users, developers, project managers and testers are able to get what they need out of the solution.

‘Companies prefer testers who have both strong technical skills and relevant domain and business knowledge’ (Pg.11)

Well the stats don’t really allude to that. The question that was asked was – When hiring testers which of the following skills are most important to you? Well obviously QA skills came out tops at 31%. Having a good grounding and understanding of the principles of Quality Assurance is key for testers, I’m actually more surprised that this figure wasn’t higher, but interestingly, the second largest desired skill is business knowledge (22%). This is something we come across time and time again with companies we talk to; so many of them utilise business users for the testing phase. Take SAP testing for example, business process is key. You really need to leverage the knowledge that the business users have about how the system is supposed to perform and exactly how they all use it. So many of the accounts we’ve been into have been literally banging their heads up against a brick wall trying to work out how to capture this knowledge or utilise these testers, knowing that it is impossible with their current toolset – HP is just too cumbersome to get non technical business users to adopt. Development skills 9% and scripting skills 10% are actually rated incredibly low when you consider that the market dominating tools actually force these prerequisites upon QA and make these skills imperative at sites where these traditional tools are embedded.

‘Nearly three quarters of respondents say that they do not follow [common test management methodologies]. Instead, their organisations develop and document their own best practices that are followed in the majority of development and testing projects. (Pg.10)

Different groups in the organisation may adopt their own ‘versions’ of the standard practices, and as a result, the company as a whole is not fully realizing the benefits of standardization, economies of scale, common metrics, unified reporting and asset reusability’

Not all companies are equal and each has different ways of doing business. One size DOES NOT fit all, so surely it is good for the industry that companies develop their own best practices? These companies are just using their brains and working out what best suits their own unique needs and circumstances.

Software vendors should be supporting this very obvious progression of development maturity. Why shouldn’t they be able to all work slightly differently, yet still enjoy the benefits of unified reporting, asset reusability, common metrics etc.? Perhaps the answer lies in the fact that HP hasn’t built its software to be this flexible? Maybe it’s time for the dominant market player in test automation and management to listen to what businesses need rather than telling them how they should be working!

This particular bone of contention was revealed earlier this year with a survey of Application Development Managers back in April 2010. The industry is really crying out for flexibility in the way that tools allow them to work, which is one of the reasons that Original Software developed Qualify, a process and methodology agnostic Application Quality Management solution. Qualify allows businesses to map their own processes, use standard methodology templates, tweak them to suit their own needs and even run multiple methodologies across different teams and projects, with – wait for it, all the added benefits of unified reporting, metrics, re-usability and economies of scale.

Go and check Qualify out.

Customers Get Quality-Centric VIP Treatment

Original Software’s robust approach to software quality helps Vermont Information Processing reduce release bottlenecks and improve customer support forecasting

Vermont Information Processing (VIP), IT solutions supplier to the beverage wholesale industry, has utilized Original Software’s end-to-end testing solution to increase its software quality. Even with an intensely accelerated development schedule, VIP has improved its customer upgrade experience and reduced pressure on its support team. This in turn has enabled the business to respond effectively to the complex requirements of its customers during volatile market conditions.

A period of market consolidation had caused significant growing pains for VIP. As its customers’ operations increased in complexity, the software that they relied on to manage their business needed to be able to handle much more complicated processes. VIP’s old development cycle just didn’t fit this new development schedule. Putting out new releases became painful, not just for customers and its development team, but the support team was also beginning to feel the repercussions.

With the help of Original Software, VIP was able to develop a much more robust approach to software quality. The company has completely turned around its QA process and overcome its customers’ cautious attitudes to upgrades. VIP has now transformed into a software testing center of excellence with happy customers who are keen to upgrade to new releases.

Since implementation, VIP’s QA department has increased the scope of test coverage, taking control its test environments, comprehensively unit testing and running full regression tests on all system updates. Previously customers went on average three years before updating systems, creating a bottleneck in the upgrade process, now 97% are running a recent version of the software. The team has also vastly reduced the strain on support. Fixing a single bug could save support around 80 hours; it takes a lot less time to run the regression suite now, which on average turns up 10-12 similar bugs. This equates to five and a half months of support time per release!

VIP moved from having one major update release every three years to producing major releases once a year, with numerous maintenance builds in-between. The old eight month-long beta testing phase was too time consuming for the business with its new rapid development schedule and it could no longer afford to do this. With the crunching development cycle, quality was becoming an issue for VIP. “Our customers were really frustrated with us,” explained Bobby Erwin, Quality Control Analyst at VIP. “To get a new feature, they’d also get all these other problems to boot. This would become a strain on the support team too, as they’d spend hours on the phone going through problems.”

“Our main business drivers were to reduce the level of risk by increasing the scope of test coverage. Original Software was the ideal choice” said Erwin.

VIP purchased the solutions from Original Software in May 2007 and once they’d had the initial training it took two or three months to develop their first regression capabilities. They’ve been developing and building on these tests ever since. “Prior to this we conducted no regression testing at all – just a month or so of acceptance testing by customer service and then the beta tests. Now we have a battery of tests we perform every time we release,” imparted Erwin.

“We are providing a much more robust and better product to the end users,” said Bobby Erwin. “Our support and customer service teams are delighted. Our builds are just so much better now, and we can also tell them exactly how risky a release is, and predict the levels of support that might be needed.”

“The regression tests help us manage our exposure: improving experience for the customer, decreasing support needs for a given update. Testing gives us a feel for how stable the build is. It’s a known entity. If the build looks good in testing, we can install with greater confidence. If we find problems, we get a better idea of the support resources needed,” confirmed Chris Boucher, Support Manager at VIP.

To read the full customer case study, please visit: http://www.origsoft.com/customer-stories/vip

Disposable test controversy

Search Software Quality has published an article this week by our very own George Wilson.

In his article he talks about the need for more disposable test assets. “In the last decade the rate of business change has risen beyond anything we could have expected. With increasing software development complexity and more and more IT departments taking on an agile approach, traditional test automation has become too cumbersome for most to sustain.”

He argues that “Test automation has failed to date simply because we could not afford to throw it away. Creating any form of automation takes effort and time, when the application under test changes and the automation ceases to work you are faced with a stark choice – either maintain it at additional effort and time or abandon it. If you abandon it you are also writing off the effort and time you invested in creating it, thus bringing the whole concept into question.” (you can read the whole article here)

It’s an interesting angle and certainly a problem that the technology we build at Original Software has been at pains to rectify, however it seems George has touched on a controversial topic. Discussions about the article have sprung up all over the internet and one commenter on a Linked In Group even went as far as saying “There is absolutely no need for ‘throwaway test automation’. Where on EARTH would he get that diagnosis or term?? The whole point of automation is that it doesn’t need to be throwaway.”

We think that the whole point of automation is that it should be flexible enough to adapt so that all your efforts don’t go to waste when applications change, but what about you? Which camp are you in? We’d love to hear your views….

The Seven Deadly Sins of Software Test Automation

Just to get into the spirit of Halloween next week, take a look at our new whitepaper, “The Seven Deadly Sins of Software Test Automation,” (cue evil laugh). We do enjoy being original and I am sure you will have a little chuckle to yourselves with this one!

In this whitepaper we explore each of the seven deadly sins as they relate to software test automation. Instances that Original Software comes across time and time again and traps clients have often fallen into because of their earthly vices.

I don’t want to give too much away,  so if you would like to download this webform free whitepaper, here is the download link http://www.origsoft.com/whitepapers/seven-deadly-sins/

I am quite eager to see what you think of this piece, so please write back with any comments! If you have a great sense of humour, why not reply back with your own software testing deadly sins!

Great Places to network with other Test and QA professionals

(Part 1 – Linked-In)

Linked-In is fast becoming one of the best platforms to pick up industry related news and network with other professionals. There are literally thousands of Linked-In groups, so much so, that finding the most relevant, biggest, best or most active groups can be a bit of a challenge!

To help you on your way, here is a selection of the groups we’re familiar with and can heartily recommend.

Agile (4,813 members)

Agile Enthusiast (638)

Agile Progect Management Group (3,851)

Agile Testing (595)

Agilistias (2,415)

American Society for Quality (11,054)

Automation Testing (518)

Methods & Tools (822)

QA Automation Architect (664)

QA Test Automation (1,010)

QA/Testing (1,193)

QAGuild Network (5,545)

Quality Assurance and Test Professionals / Analysts (242)

Senior Testing Professionals (1,579)

Software Test & Performance Group (914)

Software Testing & QA (997)

Software Testing & Quality Assurance (19,957)

Software Testing and Quality Assurance (2,561)

Software Testing Club (3,705)

SQA forums (2,366)

Test automation (3,875)

Test Development (552)

Test Management Forum (530)

Test Republic (298)

Test strategy and test management (741 members)

NEWS: AppLabs and Original Software Partner to Help Clients Take the Step from Manual to Automated Testing

AppLabs sees huge value proposition for its clients with agile-friendly automation suite

11 August 2009 – Original Software today announced a strategic partnership with AppLabs, the world’s largest software testing and quality management company. AppLabs has entered into an agreement with Original Software to take to market its innovative quality management suite. The agreement will likewise allow Original Software to position AppLabs’ services into its own customer base.

Scott Andress, Vice President, Global Alliances and Service Lines at AppLabs commented: “As a tool agnostic services company, we focus on putting the right solution into the right project. We’ve spotted a compelling value proposition for our clients with Original Software’s suite, which can be a great fit for most agile developments, can speed up time-to-market and can provide a silver bullet for those clients struggling to make the transition from manual to automated testing.”

The industry analysts agree. A recent Ovum report stated: ‘TestDrive-Assist provides you with a rich environment for manual testing, but the bigger payback is how it facilitates adoption of full automation. Uniquely for any of the guided manual test execution products that we know of, Original can automatically convert a manual TestDrive-Assist test into a fully automated TestDrive test that can be repeated.” “Overall this is a low risk, low cost way of progressively adopting test automation.” “It fits in with classic waterfall processes, and it can also be used in agile processes more easily than other functional test automation suites.”

“TestDrive-Assist cuts the time it takes to manually test in half, with some customers having recorded savings of 75-80 percent” said Colin Armitage, CEO of Original Software. He continued: “This partnership not only provides us with a strong route to market, but allows us to add services to our expanding portfolio. AppLabs’ expertise in testing services and its global, skilled resource pool will be a valued addition to our offering, giving customers the option to outsource projects or business functions or take advantage of specialized consultancy to optimize their IT and business quality management and testing capabilities.”

NEWS: Original Software Positioned as a Visionary in Magic Quadrant for Integrated Software Quality Suites

Original Software, the Application Quality Management (AQM) vendor, today announced that it has been placed in the ‘visionaries’ quadrant by leading analyst firm Gartner, Inc. in its recently published Magic Quadrant for Integrated Software Quality Suites, 31 July 2009.

“The total market for test management and functional and load/stress automation is currently valued at just under $1.2 billion, and is growing at more than 8% per year.” said Thomas Murphy in the report^1. “Testing software can be an expensive process, but poor software quality leads to user dissatisfaction, as well as increased development and maintenance. Therefore, having a well-defined set of tools and practices to drive software quality will positively affect the overall business bottom line.”

The report evaluates fourteen vendors of software quality solutions based on a rigorous set of criteria that comprise ‘completeness of vision’ and ‘ability to execute’. Original Software has been positioned as a visionary.

Colin Armitage, CEO of Original Software, said: “We consider our positioning in the visionaries quadrant to be confirmation of the real value that our approach to solving challenges in the quality process has brought to our users, extending the reach of these solutions within an organization and providing an early return on investment. We are now expanding on the unique work we have done, in reducing script maintenance, supporting manual testing, code-free automation and test data management, with the launch of a radical new ALM solution later this year.”

This report has recently been published by Gartner. More information can be found on the analyst page of Original Software’s website.

NEWS: Original Software offers ‘Escape’ for apprehensive Borland and Compuware customers

Original Software, the Application Quality Management (AQM) vendor, today announced a swap-out promotion for customers of the soon to be engulfed companies, who face uncertainty in the future of their legacy product roadmaps.

Original Software offers market-leading AQM solutions that span the application development life-cycle from requirements, planning and management to complete automated regression testing, including unique solutions for manual testing and test data management.

“Colin Armitage, CEO of Original Software, said: “Any customer on a current maintenance contract with Borland or Compuware can qualify for the Escape program and swap to one of our comparable solutions, provided they switch the maintenance agreement.” The promotion, which runs until the end of the year, includes preferential pricing on all the company’s other complimentary solutions.

“While MicroFocus has now completed the acquisitions of Borland and Compuware, the future for their customers is far from clear. It is inevitable that one of the solutions will take precedence, leaving the other customer base facing an unpleasant choice. You have to feel special sympathy for the users of the Borland testing products for who this is the second change of ownership in less than three years.”

“The great thing about the Escape program is that companies can remove the uncertainty about products futures and join a company whose vision has been widely recognized by industry analysts like Gartner and Ovum. At the same time they can benefit from great customer service and a modern AQM solution suite that addresses traditional automation and the now widely recognized needs for collaboration, manual test support and test data management”.

Good reads that come in 7’s

James Whittaker has recently started a great series of blog posts on the Google Testing Blog – The 7 Plagues of Software Testing.

So far in James’ posts we have encountered The Plague of Aimlessness and The Plague of Repetitiveness.

In The Plague of Aimlessness he asks “Where are the testing spell books? Surely the perilously attained knowledge of our tester forebears is something that we can access in this age of readily available information?” The answer is not. There is a distinct lack of collective knowledge and information sharing within teams. Testers are walking in ever more aimless circles repeatedly suffering ‘the aimless thrashing that we suffered’ already. He urges testers to “Document your successes, scrutinize your failures and make sure you pass on what you learn from this introspection to your colleagues.”

The Plague of Repetitiveness, he argues is caused by just aimlessly ‘doing it’ some more “Developers test but then we retest. We can’t assume anything about what they did so we retest everything. As our product grows in features and bug fixes get applied, we continue our testing. It isn’t long until new tests become old tests and all of them eventually become stale.” He likens it to Boris Beizer’s pesticide paradox. “Pesticide will kill bugs, but spray the same field enough times with the same poison and the remaining bugs will grow immune.”

I’m looking forward to reading more as he posts them.

In a similar vein (we must have been drinking from the same creative juice carton) our very own article – The Seven Deadly Sins of Software Test Automation – is due to be published in the latest edition of T.E.S.T Magazine. In this article we take a light-hearted look at Dante’s Divine Comedy to uncover some home truths about software test automation. We explore each of the seven deadly sins as traps people can fall into because of their earthly vices. I hope you get a chance to read it and would welcome any feedback on it. If you haven’t come across T.E.S.T Magazine already, it is a great read and well worth subscribing.