Skip to content

Feed aggregator

Great Expectations: Digital Transformation and the Path to Unified Enterprise Monitoring

Digital transformation is happening everywhere. It starts with an expectation that whatever an organization does in the real world is supported by—and expands through—its digital presence. As the process continues, digital transformation should drive innovative ways to engage customers, get the most from resources and differentiate its way into completely new markets.

All these great expectations come from both within and outside an organization, and can add up to a big challenge for any team. According to MIT Sloan Management Review, “Preparing for a digital future is no easy task. It means developing digital capabilities in which a company’s activities, people, culture and structure are in sync and aligned toward a set of organizational goals.” (MIT SMR, July 26, 2016)

Meanwhile, your customers get more demanding each day. The applications and environments you use to reach them are super complex, multi-layered and growing. But as long as you can see the details of your system up-time, availability, or the number of help desk tickets opened per month, you’re moving forward—right?

Maybe. Maybe not.

Many organizations invest in technology to monitor some digital indicators. That’s simply not enough anymore. Digital businesses need to look at the whole end-user experience, net promoter scores, JD Power rankings and app store reviews—along with how much revenue comes through digital channels. They also need to act on this information and measure the outcomes of those actions. These are the new SLAs for digital transformation.

You may think that you monitor everything you need to succeed in the new digital world, but how do you know? Are you moving in the right direction? Having the right technology installed is one thing. Making sure that you’re using it to generate the best forward velocity is something else.

Let me give you an example from a friend of mine. We’ll call him “Bob.”

Bob was a bit overweight and certainly not getting any younger. To start some positive change, Bob’s wife gave him a Fitbit for his birthday. One year later, Bob made a proud post to Facebook: “I am happy to report that I’ve worn this fitness device every day since I received it. I know how many steps I take. I know how many calories I burn. Of course, my diet is still mainly cheese and beer, and I just sit here on the couch all day, so, not much has really changed…”

It seems obvious that you can’t simply buy a piece of technology, strap it to yourself, and, then, reap the benefits of weight loss, increased energy, reduced stress—whatever. That takes commitment and willingness to change. It also takes some work.

This isn’t always so obvious in more complex situations.

Take digital transformation, for example. Technology alone won’t get you there. Just having the right monitoring technology strapped to your wrist—or enterprise—won’t guarantee success. Just seeing how many steps you take or how many calories you burn isn’t enough. You have to take action. As MIT Sloan Management Review pointed out: you need to adjust your “activities, people, culture and structure” also. But how?

For Bob to have been successful with his Fitbit, it would have helped if he had committed to a healthy lifestyle; set goals, exercised, eaten healthier—and changed some of his bad habits. The monitoring technology he has provides all the information he needs to get started. Unfortunately, that’s only part of the picture. Moving to best practices based on that information is something else.

Of course, in an organization, changing how people and processes, strategy and culture work together can be truly difficult. For starters, it takes breaking down business silos and—somehow—all sharing the same language and goals.

There is a lot to lose if you don’t.

Every enterprise requires commitment to common goals and shared values. Without such commitment, there is no enterprise; there is only a mob.” – Peter F. Drucker, The Essential Drucker

Changes in culture, strategy and processes are required to be successful in any business initiative—especially ones that involve innovation and operational excellence to achieve the best customer experience.

In my next installment, I’ll explain what an enterprise looks like that’s successful with digital transformation. Then, I’ll show how our experience has helped us to build a proven program that our customers can use to bring together stakeholders and become digital innovators.

The post Great Expectations: Digital Transformation and the Path to Unified Enterprise Monitoring appeared first on Dynatrace blog – monitoring redefined.

Categories: Companies

SonarCfamily For C/C++ Now Plays With The Big Kids

Sonar - Tue, 03/28/2017 - 14:20

Version 4.6 of our SonarCfamily for C/C++ has just been released with a shiny new Buffer Overflow detection mechanism. To get an idea of what bugs we can now spot and why you should read this post, I’ll start with a bug found in the Linux kernel:


We started developing SonarCfamily 4 years ago. It has been tough, and we faced many challenges, but we are finally seeing the result of the huge effort we have invested in it. In the last 12 months, thanks to previous work on accurate semantic analysis and analyzer configuration (thanks to build-wrapper), we were able to start working on a new advanced data flow engine, and deliver many cool rules:

  • S3519 – Memory access should be explicitly bounded to prevent buffer overflows
  • S2259 – Null pointers should not be dereferenced
  • S2583 – Conditions should not unconditionally evaluate to “TRUE” or to “FALSE”
  • S3518 – Zero should not be a possible denominator
  • S836 – Variables should be initialized before use
  • S2095 – Resources should be closed
  • S3588 – Closed resources should not be accessed
  • S3584 – Dynamically allocated memory should be released
  • S3520 – Memory locations should not be released more than once
  • S3590 – Stack allocated memory should not be freed
  • S1232 – Appropriate memory de-allocation should be used
  • S2637 – “nonnull” pointers should not be set to null
  • S3807 – Parameters should match function definition

Here are a few examples of issues found with these rules in real projects:

Data flow analysis is by definition an approximation, and our data flow engine is constantly evolving. In that evolution we follow simple principles to develop the most helpful engine: keep the signal-to-noise ratio as high as possible, do not compromise performance, listen to users’ feedback, and react quickly. We’ve already made good progress, but we have great plans for the future to help you discover the trickiest and nastiest bugs.

But data flow was not the only thing we worked on in the last 12 months. We had 10 (!!!) releases with many other features delivered:

  • Support for SonarLint for Eclipse; you can now now get real time feedback while you code
  • ~130 new rules
  • Intel compiler support on Linux and OS X
  • WindRiver GCC compiler support
  • and more…

As you can see, we now have one of the very best static code analyzers on the market for C and C++. It can be viewed and tested online on sonarqube.com, which offers analysis of open source projects for free. Just get started, no matter what OS and compiler you are using.

For more information about the product, you can visit the product page.

Categories: Open Source

Reinventing Software Testing

Software Testing Magazine - Mon, 03/27/2017 - 19:04
The status of the testing phase and the role of the software tester are challenged and changing as organizations transition from a classical waterfall methodology to software development towards...

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]
Categories: Communities

New Book: The Agile Enterprise: Building and Running your Agile Enterprise

Imagine an enterprise where everyone focuses on the highest customer value.  Where strategy to tasks are visible so everyone knows if their work is aligned with the highest value work. Imagine an enterprise where a discovery mindset wins over certainty thinking. Where experimentation with increments and feedback help define the way toward customer value.

Imagine a company where employees use 100% of their brain power to self organize around the work and be trusted to think of better ways to work. Where leaders encourage employees to put customer value first.  Imagine an enterprise where customers embrace the products and services being built because they are engaged in the building of the work all along the way.

If you can imagine it, it can be yours!  In this unique and cutting edge Agile book, veteran Enterprise Agile Coach Mario Moreira, will guide you to build and run an Agile enterprise at every level and at every point from idea to delivery. Learn how Agile-mature organizations adapt nimbly to micro changes in market conditions and customer needs and continuously deliver optimal value to customers.  Learn cutting-edge practices and concepts as you extend your implementation of Agile pervasively and harmoniously through the whole enterprise for greater customer value and business success.

Readers of The Agile Enterprise will learn how to:
  • Establish a Customer Value Driven engine with an enterprise idea pipeline to process an enterprise’s portfolio of ideas more quickly and productively toward customer value and through all levels of the enterprise
  • Incorporate the Discovery Mindset; experimental, incremental, design, and divergent thinking; and fast feedback loops to increase the odds that what you build aligns more closely to what customer wants.
  • Leverage Lean Canvas, Personas, Story Mapping, Cost of Delay, Discovery Mindset, Servant leadership, Self-organization, and more to deliver optimum value to customers
  • Use continuous Agile Budgeting and enterprise idea pipeline at the senior levels of the enterprise to enable you to adapt to the speed of the market.
  • Reinvent Human Resources, Portfolio Management, Finance, and many areas of leadership toward new roles in the enablement of customer value. 
  • Establish a holistic view of the state of your Agile Galaxy from top-to-bottom and end-to-end allowing you to understand where you are today and where you’d like to go in your Agile future.
  • Be truly Agile throughout the enterprise, focusing on customer value and employees over all else.
This book is geared for: Sponsors of Agile Transformations; Executives and Senior Management; Agile Coaches, Consultants, and Champions; Portfolio Management; Project Management Offices (PMOs); Business and Finance; Human Resources (HR); Investors and Entrepreneurs; Scrum Masters, Agile Project Managers, and Product Owners.  This book concludes with an adventuring through an Agile Enterprise story that shows you how an enterprise may transform to Agile in an incremental manner with materials in this book.  Let the material in The Agile Enterprise help you achieve your successful customer value driven enterprise.

Thank you to the contributors JP Beaudry (on Story Mapping) and David Grabel (on Cost of Delay)!  A special Thank you to the Apress editing team!
Categories: Blogs

A Powerful Case Study on Telerik Analytics: Test Studio

Telerik TestStudio - Sun, 03/26/2017 - 17:47
Telerik Analytics provides powerful tools to help you make data-driven decisions to improve and grow your product. Learn how we used analytics to improve our own software, Test Studio. 2016-06-03T15:46:07Z 2017-03-26T15:26:07Z Gaurav Verma
Categories: Companies

Rodent Controls

Hiccupps - James Thomas - Sun, 03/26/2017 - 12:01

So I wasn't intending to blog again about The Design of Everyday Things by Don Norman but last night I was reading the final few pages and got to a section titled Easy Looking is Not Necessarily Easy to Use. From that:How many controls does a device need? The fewer the controls the easier it looks to use and the easier it is to find the relevant controls. As the number of controls increases, specific controls can be tailored for specific functions. The device may look more and more complex but will be easier to use.  We studied this relationship in our laboratory ... We found that  to make something easy to use, match the number of controls to the number of functions and organize the panels according to function. To make something look like it is easy, minimize the number of controls.How can these conflicting requirements be met simultaneously? Hide the controls not being used at the moment. By using a panel on which only the relevant controls are visible, you minimize the appearance of complexity. By having a separate control for each function, you minimize complexity of use. It is possible to eat your cake and have it, too.Whether with cake in hand, mouth, or both, I would note that easy saying is not necessarily easy doing. There's still a considerable amount of art in making that heuristic work for any specific situation.

One aspect of that art is deciding what functions it makes sense to expose at all. Fewer functions means fewer controls and less apparent complexity. Catherine Powell's Customer-Driven Knob  was revelatory for me on this:
Someone said, "Let's just let the customer set this. We can make it a knob." Okay, yes, we could do that. But how on earth is the customer going to know what value to choose?  As in my first post about The Design of Everyday Things, I find myself drawn to comparisons with The Shape of Actions. In this case, it's the concept of RAT, or Repair, Attribution and all That, the tendency of users to adapt themselves to accommodate the flaws in their technology.

When I wrote about it in The RAT Trap I didn't use the word design once, although I was clearly thinking about it:
A takeaway for me is that software which can exploit the human tendency to repair and accommodate and all that - which aligns its behaviour with that of its users - gives itself a chance to feel more usable and more valuable more quickly.Sometimes I feel like I'm going round in circles with my learning. But so long as I pick up something interesting - a connection, a reinforcement, a new piece of information, an idea - frequently enough I'm happy to invest the time.
Image: https://flic.kr/p/dewUvv
Categories: Blogs

Implication of emphasis on automation in CI

Thoughts from The Test Eye - Sat, 03/25/2017 - 16:49
Ideas

Introduction
I would believe, without any evidence, that a majority of the test community and product development companies have matured in their view on testing. At conferences you less frequently see the argumentation that testing is not needed. From my own experience and perceiving the local market, there is often new assignments for testers. Many companies try to hire testers or get in new consulting testers. At least looking back a few years and up until now.

At many companies there is an ever increasing focus and interest in Continuous Deployment. Sadly, I see troublesome strategies for testing in many organisations. Some companies intend to focus fully on automation, even letting go of their so called manual testers. Other companies focus on automation by not accepting testers to actually test and explore. This troubles me. Haven’t testers been involved in the test strategy? Here are few of my pointers, arguments and reasoning.

Automation Snake oil
In 1999 James Bach wrote the article Automation Snake Oil [see reference 1], where he brings up a thoughtful list of arguments and traps to be avoided. Close to 17 years later, we see the same problems. In many cases they have increased because of the Continuous Deployment ideas, but also because of those from Agile development. That is, if you ignore all the new ideas gained in the test domain as well as all research done.
The miracle status of automation is not a new phenomenon, together with the lure of saving time and cost it is seducing. In some cases it will probably be true, but it is not a replacement of thinking people. Instead it could be an enabler for speed and quality.

Testing vs. Checking
In 2009, Michael Bolton wrote an article that clarified a distinction between Testing and Checking. Since then the definition has evolved. The latest article Testing vs. Checking Refined [see reference 2] is the last in the series. Most of the testers I know and that I debate with are aware of this concept and agree with the difference or acknowledge the concept.

If you produce test strategies in a CI-environment that put an emphasis on automation, and if it means mostly doing checking and almost no testing (as in exploration), then you won’t find the unexpected. Good testing include both.

Furthermore when developing a new feature, are you focusing on automating checks fulfilling the acceptance criteria or do you try to find things that have not been considered by the team? If you define the acceptance criteria, then only check if that is fulfilled. It will only enable you to reach a small part of the way toward good quality. You might be really happy how fast it goes to develop and check (not test) the functionality. You might even be happy that you can repeat the same tests over and over. But I guess you failed to run that one little test that would have identified the most valuable thing.

Many years ago a tester came to me with a problem. He said, “We have 16000 automated tests, still our customers have problems and we do not find their problems”. I told him that he might need to change strategy and focus more on exploration. Several years later another tester came to me with the same problem, from the same product and projects. He said, “We have 24000 automated tests, still our customers have problems and we do not find their problems!”. I was a bit surprised that the persistence in following the same strategy for automation while at the same time expecting a different outcome.

In a recent argument with a development manager and Continuous Deployment enthusiast. They explained their strategy and emphasis on automation. They put little focus on testing and exploration. Mostly hiring developers who needed to automate tests (or rather checks). I asked how they do their test design? How do they know what they need to test? One of my arguments was that they limited their test effort based on what could be automated.

We know that there is an infinite amount of tests. If you have done some research, you have an idea what different stakeholders value and what they are afraid will happen. If that is so, then you have an idea what tests would be valuable to do or which areas you wish to explore. Out of all those tests, you probably only want to run part of these tests only once, where you want to investigate something that might be a problem, learn more about the systems behavior or try a specific, very difficult setup or configuration of the system. This is not something that you would want to automate because it is too costly and it is enough to learn about it just once, as far as you know. There are probably other tests that you want to repeat, but most probably with variation in new dimensions, and do more often. It could be tests that focus on specific risks or functionality that must work at all times. Out of all those that you actually want to test several times, a part of those you plan and want to automate. Out of those that you have planned to automate, only a fraction can be automated. Since automation takes a long time and is difficult, you have probably only automated a small part of those.

If you are a stakeholder, how can you consider this to be ok?

Rikard Edgren visualized the concept of what is important and what you should be in focus in a blog post called “In search of the potato” [see reference 3].

His main points are that the valuable and important is not only in the specification or requirements, you need to go beyond that.

Another explanation around the same concept of the potato is that of mapping the information space by knowns and unknowns.

The majority of test automation focus on checking an aspect of the system. You probably want to make repeatable tests on things that you know or think you know, thus the Known Knowns. In making this repeatable checking you will probably save time in finding things that you thought you knew, but that might change over time by evolving the system, thus evaluating the Unknown Knowns. In this area you can specify what you expect, would a correct result would be. With limitation on the Oracle problem, more on that below.

If you are looking beyond the specification and the explicit, you will identify things that you want to explore and want to learn more about. Areas for exploration, specific risks or just an idea you wish to understand. This is the Known Unknowns. You cannot clearly state your expectations before investigating here. You cannot, for the most part, automate the Known Unknowns.

While exploring/testing, while checking or while doing anything with the system, you will find new things that no one so far had thought of, thus things that fall into the Unknown Unknowns. Through serendipity you find something surprisingly valuable. You rarely automate serendipity.

You most probably dwell in the known areas for test automation. Would it be ok to ignore things that are valuable that you do not know of until you have spent enough time testing or exploring?

The Oracle Problem
A problem that is probably unsolvable, is that there are none (or at least very few) perfect or true oracles [see reference 4, 5, 6].

A “True oracle” faithfully reproduces all relevant results for a SUT using independent platform, algorithms, processes, compilers, code, etc. The same values are fed to the SUT and the Oracle for results comparison. The Oracle for an algorithm or subroutine can be straightforward enough for this type of oracle to be considered. The sin() function, for example, can be implemented separately using different algorithms and the results compared to exhaustively test the results (assuming the availability of sufficient machine cycles). For a given test case all values input to the SUT are verified to be “correct” using the Oracle’s separate algorithm. The less the SUT has in common with the Oracle, the more confidence in the correctness of the results (since common hardware, compilers, operating systems, algorithms, etc., may inject errors that effect both the SUT and Oracle the same way). Test cases employing a true oracle are usually limited by available machine time and system resources.
Quote from Douglas Hoffman in A taxonomy of Test Oracles [see reference 6].

Here is a the traditional view of a system under test is like the figure 1 below.

In reality, the situation is much more complex, see figure 2 below.

This means that we might have a rough idea about the initial state and the test inputs, but not full control of all surrounding states and inputs. We get a result of a test that can only give an indication that something is somewhat right or correct. The thing we check can be correct, but everything around it that we do not check or verify can be utterly wrong.

So when we are saying that we want to automate everything, we are also saying that we put our trust in something that is lacking perfect oracles.

With this in mind, do we want our end-users to get a system that could work sometimes?

Spec Checking and Bug Blindness

In an article from 2011, Ian McCowatt expresses his view on A Universe of behavior connected to Needed, Implemented and Specified based on the book Software Testing: A Craftsman’s Approach” by Paul Jorgensen.

For automation, I would expect that focus would be on area 5 and 6. But what about unimplemented specifications in area 2 and 3? Or unfullfilled needs in area 1 and 2? Or unexpected behaviors in area 4 and 7? Partly undesired behaviors will be covered in area 6 and 7, but enough?

As a stakeholders, do you think it is ok to limit the overall test effort to where automation is possible?

Concluding thoughts

It seems like we have been repeating the same things for a long time. This article is for those of you who are still fighting battles against strategies for testing which state automate everything.

References

  1. Test Automation Snake Oil, by James Bach – http://www.satisfice.com/articles/test_automation_snake_oil.pdf
  2. Testing and Checking Refined, by James Bach & Michael Bolton – http://www.satisfice.com/blog/archives/856
  3. In search of the potato, by Rikard Edgren – http://thetesteye.com/blog/2009/12/in-search-of-the-potato/
  4. The Oracle Problem and the Teaching of Software Testing, by Cem Kaner - http://kaner.com/?p=190
  5. On testing nontestable programs, by ELAINE J. WEYUKER – http://www.testingeducation.org/BBST/foundations/Weyuker_ontestingnontestable.pdf
  6. A Taxonomy for Test Oracles, by Douglass Hoffman – http://www.softwarequalitymethods.com/Papers/OracleTax.pdf
  7. Spec Checking and Bug Blindness, by Ian McCowatt – http://exploringuncertainty.com/blog/archives/253

Categories: Blogs

Automating Shared Infrastructure Impact Analysis: Why Monitoring Backend Jobs is as important as monitoring applications

This posting illustrates how to effectively automate shared infrastructure analysis to support both your back-end jobs and your applications.

Last week Gerald, one of our Dynatrace AppMon super users, sent me a PurePath as part of my Share Your PurePath program. He wanted to get my opinion on high I/O time they sporadically see in some of their key transactions on their Adobe-based documentation management system. The hotspot he tried to understand was easy to spot in the PurePath for one of the slow transactions: Creating a local file took very long!

createFileExclusively takes up to 18s in one of their key document management system transactions

In order to find out which file takes that long to create I asked Gerald to instrument File.createNewFile. This was now capturing the actual file name for all transactions. Turned out we are talking about files put into a temporary directory on the local D: drive.

Instrumenting createNewFile makes it easy to see which files were created across many transactions that got executed

Now – this itself didn’t explain why creating these files in that directory was slow. As a next step, we looked at process and host metrics of that machine delivered by the Dynatrace Agent. I wanted to see whether there is anything suspicious.

The Process Health indicated that there is some very aggressive memory allocation going on within that JBoss. Multiple times a minute the Young Generation Heap spikes to almost 5GB before Garbage Collection (GC) kicks in. These spikes also coincide with high CPU Utilization of that process – which makes sense because the GC needs to clean up memory and that can be very CPU intensive. Another thing I noticed was a constant high number of active threads on that JBoss instance which correlates with the high volume of transactions that are actively processed:

Process Health metrics give you a good indication on whether there is anything suspiciously going on, such as strange memory allocation patterns, throughput spikes or problems with threading.

Looking at the Host Health Metrics just confirmed what I’ve seen in the process metrics. CPU spikes caused by high GC due to these memory allocations. I blamed the high number of Page Faults on the same memory behavior. As we deal with high Disk I/O I looked at the disk metrics – especially for drive D:. Seems though that there was no real problem. Also, consulting with the Sys Admin brought no new insights.

Host Health metrics are a good way to see whether there is anything wrong on that host that potentially impacts the running applications, e.g.: constraint on CPU, Memory, Disk, or Network.

As there was no clear indication of a general I/O issue with the disks I asked Gerald to look at file handles for that process and on that machine. Something had to block or slow down File I/O when trying to create files in that directory. Maybe other processes on that same machine that are currently not monitored by Dynatrace or some background threads on that JBoss is having too many open file handles leading to the strange I/O waiting times of the core business transactions?

Background Scheduler to be blamed!

Turned out that looking into background activity was the right path to follow! Gerald created CPU Sample using Dynatrace on that JBoss instance. Besides capturing PurePaths for active business transactions it is really handy that we can also just create CPU Samples, Thread Dumps or Memory Dumps for the process that has a Dynatrace AppMon Agent injected.

The CPU sample showed a very suspicious background thread. The following screenshot of the Dynatrace CPU Sample highlights that one background thread is not only causing high File I/O through the File.delete operation. It also causes high CPU through Adobe’s WatchFolderUtils.deleteMarkedFiles method which ultimately deletes all these files. Some “Google’ing” helped us learn that this method is part of a background job that iterates through that temporary directory on D. The job tries to find files that match a certain criterion, marks them, and eventually deletes them.

The CPU Sample helped identify the background job that causes high CPU as well blocked I/O access to that directory on drive D

A quick chat with the Adobe administrator resulted in the following conclusions:

  1. The File Clean Task is scheduled to run every minute – probably too frequent!
  2. Very often this task can’t complete within 1 minute which leads to a backup and clash of the next clean up task
  3. Due to some misconfiguration, the cleanup job didn’t clean up all files it was supposed to. That lead to many “leftovers” which had to be iterated by the Watch Utility every minute leading to even longer task completion time.

The longer the system was up and running, the more files were “leftover” in that folder. This led to even more impact on the application itself as file access to that folder was constantly blocked by that background job.

Dynatrace: Monitoring redefined

While this is a great success story it shows that it is very important to monitor all components that your applications share infrastructure with. Gerald and his team were lucky that this background process job was actually part of the regular JBoss instance that ran the application and that it was monitored with a Dynatrace AppMon Agent. If the job would be running in a separate process or even on a separate machine it would be harder to do root cause analysis.

Exactly for that reason we extended the Dynatrace monitoring capabilities. Our Dynatrace OneAgent is not only monitoring individual applications but additionally automatically monitors all services, processes, network connections and its dependencies on the underlying infrastructure.

Dynatrace automatically captures full host and process metrics through its OneAgent technology

Applying artificial intelligence on top of that rich data allows us to find such a problem automatically without having experts like Gerald or his team perform forensic steps.

The following screenshot shows how Dynatrace packages such an issue in what we call a “Problem Description” including the full “Problem Evolution”. The Problem Evolution is like a film strip where you can see which data points Dynatrace automatically analyzed to identify that this is part of the root cause. The architectural diagram also shows you how your components relate and cross impact each other until they impact the bottom line: which is the end user, performance or your SLAs.

Dynatrace automates problem and impact analysis and allows you to “reply” Problem Evolution to better understand how to address the issue.

If you are interested to learn more about Dynatrace, the new One Agent as well as Artificial Intelligence simply try it yourself. Sign up for the SaaS-based Dynatrace Free Trial.

If you want explore Dynatrace AppMon go ahead and get your own Dynatrace AppMon Personal License for your On-Premise installation.

The post Automating Shared Infrastructure Impact Analysis: Why Monitoring Backend Jobs is as important as monitoring applications appeared first on Dynatrace blog – monitoring redefined.

Categories: Companies

4 Ways to Combat Android Fragmentation in QA

Testlio - Community of testers - Fri, 03/24/2017 - 19:32

There are more than 24,000 different Android smartphones on the market today. For developers, this is a major headache.

iPads and iPhones come in various sizes and models, but they have a sole distributor. With Android, on the other hand, each manufacturer  tweaks the OS slightly to fit with different offerings that differentiate their brand, such as auditory or motion sensors or a curved screen.

Android fragmentation poses a challenge for developers and for QA teams: How can they deliver quality on all of their users’ devices?

Android was built to encourage device diversity and innovation. Google likes to remind developers that without its widely available, customizable OS manufacturers would each have to make their own OS, slowing hardware innovation and increasing costs in the process.

Android device coverage: the conventional wisdom

Android smartphones represented 85% of global mobile phone sales in Q3 of 2015 and 88% of sales in Q3 of 2016. With an ever growing market share, Android entices more and more developers to expand their apps beyond iOS.

With upwards of 24,000 devices, Android developers are forced to make tough decisions about testing coverage. Clearly, not every device and OS version can be tested. Even so, not every device that is tested can be tested for every use case.

It’s just too time consuming.

Teams will often test the most popular 80% or 90% of devices owned by their users, that way they get peace of mind without testing everything on the market.

If your team has struggled to identify what those devices should be, or has wondered if a higher or lower percentage is more fitting, I’m going to get into four points to consider for any device coverage strategy. But first, a word of caution…

The problem with ignoring fragmentation

Developers can be tempted to ignore Android fragmentation at any time. QA consultant Joe Schultz details working with a large bank who had chosen to test on just two Android devices when rolling out an update to its mobile app.

They went with just one Samsung model and one Motorola. Sounds a little crazy and a little risky, but even large developers often test on under 10 devices.

The bank was flooded with poor ratings that impacted their original app rating. After upping the test coverage to 200 hardware and OS combinations, the bank was able to identify improvements, roll them out in a subsequent upgrade and eventually get their app store ratings back to original standings (and halt the complaints received via social media).

Teams regularly have to decide how much flack they’re willing to risk in case there are major issues with the devices they’ve left untouched.

Combatting Android fragmentation during QA

Because the stakes are so high, teams have to be incredibly strategic when choosing which device and device-OS version combinations they will cover. You have to cater to your own user base, keep an eye on potential growth, and balance risk tolerance with deadlines. Here are four key ways to confidently narrow your Android app testing plan.  

  1. Analyzing your own data for OS version, device usage, and phone versus tablet

When strategizing device coverage for an existing application, nothing is more important than your own usage analytics.

Here’s the data you should be collecting and analyzing:

  • Operating system: do your users typically upgrade to the newest version of Android, or do you need to support past versions? If so, how far back?
  • Devices and brands: which devices are commonly used? Are your users on the most current versions? If there are multiple versions of one device, how similar or different are those versions and can they be combined?
  • Phone versus tablet: If your app has been designed for use on a phone and on a tablet, what is the breakdown of use? How much of the device coverage should be allocated to phones versus tablets?

When you know exactly how many users you risk by excluding KitKat but covering LolliPop and Marshmallow, you can feel more confident in accepting the risk and narrowing your testing combinations (thus reducing overwhelm).

  1. Making choices based on geographical distribution & market data

Teams will also find it useful to analyze general market data to give insight into the growth of their product and the type of user who might be adopting it in the near future.

This can really come into play with Android OS versions and device manufacturers. Your users may differ slightly from your market, meaning you don’t want to exclude new popular phones or old OSes that your future users might rely on.

Depending on the markets they serve, some developers may need to test on low-end devices that are not officially compatible with Android and yet run on it anyways. This could require inside knowledge about devices whose popularity isn’t well tracked.

  1. Covering manufacturers of specific hardware

For some teams, testing the right hardware isn’t just about the brand of the phone, but the manufacturer of units inside the phone. Gaming company Pocket Gems makes sure to cover high and low resolution devices and to test on devices using each of the five major graphical processing units (GPUs).

So, depending on the nature of the app, device coverage may include underlying capabilities rather than just brands and their popular offerings.

  1. Using combinatorial testing and sampling to narrow test cases

With a better understanding of device, OS version, and hardware needs of their user base and overall market, teams still have to reach a level of confidence in how they combine these requirements together with test cases.

The combinations could result in hundreds of thousands of unique test cases if not kept in check.

That would be bad math. But good math—namely analysis, sampling, and combinatorial calculations—allows for the reduction of test cases.

Luckily, there are combinatorial tools that can do the math for us. QA engineers can plug in parameters like device type, communication network, software and IoT device connections to produce statistically-derived combinations.

Randomization and sampling can also help lessen QA overwhelm when it comes to combining devices, OSes, and test cases.

Once combinations are identified, the automation of repeatable scripts and the use of device emulators can help make coverage more realistic, though manual testing is still a requisite as those strategies can’t fully mimic real user behavior.

Every QA team needs to cover the most critical user journeys in the most important devices. Fragmented platforms like Android don’t make this easy, but you can still deliver an amazing customer experience — all it takes is a little strategy.

With a global team of skilled, experienced testers, Testlio handles Android device testing and maximizes QA budgets for popular apps loved around the world. Get in touch for a customized solution to Android application testing.

The post 4 Ways to Combat Android Fragmentation in QA appeared first on Testlio.

Categories: Companies

FileInputStream / FileOutputStream Considered Harmful

Suricata suricatta -Auckland Zoo -group-8a (with captions added by Stephen Connolly)

Ok, so you have been given an array of bytes that you have to write to a file. You’re a Java developer. You have been writing Java code for years. You got this:

public void writeToFile(String fileName, byte[] content) throws IOException {
  try (FileOutputStream os = new FileOutputStream(fileName)) {
    os.write(content);
  }
}

Can you spot the bug?

What about this method to read the files back again?

public byte[] readFromFile(String fileName) throws IOException {
  byte[] buf = new byte[8192];
  try (FileInputStream is = new FileInputStream(fileName)) {
    int len = is.read(buf);
    if (len < buf.length) {
      return Arrays.copyOf(buf, len);
    }
    ByteArrayOutputStream os = new ByteArrayOutputStream(16384);
    while (len != -1) {
      os.write(buf, 0, len);
      len = is.read(buf);      
    }
    return os.toByteArray();
  }
} 

Spotted the bug yet?

Of course the bug is in the title of this post! We are using FileInputStream and FileOutputStream.

So what exactly is wrong with that?

Have you ever noticed that FileInputStream overrides finalize()? Same goes for FileOutputStream by the way

Every time you create either a FileInputStream or a FileOutputStream you are creating an object, even if you close it correctly and promptly, it will be put into a special category that only gets cleaned up when the garbage collector does a full GC. Sadly, due to backwards compatibility constraints, this is not something that can be fixed in the JDK any time soon as there could be some code out there where somebody has extended FileInputStream / FileOutputStream and is relying on those finalize() methods ensuring the call to close().

Now that is not an issue for short lived programs… or for programs that do very little file I/O… but for programs that create a lot of files, it can cause issues. For example, Hadoop found “long GC pauses were devoted to process high number of final references” resulting from the creation of lots of FileInputStream instances.

The solution (at least if you are using Java 7 or newer) is not too hard - apart from retraining your muscle memory - just switch to Files.newInputStream(...) and Files.newOutputStream(...)

Our code becomes:

public void writeToFile(String fileName, byte[] content) throws IOException {
  try (OutputStream os = Files.newOutputStream(Paths.get(fileName))) {
    os.write(content);
  }
}

public byte[] readFromFile(String fileName) throws IOException {
  byte[] buf = new byte[8192];
  try (InputStream is = Files.newInputStream(Paths.get(fileName))) {
    int len = is.read(buf);
    if (len < buf.length) {
      return Arrays.copyOf(buf, len);
    }
    ByteArrayOutputStream os = new ByteArrayOutputStream(16384);
    while (len != -1) {
      os.write(buf, 0, len);
      len = is.read(buf);      
    }
    return os.toByteArray();
  }
} 

A seemingly small change that could reduce GC pauses if you do a lot of File I/O!

Oh and yeah, we’re making this change in Jenkins

Blog Categories: Developer Zone
Categories: Companies

Quality Software Australia Conference 2017

My Load Test - Fri, 03/24/2017 - 06:51
I will be speaking at the upcoming Quality Software Australia conference in Melbourne on May 11, 2017. Those who plan to attend the conference can look forward to presentations from local and international thought-leaders on devops/QAops, CI, testing microservices, test automation, and other topics of interest for people who care about software quality. My presentation […]
Categories: Blogs

New Swag: MyLoadTest USB sticks

My Load Test - Fri, 03/24/2017 - 05:52
I have just placed an order for a large number of 8GB USB 3 memory sticks with a MyLoadTest logo on them. The USB sticks are available for clients, and for performance testers who help us out by finding bugs, by contributing to the web performance community, or by referring work to MyLoadTest. Don’t forget […]
Categories: Blogs

The State of Jenkins - 2016 Community Survey

This is a guest post by Bhavani Rao, Marketing Manager at CloudBees Last fall, prior to Jenkins World, CloudBees conducted a Community Survey. We received over 1200 responses, and thanks to this input, we have some interesting insights into how Jenkins users and their use of Jenkins are evolving. Based on the survey’s results, Jenkins is increasingly being used to support continuous delivery (CD). Adoption of Jenkins 2, which featured "Pipeline as code" and encouraged users to adopt Jenkins Pipeline, has skyrocketed to more than half of all Jenkins installations. Other data remained consistent with findings year-to-year, for example, the number of Jenkins users continues to increase and 90% of...
Categories: Open Source

Final Posting

Rico Mariani's Performance Tidbits - Thu, 03/23/2017 - 22:54

My last day at Microsoft will be tomorrow, 3/24/2017.

I really did want to get to some of the old comments that had been neglected but alas, there’s no time.

I’m not sure I will actually lose access to this blog because of the way authorization happens but I think it would be a bad idea for me to keep using it because of its strong affiliation with MS.

Best wishes to all and thanks for reading all these years.

-Rico

@ricomariani

Categories: Blogs

Test Data: Food for Test Automation Framework

Testing TV - Thu, 03/23/2017 - 20:57
Building a Test Automation Framework is easy – there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey. Teams already building 1000s of tests of various types – UI, web service-based, integration, unit, etc. is a proof of […]
Categories: Blogs

Debriefing Exploratory Testing Session Notes

Software Testing Magazine - Thu, 03/23/2017 - 20:22
Session notes that you use when you adopt an exploratory testing approach can be used to capture more than bugs. They not only serve as a memory of a bug, but a structured testing and learning method...

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]
Categories: Communities

Ranorex Object Repository

Ranorex - Thu, 03/23/2017 - 18:25
Categories: Companies

Worksoft Partners with IBM to Support SAP Testing

Software Testing Magazine - Thu, 03/23/2017 - 18:05
Worksoft, a provider of automation software for high-velocity business process discovery and testing, has announced a new collaboration with IBM to offer a unified testing approach for SAP projects....

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]
Categories: Communities

Vector Software Introduces VectorCAST/Probe

Software Testing Magazine - Thu, 03/23/2017 - 17:49
Vector Software, a provider of software solutions for embedded software quality, has announced VectorCAST/Probe. VectorCAST/Probe provides a simple way to dynamically instrument an application with...

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]
Categories: Communities

Ranorex 7.0

Ranorex - Thu, 03/23/2017 - 17:26

Be among the first to experience the matchless Selenium WebDriver integration, WPF object recognition and test suite improvements available with Ranorex 7.0.
Release Notes

The post Ranorex 7.0 appeared first on Ranorex Blog.

Categories: Companies

Knowledge Sharing

SpiraTest is the most powerful and affordable test management solution on the market today