You can find out how these in-demand disciplines intersect with the DevOps with Cloud Computing course bundle, available right now for less than $ 20 with coupon code “USA40,” an over 90 percent savings, from TNW Deals.
The Next Web
Note: No actual spoilers in this post
We have all been waiting and it’s finally here: Star Wars’ “The Last Jedi”.
I promise to only use the new Star Wars movie theme (plus a few Epic events and moments from the Star Wars trilogy) to express my excitement and correlate a story from the DevOps Galaxy. As a huge Star Wars fan, it seems only natural that I am highly influenced by this brilliant trilogy but in many ways, the force awakens and the last Jedi feel like an easy comparison to where our industry is heading at the end of 2017.
Recap: There is a disruptive force AKA digital that is forcing Dev & QA teams to move at the speed of light, reduce release cycles and shift teams left, forming the DevOps galaxy and the birth of the Dev/Test Jedi.
Similar to the way Jedi’s look for the balance between light and darkness. This article is based on ideals of helping DevOps teams find the balance of their testing activities and fulfil their Destiny as Jedi dev/test masters while keeping them away from their old tendencies that we would now be considered as the dark side….
Jedi tips to follow (small hint – I will also try to prove that it is really not the time for the Jedi to end… or in other words – there should be a happy ending).
Use the Force (Wisely…)
I recently met with a customer who is a Director of QA, that was describing the revelation of automation and CI exactly in the same words Rey was describing the feel of the Force in the movie: “Felt something… it awakened, but now I need to know how to wield it”.
To that Director of QA, the power of automation was clear, but now he needed to prove it to his senior management (in order to actually implement the process and be able to use it for his testing purposes). So his natural question, just like Rey is asking Luke Skywalker in the new movie “I need someone to show me my place in all of this”.
Dev/Test Jedi Pro Tip: You should pick one significant testing scenario (let’s take “Sanity” activity as an example) and prove that you can cross the entire galaxy to provide value with automation. The success metrics here would usually be time (which is also power according to Master Yoda as we all know):
You’ll be able to prove that the Sanity testing cycle can be shorter in x hours / days with the move to automation.
You’ll be able to prove the feedback loop cycle is faster and that issues are identified earlier in the process.
These success evidences should prove a clear value proposition when it comes to test automation and CI: “Do. Or do not. There is no try.”
“I am a Jedi. I’m one with the Force, and the Force will guide me”
According to all Analyst agencies, the strongest trend (or ‘Force’ if you like) expected in 2018 is the transformation of data into actionable insights. In other words, when big data is becoming a challenge you really need a guidance throughout your DevOps journey.
I define this force as VISIBILITY. Accordingly, It would make sense to assume that analysis (as part of test automation) shouldn’t take light years…. Fast feedback should be relying on a solid foundation of smart reporting and analytics capabilities that will assist you in identifying risk quickly and effectively.
As part of my journey in the galaxy to drive more smart data into the testing universe, I meet with many organizations that are trying to move from the dark side. A survey conducted by Perfecto revealed that 40% of organizations spend between 4-8 working hours to analyze regression test suites post execution. If this suite is running multiple times a day, the analysis process is not scalable.
Many enterprises are struggling today with managing processes of manual analysis and maintenance of reports/dashboards (like manually typing of data into an excel sheet or HTML page… cell after cell… test after test).
Dev/Test Jedi Pro Tip: Adding more man power to boost manual report/dashboard generation is old news. Adopt a Reporting Test Driven Development approach (add all contextual data like tags, steps, BDD and more) and tie it to a centralized tool to gain unified quality view.
Never tell me the Odds! (Really?!)
With DevOps aspirations in mind, and unlike Han Solo states, you should always think about the odds: your odds to release a stable version. Your odds to release on time. Your odds to find root issues quickly enough before they spill over to the next phase of your pipeline.
I would say that the DevOps play is all about odds!
Dev/Test Jedi Pro Tip: Focus your Continuous Integration management efforts to estimate whether you are heading in the right direction or not. Trending history visibility when running in CI is the way to achieve that goal. Make sure to also clean any flakiness noise (e.g. don’t execute tests that you know are going to fail/tests with known bugs) in order to maintain a reliable picture (the “Always Green approach).
As Obi-Wan-Kenobi stated: “In My Experience there is no such thing as luck”. We at Perfecto understand that and believe that it is all about establishing a DevOps process that is built on trust:
- Add only stable test scenarios to your CI. Take out noise (unstable tests) in order to make sure the data is useful for all (Mature DevOps brings cross-team collaboration and you to make sure you establish a trustworthy testing process.)
- Build the right visibility to collect the large amount of testing data and have an effective triaging process throughout your pipeline. One dashboard that shows all your data.
- As DevOps involves high frequency releases, make sure to have the right traceability of build stability and build the right visibility to support quick release decisions without the hassle of collecting the data again and again (have it ready for you to back up any open question).
Master Yoda made it very clear: “Adventure. Excitement. A Jedi craves not these things”. Like a Jedi, the new Dev/Test Jedi should not take unnecessary risks (as their duty is to eliminate risk and achieve stability). Keep things basic and establishing a routine is an important step in order to maintain an effective DevOps process that allows you to identify anomalies quickly and effectively.
May the Force Be With You.
Bug handling in DevOps can be the make-it or break-it factor- because of this, I have gathered some of the best tips and practices to treat your bugs.
1. Triage Everything
Investigate every bug to identify the root cause – there might be other bugs underneath
Fixing bugs fast is essential for healthy automation, especially as part of today’s mature DevOps processes. That said, it is also important to make sure there is no other issue hiding below that bug. Sometimes an immediate fix of a small bug with low priority may expose a bigger issue. In some cases, developers will ignore the low priority bug, without knowing there is a critical issue below. Like a triage process in the ER, only full examination will allow understanding of chronic issues, rather than being misled by specific symptoms.
2. Bugs are Actually EVERYONE’s Responsibility to Report and Fix Properly
A true DevOps process is all about people, processes and tools. Accordingly, all parties should be engaged and involved in order to build a reliable defect management process
I often hear about the classic conflicts, where an enterprise QA team claims that Dev is checking buggy code with questions like, “Why do we need to report new bugs if we still have so many unfixed bugs?”.
On the other hand, Dev teams may claim that bug reports are not comprehensive, as they lack important context and request them to be reproduced and investigated all over again. I recently got some feedback from a Dev Executive saying she asked to delete 4000+ bugs in the corporate Jira instance, as everyone just lost track and no one really resolved them.
Of course, the impact affects the entire organization. If this healthy process will not take place, the trust around defect management may be lost, and the aim for a high quality product may be affected.
3. Don’t tell stories about a bug: it’s either a bug or a story…
A “Zero-bug” approach is too dramatic, yet if there’s a bug that is not that urgent for an immediate fix – prioritize it as a user story as part of your backlog.
There are different events that were created over the years as part of agile methodologies, to go over bugs: Sprint/sprint plan, triage/defect review meetings, daily/weekly prioritization meetings, etc.
The purpose of many of these events is to categorize bugs on different levels or metrics: P 1,2,3 / important vs. urgent / business vs. tech impact / prioritization based on the estimated time to resolve bugs, etc.
Overanalyzing these questions, disturbs you from the actual discussion. Don’t waste time, either fix bugs as soon as you find them or transform them into user stories for Dev to deal with in the close future as part of the backlog priorities.
There are always bugs that can be referred to as “code red” bugs (and they usually take everyone’s attention). But… there are ONLY a few of them that truly require this level of attention. The rest create some noise if not treated properly (just like with the story about the boy who cried Wolf…).
4. Open Bug? – Ignoring it again and again is not the right approach
Clean noise out of your CI: if a test fails and you’ve submitted a bug for this failure – either fix it immediately or take it out of your execution cycle until it is being fixed
As the number of your test executions grow, you cannot keep remembering which bugs were not fixed yet and which ones are actually new. The greater the number of unfixed bugs, the harder it is to identify whether a bug has already been reported.
Automation at scale will either require you to invest significant time for this unnecessary discovery process (time you could have better spent on more important bug-related issues) or you’ll just give up due to the huge effort required to maintain this thing. On both options, you should expect some bugs that will slip through and cause duplicated work as they are reported again. The long trail of actions related to this duplication will cause bigger waste that will include more triaging, investigation, discovery of existing bugs vs. new ones and more.
So either fix immediately or comment out your test until the reported bug is being fixed. The best approach to handle this is either to integrate your framework or your reporting environment to your used defect management for full trace ability of bugs and their status.
5. Bugs cannot be lost in translation
Establish a data-driven discussion in order to shorten the fix process. Partial descriptions and missing data will cause delays, as Dev will be required to walk through the same process again
“Build #296 is on fire” would be the best way to describe the bug you’ve found as it wouldn’t serve as a good Segway for developers to start their debugging work. Remember to state the exact environment conditions of the executed test to make sure that Dev can debug under the same conditions. We see many cases of a loop between Dev and QA, where a tester submits a bug and a tester claims he cannot reproduce it and just closes the case. Until the peers share data, they don’t understand that the tester cannot reproduce over an emulator with the latest iOS version something that was found over an iOS device with an older version of iOS.
Creating a fast feedback loop that is relying on cross-team collaboration is one of today’s DevOps essentials, key for success. Aligning everyone over the same test execution report would be the recommended way here. When submitting a new bug, make sure it is linked to a report, this is the best evidence to show everyone what actually happened and the best “meeting point”.
6. Use the same metrics to measure bugs resolution
The Bug fixing process must be aligned with release cadence in CI. Pushing bug fixing to the right (e.g. to later stages of your SDLC) may create bottlenecks that will affect your quality or release.
Different teams analyze bugs in different ways: total number of bugs, number of bugs per sprint/release (density), bug lifespan, etc.
Taking aside the urgency or complexity of each case, it is clear that every bug-based metric relies on different calculation/policy. It is likely correct to assume that the number of unfixed bugs increases. With that said, you should expect an increasing challenge to maintain accurate bug status.
In continuous integration and even continuous deployment reality, these assumptions or ‘snapshots’ about the bug status – may change in a matter of minutes/hours. Merging this data to your release decision process may create an inaccurate baseline and put your quality at risk (as bugs may slip or stall development process).
7. Inject intelligence into your Defect Management Process
Correlate wisely bugs to tests, features and environments so that you have a continuous improvement of your entire portfolio
DevOps methodology requires executives to identify risks through different metrics such as teams, features, stage and business-related items. Having the ability to tag, correlate and act upon bugs as they relate to the above attributes will help optimize the entire workflow, and support a maturity leap forward in DevOps practices. As an example, a specific team that has more bugs in a specific stage compared to a different team, raises few flags that if highlighted and addressed can serve as an alignment factor for the entire R&D. Another example might be a specific feature that is more error-prone on specific OS versions compared to others – the outcome of such insights shall drive more test coverage and attention to that piece.
DevOps is all about the right balance of people, processes and tools. Problems will always pop-up between these components. With that in mind, problem solving (bugs are kind of a problem, right?) is fundamental for a healthy agile process.
The process that you build in order to treat bugs as part of automation at scale should be designed with the following ideas in mind:
- Provide the right visibility of bugs (when identified)
- Sufficient enough in order to fix them fast.
- Triaging is essential to determine where to put focus in CI/CD.
- Communicate quality findings with regards to bugs across teams for full alignment
- Measure bug fixing process using unified metrics
- Trust – a key factor in order to establish a sustainable process between teams which eventually leads to a stage where everyone take responsibility on bugs.
When the two sectors of DevOps and cloud computing get together, it’s how IT REALLY happens in the 21st century. You can get prepared right now to function in these in-demand job arenas with the Ultimate DevOps and Cloud Computing bundle. For a limited time, this package is only $ 41.65 from TNW Deals with coupon code “GIFTSHOP15.”
The Next Web
To state that DevOps and IT operations teams will face new challenges in the coming years sounds a bit redundant, as their core responsibility is to solve problems and overcome challenges. However, with the dramatic pace in which the current landscape of processes, technologies, and tools are changing, it has become quite problematic to cope with it. Moreover, the pressure business users have been putting on DevOps and IT operations teams is staggering, demanding that everything should be solved with a tap on an app. However, at the backend, handling issues is a different ball game; the users…Read More