Telecommunications is an industry which is developing at a rapid rate. Fragmented test environments, limited visibility and duplication of time spent collaborating with diverse information are recurring issues. O2/Telefónica UK aimed to improve the speed and quality of application development and delivery through improved management of test environments. It selected a Plutora solution to achieve this.
O2 is a leading mobile network operator and the principal commercial brand of Telefónica UK Limited. It has around 6,700 employees in the UK across three main offices and 450 stores.
Some of the frequent challenges O2 faced included having a fragmented and abundant number of test environments; limited visibility of test environments had a potential impact on O2’s customers and the business; and the duplication of time spent collaborating with diverse information sources for non-production environments.
The benefits of implementing the Plutora Test Environment and Release Management solution therefore included: centralised management of disparate test environments; live visibility of test environments including live dashboards and reports; new process to manage conflicts and alert key stakeholders to any IT issues or changes in test environment status; and proactive management of non-production environments improved agility and resiliency.
The results of the implementation included: 60+ critical test environments now being managed; 50% reduced unplanned outage of non-prod environments; saving admin time through automated scheduling and coordination; reduced delays on test schedules due to misconfigured environments; improved environment agility.
O2 centralised its IT test environment management
Processes
Processes were manual and lacked coordination and collaboration from the different IT teams. It was taking several days to track and rectify unplanned test-environment outages. Conflicts (double-bookings) were common through not having visibility of other bookings on the same environments.
Challenges in software delivery
Telecommunications is a fast-moving industry. As a leading brand at the forefront of the industry, O2 has continually innovated new mobile products and has championed customer experience. In order to maintain and extend its leadership position, O2 maintains the highest standards in its IT application delivery in order to support its massive customer base which comprises over 33 million connections. However, its IT infrastructure experienced a number of challenges which could slow down the planned improvements.
The UK IT test environment management group supports a large number of internal teams. They were faced with over 30,000 emails and remedy requests relating to non-production environments. In addition, scheduling environments were not centrally managed, allowing each team to manually add their bookings into spreadsheets with little or no collaboration or arbitration on conflicts. Management of communication and schedules was an unsustainable situation without more effective tools or a lot more people. In the case of unplanned outages (i.e. failures in non-production systems or applications), it could take several days to work through the various architectures to understand where the root cause of a test system outage originated. Furthermore, it drew heavily upon many different support teams which had to be coordinated using manual lists. Ultimately, IT testers were often unable to carry out their testing projects while issues with the test environments were being investigated.
Mark Lewis, UAT Test Environment Manager at O2, commented: “One solution we sought was visibility of all of our test environments to avoid double-bookings and uncontrolled changes and make sure sufficient management is in place to be effective.”
The UK IT test team – in conjunction with its quality test partner, NTT DATA – decided to undertake a review of IT test environment management tools to provide a solution to address these issues.
“We are rolling out a dynamic, company-wide IT test environment view in the UK where key stakeholders will be able to self-serve with customised dashboards and reports and see environment utilisation centrally for the first time. Plus they can be alerted to any potential system conflicts, enabling them to have a direct conversation with the stakeholders affected and take immediate action,” said Rob Hutchinson, Senior Test Manager, IT Billing and Charging, O2.
“One of the main problems was that we just didn’t have visibility of all of our test environments and that frequently resulted in double-bookings and uncontrolled changes because there wasn’t sufficient management in place to be effective,”said Lewis.
Partnering with Plutora
After a two-month external review of IT test environment tools, O2 and NTT DATA selected Plutora as it is an independent, dedicated, test environment management platform that could be rolled out across the business. Plutora was brought in to deliver the following outcomes:
- 100% visibility of the test environment planning
- Enable unified scheduling for the first time
- Centralised environment booking and change requests for testing and change management
- Automation of manual improvement processes to increase productivity and cut resource duplication
- Integrate seamlessly with Plutora’s standard APIs so other systems and tools used in the O2 ecosystem can benefit and further reduce effort and manual updates
- New self-service capability for the senior leadership team via Plutora’s reporting feature
- Up-to-date dashboards, alerts, notifications and analysis to engage stakeholders and reduce unplanned outages
“Plutora was the only tool dedicated to IT test environment management, rather than it just being an add-on, and the team was really flexible and supportive,” said Lewis.
The focus of the O2 IT test team is to support the IT environments, including physical hardware and application upgrades that support their customers during their test lifecycles. This spans simulation of buying a new mobile phone to services that support their provisioning such as operations and billing upgrades. The implementation of Plutora was carried out in two phases to ensure a smooth transition from the existing two tools which only included IT testing as an add-on.
In phase one, O2 and its QA test partner, NTT DATA ran a pilot and was satisfied that the tool contained all mission-critical processes to ensure that when any service exits the test phase, it can be fully integrated into production with minimal risk.
After four months, phase two of the implementation extended a rollout to 60 test environments including support of the company’s compliance with the European Union GDPR data compliance regulation which came into force in 2018.
“Our main priority for the next two years is to ensure that test environment and release management are given the focus they need and we are as efficient as possible,” said Lewis.
“Plutora gives us a solution that helped reduce unplanned non-production outages by 50% in a year,” said Hutchinson.
Benefits
The telecommunications company has made immediate gains following the Plutora implementation. Many IT testing problems that the company experienced in the past have disappeared due to the hard work of O2’s IT Environment Management team with quality test partner, NTT DATA and Plutora. Standardising the handling of IT test environments with Plutora has meant a considerable reduction in testing effort and a much easier, more efficient way of managing IT schedule conflicts. Now, key stakeholders can be alerted automatically to any issues or changes in IT environment status.
Release and environment teams are no longer relying on spreadsheets. Manual improvement processes have been automated and standardised. Consequently, a single, dynamic view of IT test environments has been created from which vital intelligence is garnered to prevent any detrimental impact on internal project delivery. Implementing Plutora has contributed to O2’s IT modernisation and drastically reduced unplanned non-production downtime by 50% in a year. Before, these outages were managed by identifying incidents manually. Then they were communicated to affected stakeholders and project teams using a shared mailbox. Now, the Environment request module in Plutora is used to capture environment bookings and system changes. Also, reports and workflows have been automated to broadcast communication to projects and support groups.
Furthermore, O2 is using Plutora’s reporting feature to automate monthly KPI reports, thereby improving measurement and giving a foundation for productivity improvement.
