Major financial services firms are turning to devops practices to enable faster, more stable software releases in the race to keep up with quickly changing customer expectations and a competitive assault from a new breed of fintech challengers.
“When I look at disruptors in the financial services space, they exploit speed. That is the cheat code to survive,” Julienne McLean, a former consultant and now agility lead for consumer and community banking technology at JPMorgan Chase said during the virtual Devops World event last month.
The Covid-19 pandemic has only accelerated longstanding issues for these companies, as customers demand greater access to digital tools and services to manage their money.
“Many banks have yet to see this mindset shift translate into actual user behaviour, perhaps due to limitations of their digital capabilities. Should these emerging preferences become banking’s post Covid-19 ‘next normal,’ retail banking distribution will experience up to three years of digital preference acceleration in 2020,” McKinsey analysts predicted back in June.
Naturally, any serious change of speed for digital delivery will have to be balanced with the high security and compliance bar that banks must meet to stay on the right side of the regulators and maintain customer trust.
Here are just some of the ways four major financial services providers have adopted and scaled devops practices across their organisations.
JPMorgan Chase: Like getting an elephant to run
JPMorgan Chase has been pushing towards agile practices since before McLean joined the bank as an agility coach back in 2017.
“To compete in this landscape, it is critical to deploy features at a rapid pace,” she said. “By leveraging devops, CI/CD, and automation tools, we can speed up that release timeline significantly. I find in teams I have worked with developers are then able to focus more on the fun things by being creative and innovative.”
McLean admits that getting an established, risk-averse organization like JPMorgan to completely upend the way they work “can feel like trying to get an elephant to run.”
Finding common ground as to what agile means to your whole organization and being transparent regarding progress were key for JPMorgan. Providing leadership with regular updates on key metrics related to the speed and stability of releases helps to gain that senior buy-in, especially as it relates to the financial impact of that work. This is a bank, after all.
“Think about what you are looking to gain and why,” she said. “These methods don’t work for every organisation or team. Figure out your business model and if this is something that would help and why, instead of feeling forced to take on methods that might not work for you.”
It also helps to have a business stakeholder as part of those devops teams, as this allows for a regular cadence of communication regarding changes of business priorities and also keeps the more technical teams oriented around customer needs.
When it comes to some common pitfalls, McLean says it is important to not get bogged down in the technology stack or tools. “One of my favourite principles from the Agile Manifesto is people and conversations over process and tools,” she said, advising that the stack should be the last part of the process.
Finally, McLean advises against any thinking that devops can be a destination to be worked towards, or that agile-in-a-box can be picked off the shelf. “The mindset of it being a destination rather than a journey is really a common pitfall,” she said.
Capital One: Automating quality checks
Capital One was talking about agile and the cloud as far back as 2010, and was one of the first major US banks to adopt devops practices at scale.
“Our product managers obsess over customer feedback and embrace moving customers’ ideas into products to make their banking and financial services experiences top-notch,” John Andrukonis, chief architect at Capital One has said as part of an Amazon Web Services (AWS) case study. “Insights are only as valuable as our ability to act quickly on them, and that’s what devops helps us do.”
The latest part of this journey has seen the bank increasingly automate key parts of the software deployment process. Being a regulated bank, all deployments have to pass 16 quality gates at Capital One, which creates a high bar for developers and a fair amount of toil to get releases out of the door.
Now, the bank has built an automated approval process which pushes developers to use pre-built Jenkins pipelines which have been fully vetted by the compliance team. This ensures that all releases are compliant by default, even against a rapidly changing set of requirements.
These are managed centrally and given regular deep code reviews, integration testing, and unit testing by dedicated devops teams.
At runtime, the base Jenkins file runs a check that the application is using a managed pipeline and the approval process is fully automated.
“Adopting a managed pipeline framework has helped us minimise the risks and challenges associated with maintaining enterprise compliance, while at the same time holding our engineers to the highest level of engineering excellence,” Linda Oyolu, senior associate software engineer said during Devops World.
Fidelity Investments: Driving engineering excellence
Similar to Capital One, Fidelity Investments has been a financial services front runner in the gradual shift from on-premises to the cloud.
“We wanted to modernise applications and minimise tech debt, make this process repeatable, and drive transformation of how we develop software at scale,” Nicholas Penston, director of software engineering at Fidelity Investments said during Devops World.
The Boston, Massachusetts-based investments firm started by turning to containers, microservices, and APIs for new applications. Now, the company is looking to modernise a broader swath of workloads and spread them across a variety of private and public cloud platforms.
The goals are familiar: greater speed and higher quality of software delivery, all while freeing up developers from manual deployment work so they can focus on work that adds value.
One such move has seen Fidelity engineers move their continuous integration (CI) process on to local machines. Where before CI could take minutes per deployment when run in a dev-style environment, now the experience takes seconds when mirrored on a developer’s local workstation. Those incremental gains can really add up when you are looking to deploy on a daily basis.
“The heartbeat here is Skaffold, an open source framework to create a hands-free automation to get those applications into Kubernetes,” Penston explained further during his presentation. “When you go local, use it as an enabler to drive maturity in principles and practices.”
That’s not to say there wasn’t pushback. Penston and his team had to advocate for more local development and push for what he calls “engineering excellence,” where all engineers are driven to increase value, take ownership for their workloads, and engage in a culture of knowledge sharing wherever possible.
Nationwide Building Society: Automation is the icing on the cake
The devops phenomenon in financial services isn’t limited to the Unites States either. In the UK, high street banks like Lloyds, Royal Bank of Scotland, and Barclays have been hiring for devops engineers and Nationwide Building Society has been very public about its own devops journey.
In September 2019, Nationwide committed to a £4.1 billion ($5.3 billion) investment in technology over the next five years to help it keep up with a raft of fintech challengers, such as popular neobanks Monzo and Revolut.
These digital-first challengers have moved the goalposts for legacy outfits like Nationwide, forcing them to adopt more modern development techniques to meet those expectations. For example, “as customers log into our apps more, and send more payments, they expect real time data,” Rob Jackson, head of application architecture at Nationwide said during the Kafka Summit in London last year.
Devops is a key cog in this drive towards greater speed and agility.
Patrick Eltridge, Nationwide COO, has pushed for more agile practices in banking both in his previous role as CIO at RBS and now at Nationwide. One of his key principles is aligning the bank around its key value streams, grouping engineers around a product that they own end-to-end. This might be the customer-facing mobile app, mortgages, or credit cards.
“We fundamentally believe the people who do the work should be involved with changing and improving the work,” he told InfoWorld earlier this year. “Additional value comes over time with automation, but that is icing on what needs to be a mature cake already.”
While speed is important to Eltridge and his team, stability of deployments is also a key metric, with incident volumes and mean time to resolution both key metrics for the bank as it measures its devops progress.
Lastly, Eltridge makes it clear that devops does not have to equate to cloud in the banking world, where some workloads will have to stay on premises for the foreseeable future.
“You can do devops without cloud but cloud is a massive accelerator,” Eltridge said. “It is ultimately easier, quicker, and safer to do your work in a public cloud environment, but that depends on sound engineering.”