#2 Creating Judopay's Next-Gen Tech Platform

Following on from our inaugural engineering blog, #1:  Challenging the Status Quo in Tech, I thought I would introduce you to some of Judopay’s engineering team, to talk about their journey in creating their next-gen tech platform:

‍

David Pollock, Head of Infrastructure and Automation Engineering

I have been with Judopay for over 6 years now, and have over 15 years of experience in the IT industry.  I’m passionate about building reliable, scalable systems and reducing toil with automation.

‍

Philip Keen, Senior Software Engineer

This is my fifth year of working for Judopay. I’ve been a software engineer for just over 10 years.

‍

James Patey, Senior Software Engineer

I have been with Judopay for 4 years now, with over 15 years working as a software engineer. I’ve long had a strong interest in building scalable systems, and learning new technologies along the way.

‍

As we mentioned in our inaugural blog Challenging the Status Quo in Tech, we are proud to have built a next-gen payments platform, running exclusively on cloud native principles. This enables Judopay to process our customers' payments with:

Considering this ambition, our engineering team is lean. Every single member of our team is capable of making significant contributions to our goals. Operating with such limited resources presents not only a challenge, but also a great opportunity in forcing us to optimise our processes and focus on building features in order to provide the biggest value to our customers.

Like many engineering teams, we were faced with an outdated and struggling system, which was not designed for the needs of our business today and was beginning to impede our release agility.

So, to comprehensively address this, we have been going through a process of rebuilding our core systems onto a new tech stack. By embracing the advances in cloud platforms and more modern microservice architectures, to deliver a next-gen payment platform as part of our Sho~dan programme.

As the Sho~dan programme approaches completion, we are proud of our accomplishments so far and would like to share our story behind our modernisation project, with the hope it will resonate with many engineering teams out there undergoing similar transformations.

Over the coming months, we will be releasing a series of engineering blog posts providing a deeper-dive into  aspects of the Sho~dan programme and the approaches we have taken towards engineering at Judopay.

‍

Why Sho~dan?

We named this project Sho~dan (Japanese for first dan, or first rank), because we consider it the first phase of a new beginning in our tech platform.

‍

What are some key tenets for your team which have lead to your success?

Taking the time to do things right…

As a development team, we have put an emphasis on taking the time to do things right, helping to avoid as much technical debt later on.

We take the opportunity to explore new approaches and technology in order to achieve more optimum solutions. I have relished the opportunity to innovate and try something new, rather than sticking with the status quo.

Taking the time as a team to have these discussions upfront and bounce ideas off each other, in my mind  has led to better solutions and a great team dynamic.

‍

Reducing Toil…

As a system grows and matures, it is very easy to become bogged down in regular maintenance tasks just trying to keep everything running and secure.

Keeping systems up to date with:

• Patches
• Rotating keys
• Certificates or Passwords
• Data clean-ups
• Scaling Systems to meet growing demand
• A never ending list of other day-to-day tasks

If we didn’t automate these tasks we would be spending all our time just keeping the lights on! Our team has put a high emphasis on reducing this manual toil, making sure that as we implement new systems or tools, we immediately automate any accompanying maintenance tasks.

This means it can take a bit longer to implement a new system, but we feel that the longer a manual task is left un-automated, the more entrenched it becomes.  So wherever possible we try to automate as much as we can up front.

I think this has been crucial to the success of the team, both in being able to keep up with the evolving needs of the business, and also increasing the satisfaction of the engineers in the team since almost all mindless, repetitive tasks have been eliminated.

‍

What is your team’s biggest challenge?

Balancing reliability and stability versus innovation…

Our engineering team faces significant challenges:

• Balancing our need to undertake this modernisation effort
• Deliver new features
• Our responsibility to maintain a stable and reliable system for our merchants

Payments is a 24/7/365 service. We do not have the luxury of maintenance windows to leverage, along with a low appetite for risk given the zero-tolerance expectation with regards to service outage.

It was a constant consideration not to impact our merchants. This meant adopting a gradual phasing in of Sho~dan, all while our legacy system was still running. Given the size of the Sho~dan programme, it was also not practical to put a change freeze on our legacy system, as continually maintaining and adding new features was essential to our business.

This meant simultaneously building an entirely new system, then seamlessly integrating it with our legacy system as it began to take over various functions.

To reduce risk, we adopted the stangler pattern. Our microservice based architecture helped a lot too.

Our CEO, Jeremy, described the process as:

“We upgraded our aircraft to a rocket ship, all whilst the aircraft is in mid-flight”

‍

New Technologies, New Products…

As a developer you are always learning. The Sho~dan programme presented some enjoyable learning challenges.

We wanted to build a payments system that was both reliable and scalable. We chose to use the Lagom framework for this task. However, this was a new technology for all of us. Learning how to make the best use of it was a key challenge. A challenge that we met.

Over the past few years, we have managed to settle on our own preferred practices and even built up our own Lagom-based software libraries to help with our development tasks. Writing a Lagom microservice is now much easier for us!

As Sho~dan was the preferred platform for new products, this meant we had to understand how new products worked, for example Pay by Bank app and PayPal. This was in addition to deepening our knowledge of the evolving payment landscape.

‍

With the power of hindsight, is there anything you would do differently?

Working harder to get everyone on the same page…

Overall, I believe this was a proud and exciting journey for the engineering team. However, this involved a considerable shift in thinking, and at times the learning curve was pretty steep.

We facilitated this with regular discussions in open forums where the team could challenge ideas, address concerns and review materials introducing the concepts we were bringing in.

Also, we gave ourselves space for trial and error. When you are learning to use a new technology it is to be expected that some of your initial coding attempts will be missteps. However, in some instances we could have been faster to pick up where improvements needed to be made.

We hope to bundle our learnings into an induction program, so that new joiners have a running start when they begin to work with our code.

‍

Not a lot…

Honestly, there’s not a lot I would do differently. There are certainly minor technology choices I would change. The world of IT moves so quickly. There is always a new technology, solving a problem in a slightly better way.

On the whole I am really happy with the approaches we have taken and our technology choices.

‍

Did you encounter any unexpected developments or learnings during the Sho~dan programme?

Scala has been great…

One of the unexpected outcomes going into Sho~dan has been the adoption of Scala as our backend language of choice, even though this wasn’t necessarily a change we were looking at.

However, Scala is a popular language with a wealth of libraries available, and a dedicated community able to offer advice. We relished the challenge of mastering this language.

Speaking personally, I initially found Scala’s more functional style rather fussy, but I came to appreciate it. It encourages us to write more modular code - functions that are easy to read, test and re-use. Not to mention Scala also frees us from the curse of the null pointer exception.

‍

All the time…

Sho~dan has transitioned us from a monolithic .NET system running on Windows servers, to a microservice based, mostly Scala system, running in Kubernetes.

We have moved from storing data using traditional, self-managed RDBMS, to unlocking greater data potential with Cassandra, Kafka, CloudSQL and BigQuery.

With such a monumental shift in technologies and approaches, we were always encountering the unexpected, but learning different tools and technologies is one of the best parts of my job. Adapting to the unexpected challenges we encounter is great fun and pushes us all to become better, more well-rounded engineers.

‍

A Change of Course…

Sho~dan has been a significant project, and one in which a lot of time has been invested. Of course, businesses don’t stand still, and the Sho~dan programme has also catered for changing priorities and new opportunities.

Around the same time we began work on the Sho~dan architecture, Judopay was given the opportunity to offer Pay-by-Bank-app as a payment option. Pay by Bank app is not card-based, and it would have been challenging to incorporate it with our existing solution - a monolith system designed around card payments.

However, given the way we designed Sho~dan’s microservice architecture, it was more straightforward to create a dedicated Pay-by-Bank-app service that would work within the network of services we were writing.

As the Sho~dan systems have been designed with flexibility in mind, adapting to these new requirements and priorities has been relatively easy.

‍

Functional Programming…

Scala encourages some programming techniques such as adopting use of pure functions and immutability wherever possible (which are tenets of functional programming) - this was a learning curve. Taking the effort to go ‘all-in’ with Scala at times took longer, but ended up being a reward, as I believe it has led to us writing more robust code with less bugs.

The functional approach also lends itself well to the distributed architecture we are using with event stores and eventual consistency.

‍

What are you most proud of?

Our Automation…

 I believe we have done a great job automating as much as we have!

• We have a really mature continuous integration and delivery platform, with a short feedback loop for developers.
• We have covered virtually 100% of our infrastructure, as Infrastructure-as-code.
• We have also developed mature continuous deployment pipelines for our infrastructure changes, and have built continuous maintenance systems to keep our applications and infrastructure up to date.

Many engineering teams have some or all of this in place, but in my experience not many have executed it as well as we have.

‍

Reduce re-inventing the wheel…

Our adoption of microservices has required us to implement a number of services, with some similar tasks and shared structure. In many cases related building blocks of logic are used in multiple services creating potential repetition. This has led to us building a fairly comprehensive set of ‘commons’ tools and programming packages.

This has been a benefit for improving maintainability, while also building a familiar structure for each service and has helped us achieve high unit test coverage with a lower amount of effort.

What does the future look like for Judopay’s Engineering team?

A greater focus on data engineering…

As Sho~dan moves towards completion, we have been working on plans for Ni~dan (our second dan).

A large component of this will be focused on improving our data engineering practices. Building real-time data systems based on ideas such as Data Mesh and Event Sourcing. We are excited to see the value we can provide to our merchants, with these improved data insights.

‍