Launch day, no problem: FAIRTIQ’s smooth ride into Denmark

back to tech blog

On the 4th of September 2024, we reached an exciting milestone at FAIRTIQ — the nationwide launch of our solution in Denmark! As the project’s lead engineer, I was responsible to ensure our system could handle the increased demand. In this blog post, I invite you to walk through the timeline of events before, during and after the launch. It’s the story of the preparation, the challenges, and the moments that made this launch a success.

Launch timeline: from testing to triumph

The journey to Denmark’s nationwide rollout started months earlier. Here’s how we planned the launch and prepared our teams for this important milestone.

Five months before launch: building confidence with an MVP

In April 2024, we launched a minimum viable product (MVP) with a limited number of users. Between April and June 2024, we increased the limit multiple times. This iterative process allowed us to test the system under growing load. The lessons learned from the MVP phase gave us the confidence to remove the user limit and proceed with the public launch.

Two weeks to two days before the launch: final preparation

The final preparation of this event was greatly supported and inspired by our lead architect Sebi who summarised very well the mindset of the Tech team with these words:

Hope is not a strategy! Failure is inevitable. Let’s plan for it.

Capacity Planning: We followed our capacity planning strategy to ensure the system can handle both the expected spikes and increased load over time efficiently. We identified nine areas in the system that needed adjustments. For each of them, an appointed engineer proposed a scaling plan, assessed potential failure points and prepared their colleagues to act fast in case of a problem during the load ramp up.

Vendor Coordination: We informed all third-party system vendors about the upcoming launch and confirmed their readiness to handle increased traffic seamlessly.

Scaling the System: We reviewed the capacity plans with all appointed engineers, who scaled services and databases capacity accordingly.

Company-wide Communication: We shared the event timeline and technical measures with the entire company to ensure alignment and readiness.

The day before the launch

• Morning hiccup: We encountered a bug in our monitoring system, introduced by the system scaling. We quickly identified the problem, since we were no longer receiving any metrics, and we fixed it within an hour.
• Third-party system issues: A regularly problematic third-party system generated a flood of irrelevant error messages, cluttering our logs and alert channels. We quickly resolved this issue by filtering out irrelevant messages from this system, allowing us to focus solely on the relevant data.
• Payment Issue: We encountered an issue with Apple Pay, related to our acquirer, that prevented users from linking ApplePay to their account. Our team quickly resolved the issue, ensuring there was no disruption for users.
• Unsupported Danish special characters: the partner informed us about some special characters present in Danish names that wouldn’t display correctly in the mobile app. We deployed a last-minute fix to those special characters.
• Green light at 13:30: After resolving the morning's issues, we received the go-ahead from our partner RKRP for the public release.
  

Launch day: 4th of September 2024

The big day began with a mix of anticipation and meticulous monitoring. Here’s how it unfolded:

• 06:30: All appointed engineers and project leads gathered in the virtual launch room, where our CTO Michel delivered a warm-up speech to set the tone for the day.
• 06:35: The first peak of 66 registrations per minute was handled smoothly by the system, with no signs of strain.
• 06:47: Project leads shared updates about the press coverage, including national news articles and live interviews on radio and TV.
• 07:00: A national Danish radio broadcast about the launch boosted user engagement, resulting in a significant spike.
• 07:06: We recorded 315 registrations per minute — a critical moment that the system handled flawlessly.
• 09:30: During the first status update, the appointed engineers and project leads confirmed that the system was handling the load without any issues. Customer care service reported no issues.
• 11:45: Metrics showed consistent registration and journey activity, confirming system stability. The engineering teams resumed day-to-day work.
• 14:30: By this time, registrations had increased by 35% compared to the MVP, and journeys went up by 25%.
• 22:20: The day ended with 50% more registrations and journeys. All invoices were successfully settled. The appointed engineers analysed the system’s performance throughout the day and concluded that it was oversized. However, we decided not to change anything in anticipation of the expected load peak the next morning and then decide on a resizing plan.
  

Post-launch: monitoring and optimisation

• 5th September, early morning: The system handled another load peak without issues, reaffirming the robustness of our preparation.
• 5th September, 09:30: Appointed engineers evaluated again the impact of the launch and decided  to scale in/down the system to optimise costs while maintaining performance.

Lessons learned and look-ahead

This launch was a significant achievement, but it also provided valuable insights that will shape how we approach similar challenges in the future:

1. Planning and preparation are key: By systematically analyzing potential failure points and preparing mitigation strategies, we ensured the system could handle even the most intense load peaks. On top, this preparation allowed us to handle encountered hiccups very smoothly. All in all, that emphasises how important this preparation was.
2. Collaboration across teams and vendors: The coordination between FAIRTIQ teams and third-party vendors played a crucial role in preventing disruptions. Clear communication and early alignment were key in keeping things running smoothly.
3. Adaptability is essential: Despite careful planning, unexpected issues arose, such as unsupported special characters in Danish names. Our ability to adapt quickly and safely deploy last-minute fixes highlighted the importance of flexibility and a proactive mindset. (read also CI-CD on mobile)
4. The power of testing: The phased rollout of the MVP taught us critical lessons about system performance under load. This gradual testing gave us confidence in the system's stability for the public launch.

These learnings will strengthen our approach as we grow in Denmark and expand to new markets, ensuring we deliver even better experiences for our users.

Wrapping up

The nationwide launch of FAIRTIQ in Denmark highlighted the importance of thorough preparation, strong collaboration, and adaptability. Rigorous capacity planning and testing during the MVP phase gave us the confidence to scale, while clear communication with third-party vendors ensured alignment and readiness.

Despite meticulous planning, unexpected challenges like unsupported Danish characters underscored the need for flexibility and quick problem-solving. Each step of this journey reinforced that preparation and adaptability are the foundation of successful rollouts, lessons we’ll carry forward as we expand to new markets.

back to tech blog