Not a Byte Lost: Qualtrics Moved 1.5 Petabytes from IBM Bare Metal to AWS

Executive Summary:

Qualtrics, a global experience management company whose platform answers more than 100 million search queries a day, needed to move off rigid IBM bare-metal infrastructure that was slow to scale, offered limited disaster recovery and depended on manual operations.

Rapyder migrated 1.5+ petabytes of data across 13 production Elasticsearch clusters, the full Kafka estate and all PostgreSQL databases to AWS, rebuilt the infrastructure as code with Terraform and Ansible, and put automated monitoring and alerting in place.

The result: zero data loss, 99.99% availability through cutover, deployments cut from 3–5 days to under 2 hours, 70% faster incident resolution and headroom to grow to 5+ petabytes without redesign.

About Customer:

Qualtrics runs a global experience management platform that helps organisations capture, analyse and act on customer and employee feedback. Its search and analytics layer processes more than 100 million queries a day. Behind it sits a petabyte-scale data estate: Elasticsearch clusters across North America and Europe, high-throughput Kafka event streaming handling millions of events per second, and PostgreSQL databases for customer accounts, subscription billing and operational data, running at more than 10,000 transactions per second at peak.

Industry:

Industries, IT- ITeS

Offering:

Offerings, Migration

Business Challenges:

1. Capacity that couldn’t keep up – New capacity took 72+ hours on IBM bare metal. With 100M+ daily queries, every demand spike put performance and customer experience at risk.
2. Single-datacentre risk – One datacentre, no multi-AZ resilience and limited DR stood behind a 99.99% uptime commitment. A single site failure could take search, streaming and billing down together.
3. Slow environment delivery – Each new environment took 3–5 days to deploy, delaying testing and slowing releases to customers.
4. A hard growth ceiling – Scaling from 1.1 PB to 5+ PB would have forced an architectural redesign, with backup storage growing alongside the data.
5. Manual incident response – Issues across 500+ servers were investigated by hand, stretching resolution times and tying engineers up in routine work.
6. A zero-tolerance migration – 1.5+ PB across 13 production clusters, plus Kafka and PostgreSQL, had to move live with zero data loss and no breach of latency SLAs.

Solution Implemented:

1. Infrastructure as code – Terraform provisions all AWS infrastructure from versioned state, and Ansible rolls out changes across 500+ servers. Capacity is now added in hours, not days.
2. Built-in resilience and DR – Synchronous PostgreSQL replication delivers zero RPO, backed by pgBackRest backups on Amazon S3 and quarterly recovery tests. DR targets: RPO under 5 minutes, RTO under 30 minutes.
3. Standardised environments – Every service ships as a versioned Docker image with resource limits and health checks, while Hiera and Vault manage configuration and secrets.
4. Built for multi-petabyte scale – The AWS architecture scales to 5+ PB and 300% transaction growth without redesign, and S3 Intelligent-Tiering and lifecycle policies keep backups lean.
5. Automated monitoring and alerting – Grafana (50+ panels), Cerebro, Elasticvue and Kafdrop give live visibility, while Slack alerts, escalation policies and runbooks speed up response.
6. Workload-specific, zero-loss migration – Elasticsearch moved live through shard rebalancing (snapshot and restore for clusters over 80 TB), Kafka through broker expansion and PostgreSQL through pgBackRest replication, all over encrypted VPN with tested rollback.

Technology Stack:

• Amazon EC2
• Amazon EBS
• Amazon S3
• Amazon VPC
• AWS IAM
• AWS KMS
• AWS CloudTrail
• Amazon VPC Flow Logs
• Terraform
• Ansible
• Docker
• Jenkins
• Grafana
• Apache Kafka
• Elasticsearch
• PostgreSQL
• pgBackRest

Business Benefits:

1. New capacity in under 2 hours, down from 7 -+ – – Qualtrics now scales to meet demand as it happens instead of waiting days for hardware.
2. 99.99% availability. RPO of zero – – Qualtrics protects its uptime commitments and billing data, with disaster recovery designed and tested rather than assumed.
3. 95% faster deployments – – Environments that took 3–5 days now deploy in under 2 hours, so engineering teams test and release sooner.
4. Headroom to 5+ PB. 75% smaller backup footprint – – Qualtrics can grow data volumes and transactions by 300% without redesign, while cutting backup storage and keeping 7-year retention.
5. 70% faster incident resolution. 85% fewer manual tasks – – Qualtrics spots and fixes issues sooner and frees engineers from hand-run procedures across 500+ servers.
6. Zero data loss across 1.5+ PB – – Every Elasticsearch cluster, the full Kafka estate and all PostgreSQL databases moved with 100% data integrity, with p95 query latency under one second, Kafka acknowledgements under 100 ms and PostgreSQL commits under 50 ms.

Key Takeaways:

• Migrate live where you can – expanding running Elasticsearch and Kafka clusters into AWS let data move while customers kept using the platform.
• Match the method to the workload – live rebalancing, snapshot-and-restore and physical replication each suited a different part of the estate.
• Codify first, then migrate – the Terraform and Ansible foundation that made the migration repeatable also cut provisioning from days to hours.
• Always keep a way back – tested per-service rollback and a 30-day warm standby gave the business confidence to proceed.