Test against real data, not real customer data
If dev and test were copied from production, your customers' real names, emails and card numbers are in them right now
Keep personal data out of dev and test
Every time a team copies production into dev, test or a sandbox to get realistic data, the personal data comes with it, into environments far more people can reach. Safe Test Data runs your scrub rules against an isolated copy of an RDS snapshot, so only the sanitised result reaches those environments.
Create a sanitised copy of production for dev and test
The scrub runs in your AWS accounts and the sanitised copy is stored in your S3 bucket, or as an RDS snapshot, ready for your team to use. The scrubbing environment is destroyed once the run completes.
Point it at a snapshot
Give it an RDS snapshot from this account, or one shared from another account in the same AWS region.
Build a throwaway copy
A CloudFormation stack creates its own VPC, restores the snapshot into a new database and resets that copy's master password. Your live database is never connected.
Scrub with your rules
A Fargate task fetches your scrub SQL from S3 and runs it against the copy, masking, replacing or deleting data exactly as your script says.
Take the sanitised result
Choose a dump in your S3 bucket, a scrubbed RDS snapshot or both, then restore it into dev or staging with the process your team already uses.
It tears itself down
The stack is deleted at the end of every run, taking the throwaway database and its network with it. You keep the sanitised dump or snapshot.
Scrub rules written for your schema, by your team or by base2Services
The scrub rules decide exactly which fields are masked, replaced or removed. Your team can write them, or base2Services builds them with you and keeps them current as your schema changes. Either way they are plain SQL kept in your S3 bucket, so they are reviewed and versioned like any other change. Safe Test Data does not scan for personal data on its own; the rules define what changes.
- Mask or replace names, emails, card numbers and any other field your schema holds
- The copy's master password is reset automatically, and the rules remove other database logins so production credentials stay behind
- Delete old or unneeded rows so the dump is smaller than production
Realistic data for dev, staging and developer laptops
The copy starts from a real production snapshot and is scrubbed in place, so it keeps production's schema, volumes and relationships. Only the fields your script changes are different. Developers build against data that behaves like the real thing, rather than thin synthetic data that hides the bugs real data finds.
- Production's volumes, relationships and edge cases, not generated rows
- Personal fields carry whatever stand-ins your script writes, from simple patterns to realistic names
- Pull the sanitised dump from your S3 bucket to a laptop, or restore the scrubbed snapshot into dev or staging
Works with your RDS databases, in your AWS accounts
Safe Test Data runs entirely in your AWS accounts on Fargate, and the data never leaves them. When the snapshot lives in another account, it assumes a role you grant. What you get back depends on the engine, and any engine can also produce a scrubbed RDS snapshot.
- Amazon RDS for MySQL, PostgreSQL and SQL Server, plus Aurora MySQL and Aurora PostgreSQL
- Snapshots shared from another account work when they are in the same AWS region as the run
- Encrypted snapshots work when the run's role can use the KMS key
- RDS for Oracle and RDS for Db2 are not supported
Built by base2Services
The pipeline we built to give our managed customers realistic data in lower environments, without copying their customers' personal data along with it.
base2Services is an AWS Advanced Consulting Partner specialising in platform engineering and managed AWS operations. This is part of how we run AWS for people, not a side project. If you would rather not run it yourself, the team that built it can run it against your snapshots whenever you need a fresh environment.
Helps keep regulated personal data out of dev and test
Frequently asked questions
Does Safe Test Data find the personal data for me?
No. The scrub rules decide which columns change, whether your team writes them or base2Services builds them with you, so the result is only as complete as the rules. Review them whenever the schema changes.
How long does a run take?
Each run restores, scrubs and dumps the full snapshot one step after another, so run time grows with database size. For very large databases you can skip the dump, take a scrubbed snapshot instead and give the task more CPU and memory.
How often can it run?
As often as you need fresh data. Each run is independent, so you can trigger it from your CI pipeline or a scheduled job to refresh dev and test.
Do we run it, or does base2Services?
Either. It runs in your AWS accounts with a role you grant, so your own team can run it. base2Services can also run it for you as part of Cloud Management, keeping the scrub script with your infrastructure code and scheduling the refreshes.