Automatically replicate your card payment keys across AWS Regions
In this blog post, I dive into a cross-Region replication (CRR) solution for card payment keys, with a specific focus on the powerful capabilities of AWS Payment Cryptography, showing how your card payment keys can be securely transported and stored… In today’s digital landscape, where online t…
In this blog post, I dive into a cross-Region replication (CRR) solution for card payment keys, with a specific focus on the powerful capabilities of AWS Payment Cryptography, showing how your card payment keys can be securely transported and stored.
In today’s digital landscape, where online transactions have become an integral part of our daily lives, ensuring the seamless operation and security of card payment transactions is of utmost importance. As customer expectations for uninterrupted service and data protection continue to rise, organizations are faced with the challenge of implementing robust security measures and disaster recovery strategies that can withstand even the most severe disruptions.
For large enterprises dealing with card payments, the stakes are even higher. These organizations often have stringent requirements related to disaster recovery (DR), resilience, and availability, where even a 99.99 percent uptime isn’t enough. Additionally, because these enterprises deliver their services globally, they need to ensure that their payment applications and the associated card payment keys, which are crucial for securing card data and payment transactions, are securely replicated and stored across AWS Regions.
Furthermore, I explore an event-driven, serverless architecture and the use of AWS PrivateLink to securely move keys through the AWS backbone, providing additional layers of security and efficiency. Overall, this blog post offers valuable insights into using AWS services for secure and resilient data management across AWS Regions.
Card payment key management
If you examine key management, you will notice that card payment keys are shared between devices and third parties today the same as they were around 40 years ago.
A key ceremony is the process held when parties want to securely exchange keys. It involves key custodians responsible for transporting and entering, key components that have been printed on pieces of paper into a hardware security module (HSM). This is necessary to share initial key encryption keys.
Let’s look at the main issues with the current key ceremony process:
- It requires a secure room with a network-disconnected Payment HSM
- The logistics are difficult: Three key custodians in the same place at the same time
- Timewise, it usually takes weeks to have all custodians available, which can interfere with a project release
- The cost of the operation which includes maintaining a secure room and the travel of the key custodians
- Lost or stolen key components
Now, let’s consider the working keys used to encrypt sensitive card data. They rely on those initial keys to protect them. If the initial keys are compromised, their associated working keys are also considered compromised. I also see companies using key management standards from the 1990s, such as ANSI X9.17 / FIPS 171, to share working keys. NIST withdrew the FIPS 171 standard in 2005.
Analyzing the current scenario, you’ll notice security risks because of the way keys are shared today and sometimes because organizations are using deprecated standards.
So, let’s get card payment security into the twenty-first century!
Solution overview
AWS Payment Cryptography is a highly available and scalable service that currently operates within the scope of an individual Region. This means that the encryption keys and associated metadata are replicated across multiple Availability Zones within that Region, providing redundancy and minimizing the risk of downtime caused by failures within a single Region.
While this regional replication across multiple Availability Zones provides a higher level of availability and fault tolerance compared to traditional on-premises HSM solutions, some customers with stringent business continuity requirements have requested support for multi-Region replication.
By spanning multiple Regions, organizations can achieve a higher level of resilience and disaster recovery capabilities because data and services can be replicated and failover mechanisms can be implemented across geographically dispersed locations.
This Payment Cryptography CRR solution addresses the critical requirements of high availability, resilience, and disaster recovery for card payment transactions. By replicating encryption keys and associated metadata across multiple Regions, you can maintain uninterrupted access to payment services, even in the event of a regional outage or disaster.
Note: When planning your replication strategy, check the available Payment Cryptography service endpoints.
Here’s how it works:
- Primary Region: Encryption keys are generated and managed in a primary Region using Payment Cryptography.
- Replication: The generated encryption keys are securely replicated to a secondary Region, creating redundant copies for failover purposes.
- Failover: In the event of a regional outage or disaster in the primary Region, payment operations can seamlessly failover to a secondary Region, using the replicated encryption keys to continue processing transactions without interruption.
This cross-Region replication approach enhances availability and resilience and facilitates robust disaster recovery strategies, allowing organizations to quickly restore payment services in a new Region if necessary.
Figure 1: Cross-Region replication (CRR) solution architecture
The elements of the CRR architecture are as follows:
- Payment Cryptography control plane events are sent to an AWS CloudTrail trail.
- The CloudTrail trail is configured to send logs to an Amazon CloudWatch Logs log group.
- This log group contains an AWS Lambda subscription filter that filters the following events from Payment Cryptography: CreateKey, DeleteKey and ImportKey.
- When one of the events is detected, a Lambda function is launched to start key replication.
- The Lambda function performs key export and import processes in a secure way using TR-31, which uses an initial key securely generated and shared using TR-34. This initial key is generated when the solution is enabled.
- Communication between the primary (origin) Region and the Payment Cryptography service endpoint at the secondary (destination) Region is done through an Amazon Virtual Private Cloud (Amazon VPC) peering connection, over VPC interface endpoints from PrivateLink.
- Metadata information is saved on Amazon DynamoDB tables.
Walkthrough
The CRR solution is deployed in several steps, and it’s essential to understand the underlying processes involved, particularly TR-34 (ANSI X9.24-2) and TR-31 (ANSI X9.143-2022), which play crucial roles in ensuring the secure replication of card payment keys across Regions.
- Clone the solution repository from GitHub.
- Verify that the prerequisites are in place.
- Define which Region the AWS Cloud Development Kit (AWS CDK) stack will be deployed in. This is the primary Region that Payment Cryptography keys will be replicated from.
- Enable CRR. This step involves the TR-34 process, which is a widely adopted standard for the secure distribution of symmetric keys using asymmetric techniques. In the context of this solution, TR-34 is used to securely exchange the initial key-encrypting key (KEK) between the primary and secondary Regions. This KEK is then used to encrypt and securely transmit the card payment keys (also called working keys) during the replication process. TR-34 uses asymmetric cryptographic algorithms, such as RSA, to maintain the confidentiality, integrity and authenticity of the exchanged keys.
Figure 2: TR-34 import key process
- Create, import, and delete keys in the primary Region to check that keys will be automatically replicated. This step uses the TR-31 process, which is a standard for the secure exchange of cryptographic keys and related data. In this solution, TR-31 is employed to securely replicate the card payment keys from the primary Region to the secondary Region, using the previously established KEK for encryption. TR-31 incorporates various cryptographic algorithms, such as AES and HMAC, to protect the confidentiality and integrity of the replicated keys during transit
Figure 3: TR-31 import key process
- Clean up when needed.
Detailed information about key blocks can be found on the related ANSI documentation. To summarize, the TR-31 key block specification and the TR-34 key block specification, which is based on the TR-31 key block specification, consists of three parts:
- Key block header (KBH) – Contains attribute information about the key and the key block.
- Encrypted data – This is the key (initial key encryption key for TR-34 and working key for TR-31) being exchanged.
- Signature (MAC) – Calculated over the KBH and encrypted data.
Figure 4 presents the entire TR-31 and TR-34 key block parts. It is also called key binding method, which is the technique used to protect the key block secrecy and integrity. On both key blocks, the key, its length, and padding fields are encrypted, maintaining the key block secrecy. Signing of the entire key block fields verifies its integrity and authenticity. The signed result is appended to the end of the block.
Figure 4: TR-31 and TR-34 key block formats
By adhering to industry-standard protocols like TR-34 and TR-31, this solution helps to ensure that the replication of card payment keys across Regions is performed in a secure manner that delivers confidentiality, integrity, and authenticity. It’s worth mentioning that Payment Cryptography fully supports and implements these standards, providing a solution that adheres to PCI standards for secure key management and replication.
If you want to dive deep into this key management processes, see the service documentation page on import and export keys.
Prerequisites
The Payment Cryptography CRR solution will be deployed through the AWS CDK. The code was developed in Python and assumes that there is a python3 executable in your path. It’s also assumed that the AWS Command Line Interface (AWS CLI) and AWS CDK executables exist in the path system variable of your local computer.
Download and install the following:
It’s recommended that you use the latest stable versions. Tests were performed using the following versions:
- Python: 3.12.2 (MacOS version)
- jq: 1.7.1 (MacOS version)
- AWS CLI: aws-cli/2.15.29 Python/3.11.8 Darwin/22.6.0 exe/x86_64 prompt/off
- AWS CDK: 2.132.1 (build 9df7dd3)
To set up access to your AWS account, see Configure the AWS CLI.
Note: Tests and commands in the following sections where run on a MacOS operating system.
Deploy the primary resources
The solution is deployed in two main parts:
- Primary Region resources deployment
- CRR setup, where a secondary Region is defined for deployment of the necessary resources
This section will cover the first part:
Figure 5: Primary Region resources
Figure 5 shows the resources that will be deployed in the primary Region:
- A CloudTrail trail for write-only log events.
- CloudWatch Logs log group associated with the CloudTrail trail. An Amazon Simple Storage Service (Amazon S3) bucket is also created to store this trail’s log events.
- A VPC, private subnets, a security group, Lambda functions, and VPC endpoint resources to address private communication inside the AWS backbone.
- DynamoDB tables and DynamoDB Streams to manage key replication and orchestrate the solution deployment to the secondary Region.
- Lambda functions responsible for managing and orchestrating the solution deployment and setup.
Some parameters can be configured before deployment. They’re located in the cdk.json file (part of the GitHub solution to be downloaded) inside the solution base directory.
The parameters reside inside the context.ENVIRONMENTS.dev key:
{
...
"context": {
...
"ENVIRONMENTS": {
"dev": {
"origin_vpc_cidr": "10.2.0.0/16",
"origin_vpc_name": "origin-vpc",
"origin_subnets_mask": 22,
"origin_subnets_prefix_name": "origin-subnet-private"
}
}
}
}Note: You can change the parameters
origin_vpc_cidr,origin_vpc_nameandorigin_subnets_prefix_name.
Validate that there aren’t VPCs already created with the same CIDR range as the one defined in this file. Currently, the solution is set to be deployed in only two Availability Zones, so the suggestion is to keep the origin_subnet_mask value as is.
To deploy the primary resources:
- Download the solution folder from GitHub:
- Inside the solution directory, create a python virtual environment:
- Activate the python virtual environment:
- Install the dependencies:
- If this is the first time deploying resources with the AWS CDK to your account in the selected AWS Region, run:
- Deploy the solution using the AWS CDK:
Expected output:
- If the solution is correctly deployed, an AWS CloudFormation stack with the name
apc-crrwill have a status ofCREATE_COMPLETEstatus. You can check that by running the following command:Expected output:
apc-crr
Deployment time: 307.88s
Stack ARN:
arn:aws:cloudformation:<aws_region>:<aws_account>:stack/apc-crr/<stack_id>
Set up cross-Region replication
Some parameters can be configured before initiating the setup. They’re located in the enable-crr.json file in the ./application folder.
The contents of the enable-crr.json file are:
You can change the dest_region, dest_vpc_name, dest_vpc_cidr, dest_subnet1_cidr, dest_subnet2_cidr, dest_subnets_prefix_name and dest_rt_prefix_name parameters.
Validate that there are no VPCs or subnets already created with the same CIDR ranges as are defined in this file.
To enable CRR and monitor its deployment process
- Enable CRR.
From the solution base folder, navigate to the application directory:
Run the enable script.
Expected output:
This will launch a CloudFormation stack to be deployed in the AWS Region that the keys will be replicated to (secondary Region). Logs will be presented in the
/aws/lambda/apcStackMonitorlog (terminal from step 2).If the stack is successfully deployed (
CREATE_COMPLETEstate), then the KEK setup will be invoked. Logs will be presented in the/aws/lambda/apcKekSetuplog (terminal from step 3).If the following message is displayed in the
apcKekSetuplog, then it means that the setup was concluded and new working keys created, deleted, or imported will be replicated.There should be two keys created in the Region where CRR is deployed and two keys created where the working keys will be replicated. Use the following commands to check the keys:
Command output showing the keys generated in the primary Region (
us-east-1in the example):The following is the command output showing the keys generated in the secondary Region (
us-west-2in the example): - Monitor the resources deployment in the secondary Region. Open a terminal to tail the apcStackMonitor Lambda log and check the deployment of the resources in the secondary Region.
The expected output is:
- Monitor the setup of the KEKs between the primary and secondary Regions. Open another terminal to tail the
apcKekSetupLambda log and check the setup of the KEK between the key distribution host (Payment Cryptography in the primary Region) and the key receiving devices (Payment Cryptography in the secondary Region).This process uses the TR-34 norm.
The expected output is:
Testing
Now it’s time to test the solution. The idea is to simulate an application that manages keys in the service. You will use AWS CLI to send commands directly from a local computer to the Payment Cryptography public endpoints.
Check if the user or role being used has the necessary permissions to manage keys in the service. The following AWS Identity and Access Management (IAM) policy example shows an IAM policy that can be attached to the user or role that will run the commands in the service.
Note: As an add-on, you can change the *(asterisk) to the Amazon Resource Name (ARN) of the created key.
For information about IAM policies, see the Identity and access management for Payment Cryptography documentation.
To test the solution
- Prepare to monitor the replication processes. Open a new terminal to monitor the
apcReplicateWklog and verify that keys are being replicated from one Region to the other. - Create, import, and delete working keys. Start creating and deleting keys in the account and Region where the CRR solution was deployed (primary Region).
Currently, the solution only listens to the
CreateKey,ImportKeyandDeleteKeycommands. CreateAlias and DeleteAlias commands aren’t yet implemented, so the aliases won’t replicate.It takes some time for the replication function to be invoked because it relies on the following steps:
- A Payment Cryptography (
CreateKey,ImportKey, orDeleteKey) log event is delivered to a CloudTrail trail. - The log event is sent to the CloudWatch Logs log group, which invokes the subscription filter and the Lambda function associated with it is run.
CloudTrail typically delivers logs within about 5 minutes of an API call. This time isn’t guaranteed. See the AWS CloudTrail Service Level Agreement for more information.
Example 1: Create a working key
Run the following command:
Command output:
From the terminal where the
/aws/lambda/apcReplicateWklog is being tailed, the expected output is:Example 2: Delete a working key:
Run the following command:
Command output:
From the terminal where the
/aws/lambda/apcReplicateWklog is being tailed, the expected output is:See the service documentation for more information about key management operations.
- A Payment Cryptography (
Clean up
You can disable the solution (keys will stop being replicated, but the resources from the primary Region will remain deployed) or destroy the resources that were deployed.
Note: Completing only Step 3, destroying the stack, won’t delete the resources deployed in the secondary Region or the keys that have been generated.
- Disable the CRR solution.
The KEKs created during the enablement process will be disabled and marked for deletion in both the primary and secondary Regions. The waiting period before deletion is 3 days.
From the base directory where the solution is deployed, run the following commands:
Expected output:
- Monitor the Lambda functions logs.
Open two other terminals.
- On the first terminal, run:
- On the second terminal, run:
Keys created during the exchange of the KEK will be deleted and the logs will be presented in the
/aws/lambda/apcKekSetuplog group.Expected output:
Second, the CloudFormation stack will be deleted with its associated resources.
Logs will be presented in the/aws/lambda/apcStackMonitorLog group.Expected output:
- Destroy the CDK stack.
Expected output:
- Delete all remaining working keys from the primary Region. If keys generated in the primary Region weren’t deleted before disabling the solution, then they’ll also exist in the secondary Region. To clean up keys in both Regions, get the key ARNS that have a status of
CREATE_COMPLETEand delete them.
Security considerations
While using this Payment Cryptography CRR solution, it is crucial that you follow security best practices to maintain the highest level of protection for sensitive payment data:
- Least privilege access: Implement strict access controls and follow the principle of least privilege, granting access to payment cryptography resources only to authorized personnel and services.
- Encryption in transit and at rest: Make sure that sensitive data, including encryption keys and payment card data, is encrypted both in transit and at rest using industry-standard encryption algorithms.
- Audit logging and monitoring: Activate audit logging and continuously monitor activity logs for suspicious or unauthorized access attempts.
- Regular key rotation: Implement a key rotation strategy to periodically rotate encryption keys, reducing the risk of key compromise and minimizing potential exposure.
- Incident response plan: Develop and regularly test an incident response plan to promote efficient and coordinated actions in the event of a security breach or data compromise.
Conclusion
In the rapidly evolving world of card payment transactions, maintaining high availability, resilience, robust security, and disaster recovery capabilities is crucial for maintaining customer trust and business continuity. AWS Payment Cryptography offers a solution that is tailored specifically for protecting sensitive payment card data.
By using the CRR solution, organizations can confidently address the stringent requirements of the payment industry, safeguarding sensitive data while providing continued access to payment services, even in the face of regional outages or disasters. With Payment Cryptography, organizations are empowered to deliver seamless and secure payment experiences to their customers.
To go further with this solution, you can modify it to fit your organization architecture by, for example, adding replication for the CreateAlias command. This will allow the Payment Cryptography keys alias to also be replicated between the primary and secondary Regions.
References
- Move Payment Processing to the Cloud with AWS Payment Cryptography
- AWS Payment Cryptography
- AWS Payment Cryptography Service Level Agreement
- ANSI X9.24-2-2021 standard
- ANSI X9.143-2022 standard
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
Author: Ruy Cavalcanti