Delivering data into customer systems, securely

Context

When a crawl finishes, the delivery service pushes its files to wherever the customer wants them: their S3 bucket, an SFTP server, email, cloud drives, a partner’s API. To do that it has to hold the customer’s credentials: keys, passwords, secrets.

I have been one of its main maintainers since 2023. This case study covers the security and connectivity work I did there: how stored credentials are protected, how we deliver into a customer’s AWS account without holding their keys, and how new destinations ship without risk to the old ones.

Constraints

• Credentials at rest. Delivery settings are stored as parameter blobs in MySQL. Third-party secrets must not be readable from the database or its backups.
• Existing rows. Encryption had to be applied to every destination already configured, without breaking a single delivery.
• Customers’ rules, not ours. Their SSH key types, IAM policies and OAuth gateways are fixed. The service has to fit them.
• Many destinations, one dispatcher. A change for one destination type must not touch the others.

Design

Key decisions

Encrypt credentials — then make it invisible. My first version was application-level envelope encryption: each settings blob encrypted with AES, and the AES key wrapped with RSA. It worked, and a one-off migration encrypted every existing row. But every read and write path had to remember to call it. Three months later I replaced it with a custom SQLAlchemy column type that encrypts on write and decrypts on read inside the query, using MySQL’s AES functions. Any code reading the model gets plaintext, and the database never stores it. Scripts moved the existing rows from the first scheme to the second.

Never hold the customer’s AWS keys. For S3 delivery, customers can grant us a role instead of handing over access keys. The service first assumes its own delivery role, then assumes the customer’s role with an external id, which is AWS’s guard against the confused-deputy problem. One function handles the role assumption for both validation and delivery, so “test this destination” and the real delivery cannot disagree.

Fit the customer’s SSH keys. Some customers’ SFTP servers required ed25519 keys, and the old SFTP library could not use them. I rewrote the SFTP client on paramiko, the library underneath it.

New destinations are additive. For a partner API destination, I wrote the plan before the code:

• the new type is appended to the end of the enum, in protobuf and in MySQL, so no stored value changes meaning;
• it follows the per-destination pattern a teammate had set — a helper, an upload method, a validator;
• no existing destination’s code path is touched.

It shipped in a day, with OAuth 2.0 client-credentials auth, multipart upload and tests.

Failure handling

• Expired tokens. A 401 from the partner API triggers one token refresh and one retry. Other errors fail immediately, so a broken gateway does not get a retry storm.
• Validate at save, not at delivery. The first validator for the new destination missed required fields, so an incomplete configuration could be saved and fail at the first delivery. I fixed it the same day to reject incomplete settings when they are saved.
• Safe migration. Encryption skipped values that were already encrypted, so the migration could be re-run safely.

Result

Delivery credentials are encrypted at rest without any caller having to remember. Customers can receive data in their own AWS account without sharing keys. A new destination type is a self-contained change that cannot break the others.