Insecure practice of uploading TLS certificate private keys in to Certificate Manager
I am really surprised (rather appalled) to know GCP wants users to upload TLS Certificates and their private keys in to the Certificate Manager for being referenced by the services such as Application Load Balancers. Though its secured with encryption at rest, there are chances of unnecessary exposure by human error in handling the private keys at the first place and as a responsible CSP, I would expect GCP to do something about this.
At its face, it appeals to be a not-so-secure practice. I can understand even if the requirement is to upload PFX and password or even better an option to generate a CSR, that can be signed by an External CA and only the signed certificate need to be uploaded.
I want to hear the story on the side of GCP Engineering on their choice for users to upload private keys directly.
Though Certificate Manager is not a security product, I hope I can attract the attention of the Security Experts in contributing to this question.
Page 1 / 1
Thank you Seethar for your suggestions and caring deeply about security!
While private keys are encrypted at rest and transferred over an encrypted channel, we understand your desire to make the protection even stronger. For maximum level of private key protection and to avoid the issue of private key transfer altogether, we recommend using Google-managed Certificates whenever possible.
As we are aware that some customers have unique requirements that prevent them from using Google-managed Certificates, we also offer self-managed Certificates with private key upload. While CSR-based certificate upload is indeed more secure than private key upload, it wouldn’t fully replace the need for the private key upload - e.g. when a pre-existing Certificate needs to be uploaded. Also note that password-protecting the private key does not meaningfully increase security by itself, unless the password and the private key data are transferred over different channels.
I cannot comment about ongoing efforts, but rest assured that we're continuously monitoring and working on strengthening the security posture of our products.
Thanks again for taking the time to provide feedback!
Hi @stanwise_goog,
Thanks for your response.
I wanted to clarify a couple of things.
Speaking of unique requirements of some customers, I don’t see any other means to handle certificates for Internal Load Balancers. Are Google recommending Google-managed Certificates for them? I thought this was not possible. Even if they are, my internal domain names may get recorded in the Certificate Transparency logs, which may not be desirable.
I agree that there are scenarios where the public certs and private keys need to be uploaded, but the PKCS#12 standard helps here with the guideline to do this securely. Though uploading pfx file and password can be arguable not the best approach, it improves the security posture hugely compared to the current approach of directly uploading/pasting private key on the portal as plain text, which is not cool by any standard.
Hi Seethar,
To your point number (1), it appears that you are trying to use “Classic SslCertificates”, which indeed do not offer Google-managed certificates.
Unfortunately, the UI for Regional Internal Application Load Balancers doesn’t support them yet, hence this confusing message. You can either use gcloud or set up a Cross-region internal Application Load Balancer, which already supports Certificates from Certificate Manager in the UI, including the Google-managed ones.
I have to disagree with you on (2) though. Although transferring PKCS#12 file + password over HTTPS may appear more secure, it’s actual security is exactly the same as PEM key over HTTPS. In both cases you are actually relying on HTTPS and on the proper logs redaction to maintain secrecy.
Of course you can achieve a meaningful difference in security by using PKCS#12, but the password would need to be transferred separately.
In any case, I hope your concerns are addressed by the clarification from (1).
Thanks @stanwise_goog I think that is the missing piece.
With any approach, I could not find the ILB supporting those certificates in UI. From your write up, I infer that could be a bug and try that instead.
Due to compliance reporting overhead which I am pretty sure CAS supports, some of my clients are reluctant in changing their approach to Google CAS service. But this is a compelling reason for them to consider this approach.
For 2, I see the validity in your argument.
But logistically PKCS is a much better approach. Most public CA vendors let customers download the signed certificates as password protected .pfx file. To put it in perspective, for a Junior Engineer to handle so, seems much safer than letting them unpack the pfx file, copy the private key and paste it on Google Portal. It reduces direct human touch on plain text private keys. Simply put, it makes it harder for the person to go wrong.
I wish GCP Product/Engineering teams would consider Customers’ practical concerns.
Thank you for flagging the logistics/practical aspect.
I’ll discuss this further internally and see what we can do. I cannot promise you any timelines though.
