Evolution of identity first security and secret zero problem

[Music] Hello everyone and welcome to another episode of the identity navigator. This is your host Rohit. So before we deep dive into the topic for today, uh let's talk about what I have been up to in this new year.

So New York has been kind to us so far to me and my family. Uh it's very cold where we live in New Jersey and we have had a few inches of snow. Uh the New York obviously uh there were some bad news in the papers about some layoffs in Google and Wayfair and some of the other tech companies.

So, uh I would request you all if if you are in a position to help anybody out, send a referral, write a reference, uh please do that. And hopefully this this eases out really soon. But uh not to put a damper on today's issue, let's dive into something that I'm really passionate about and I've been really watching this space closely for quite some time.

uh and at the same time this requires so much of expertise across so much of uh different platforms that it is very very hard to keep up with this. So today we are going to talk about evolution of identity first security and secret zero problem. So all of us have heard the phrase or must have heard the phrase that identity is the new parameter.

Now even though this is true in theory, identity security maturity hasn't matched up with the security that network security provided uh in 1990s and early 2000s. Let me ask you a question that you should ask your privileged access management team. So the services that needs to connect to the database connect to the vault fetch the DB credentials.

But how does the app authenticate with the vault? Why bootstrapping especially? I'm pretty sure if if you apply first principle thinking to this problem and ask this with your privileged access management team, you would be able to better understand and remediate some of the problems uh with with your privileged access management program.

So in this episode, we'll dive into the evolution of identity first security. So when the internet first became widely available in 1981, it had just 213 different servers and the usage was mostly for academic purposes. So the original internet use cases were focused on academia with the intent to share information and not prevent access.

Then we started using computers for business and the use cases imploded. But still we relied heavily on physical parameters and physical attestations for asurances that the individual accessing the network were authorized to do so. Insider threats were either minimal or entirely unknown.

Now when we think of security we visualize walls and and guards or at least I do. And for some reason they are always Roman guards. I don't know why.

So when we were uh when the use cases were evolving from academic to businesses, we emulated the walls and guards with firewalls and network segmentations and private address spaces ACL with the number of access points being pretty minimal and clearly defined. This was the best solution. So as use cases evolved from academic to businessoriented use cases and software evolved from monoliths to microservices, security became a barrier to growth.

Why do we need to move towards identity focused security and why are traditional parameters patterns are coming up short today? So as the decades have passed, conventional parameter defense patterns of yester years are not well suited to the evolving computing architectures and boundaries of modern technologies which there are none. Point solutions and technologies have stacked one onto other to cover growing cracks while underlying network security concepts have failed to follow modernization trends.

We no longer have monolithic apps. In the world of microervices, services can be individually developed and deployed. Can exist on premise, can exist on cloud, could be hybrid in nature and can be scaled up and down.

You can't you now cannot draw a precise line around the services that needs to be secured. And that line my friend was the network. And another thing is we don't write everything from scratch.

Either you are using commercial of the self solutions uh or or you are writing your own but definitely you would be including some third party libraries. You cannot trust that software is secure. For reference the national vulnerability database reports more than 15,000 new software vulnerabilities every year at an average.

Imagine this 15,000. So I can say it with pretty much certainty and actually put my money onto it that your softwares would have some vulnerabilities. You cannot look at these security gaps or vulnerabilities as flies that you would be able to swat away with when point of tool.

These are like bees, right? Just just coming one after another, right? So another thing that is you know that insider thread and social engineering is a real threat right now and I don't need to tell you all about this and last but not the least change in working models and pace of business has made it mandatory to allow access from multiple places.

Now if you can get into the door so can the attacker. If we decide to operate in 1990s onromised data centers, no VPN access, no access from outside the office, the parameter is still our best friend for rest of us in 2024, not so much. And you know in addition to all of this uh better ways of working there was also this cloud movement and requirement of doing more with less and then DevOps which somehow has metamorphic evolution into an automation.

I don't want to even get into how many tools you have to learn as a DevOps engineer nowadays which is absolutely crazy but that would be my rant for some other day. But obviously cloud and you know faster pace of business and devops has increased our security challenges. Now everything is fast today on demand and this highly dynamic and elastic environment broke accepted parameter security concepts requiring better service level interactions that was agnostic of the underlying network.

Conventional boundary enforcements used IPs and ports for authentication, authorization and audibility. Uh which under cloud computing paradigm no longer clearly mapped to workloads. Cloud patterns such as API gateways or manage load balances for multi-ervices workload required that identities are not dependent on network topology or paths.

So protecting the integrity of these serviceto-service communication has become more important particularly with teams needing a uniformity across workloads. Now for human identities it was probably simpler to solve right we solve these problems with multiffactor authentications and behavioral analytics and conditional access policies. But what about machine identity or workload identity?

With increased adoption of new technologies such as containers, microservices, cloud computing, serverless functions and I can go on and on. We need to enable serviceto-service access control. This is at the root of the problem.

The services should be able to identify each other especially the services which are being bootstrapped. We also need to think of authorization and not just authentication. We need to think in terms of policies.

So how should these services interact with each other? I would say shared secret is something that comes to mind. Shared secrets such as password and API keys provide a simple option for access control and distributed systems.

Now hold on a second. I'm not saying that network cannot solve this issue. But do you want to have a dedicated team of 100 people opening ports and closing ports and opening firewalls and closing firewalls and creating firewall rules 24 + 7 so that they can match the needs of your DevOps team.

Now if the answer is you do not want 100 people doing it around the clock then probably you are in the right place. But if that that works it just doesn't work at scale. Now, so we were talking about passwords and secrets being a viable option for access control in distributed systems, but password and API keys can easily be compromised.

If you don't believe me, uh try searching GitHub for a phrase such as client secret and see what happens. Another issue is in in large organizations, secrets can be hard to rotate in response to a compromise since every service needs to be aware of the change in a coordinated fashion and missing a service could and in most cases would cause an outage. So you shouldn't store these secrets in your app which poses a very interesting problem.

And this problem is a turtle zero problem. The origin of of the name is from a story about a woman who interrupted a philosopher's lecture to tell him the word rested on a turtle's back which I completely agree with. Yeah, why wouldn't I?

So the word rested on a turtle's back and then the philosopher asked her what's the turtle rested on and she said it's turtle all the way down. So what does it means for the identity fox? What what does this turtle zero or turtles all the way down means for us?

Think of it this way. Enabling access control to a resource such as a database or a service requires a secret such as an API key or a password, right? So to access the database, you would need some sort of secret.

Let's say it's an API key or a password. So that API key or password to the database needs to be protected. So you could protect it with encryption.

But then you still need to worry about the secret decryption key. The key that has encrypt encrypted it or key that would be able to decrypt it. So now you have to store this or securely store this secret key or secret dis decryption key.

That decryption key could be put into a HSM or a secrets repository or a vault. But then to access that key, you would need to access that vault, right? So there is another secret that you have to store or safeguard the secret that gets you into the vault.

So you see this this flow of ultimately protecting access to one secret is resulting in a new secret you need to protect. So we are not solving or we are not solving for the root problem. We are delegating.

We need to find that turtle on which all other turtles stand in our language. We need to find secret zero. So how do we define secret zero?

So before applications can retrieve secrets from a secret service or a vault, they need to be given a secret from which they can authenticate. Right? So your applications needs to access a database.

That means application needs access to database password. But you do not want that password to be stored within the application. So you stored it within the vault but now application needs a password or an authentication mechanism to the vault.

Right? So this is a bit of a chicken and egg problem and this is called secret zero in our language. So what are the naive solutions we can come up with to solve this?

Right? Let's manually copy a secret or a config file after the instance is booted. But it would be a nightmare to scale this solution.

Or we can just encrypt the secrets. But then how do we get the decryption key? Or we can host the secrets somewhere at a hidden URL.

But then now the hidden URL is a secret that needs to be protected. So most solutions and most of the teams even today not most but at least a very high percentage of the teams today are just changing the secrets that they are protecting right and if you go down the chain there is always a secret that they are not protecting. So you protect one secret, you have another secret to protect.

You protect that secret, you have another secret to protect. Executives ask you, are you protecting that? As the long does the answer is yes.

Who cares what happens to the other secret? So it's like turtles all the way down, right? Storing secrets in the cloud is relatively easier, right?

uh and how do we solve this identity problem? So we'll come back to the cloud in a second. But how do we solve this identity problem, right?

Like how do I identify in that the service which was bootstrapped? What is its identity? Should I be able to trust it?

What services can it authenticate to? What is the authorization level of this new service that has just come up? Right?

Traditional remote authentication schemas includes username, passwords, client token, HMAC, TLS certificates. But all these schemas involve proving possession of a secret, right? So how do we solve this bootstrap identity problem?

Why don't we use IP for identity? So when a service comes up, an IP should be associated with it. Let's use IP as an identity.

But then there are challenges associated with it. What happens if your services are behind in that? In that case, the calling service would just see a 100 IP rather than a 192 IP, right?

How do you scale? So if all your deployments is in cloud, then this becomes easy, right? uh you remember I spoke about cloud but what do I really mean here that if you could have a central repository of everything that is coming up or going down uh if you have a federated trust model or remote attestation in language of AWS then this would work right so in the cloud and I'm specifically talking about AWS but it holds true for GCCP and also not the cloud but the same concept applies exactly to Kubernetes is in the cloud our provider knows what application images are running and where.

This means the cloud provider can facilitate remote attestation. Right? In AWS instances can request a metadata document signed by AWS.

AWS already is aware of that instance and this metadata document that they request could be used to prove their identity to a secret service or a vault and this document is unique to each instance. Let's say it's an EC2 instance. So it would be unique to that EC2 instance that calls it and we can use it to prove what code is running or what AMI is running.

So imagine AWS is aware of every instances that is coming up. So instance can go to AWS and ask for a document that could be used to prove its identity to all the other instances in the ecosystem. It could be used to prove its identity to a secret service or a vault.

So you know if it needs to talk to a service B it can if it used if it needs to have the password to service B it can go to the vault and get those password by proving its identity because the vault or the secret service trust AWS and this also results in mutual TLS throughout the cloud ecosystem. Right? So now in this case that means that you know that turtle zero becomes our secret service but it can bootstrap itself right so let's say there are three instances of this service and one of them goes down a new service could be spun up it could connect to AWS uh get that verification of documents and then the other instances of this service will trust it and and unless you know there there is an apocalyptic state or there is an apocalypse in which the entire secret services goes down you would have solved this problem.

So solving the secret storage problem or secret zero problem meant that we had to solve the problem of bootstrapping identities as application startup. But as a bonus, this resulted in U reusable identities throughout the ecosystem. A mutual TLS the secret service itself is also called a secret service client and it could be used to be bootstrap its own master key.

Right? So that means it is end to end scalable and self-healing solution. So we now have a clear simple answer to the question where do I put my secret?

put it in the secret service. Secret service is autohealing. AWS could assign an identity to any application that is booting up.

So you have essentially solved the secret zero problem, right? But we generally see three methods of managing access today. Cloud IM works well for controlling access to native services at a single cloud provider but breaks down once you start crossing boundaries into other clouds SAS services or even your own onremise software.

Right? So if all of your services workloads are running on a single cloud provider, you don't have this problem or you have a way to solve this problem. But you have a if you have a cloud IM solution uh that spans across multiple providers or you have a hybrid nature or if you have SAS services then you need to really look into how do we want to solve this problem.

Uh another method of managing access today is via some sort of vaults. Now they are great for storing secrets but using them implicitly assumes that you understand the identity of the client requesting access. So vault would need to understand the identity of who is requesting access.

Further they typically require effort and maintenance from your dev team to integrate into your application. So let's say you have to uh integrate a vault such as a hashy core with I don't know AWS RDS you would have to create that pattern so that the AWS can integrate uh with your vault and DIY this is actually what we see most during today uh especially in the DevOps space doityourself approach generally includes a set of manual configurations best practices and allow list that don't scale well and can easily fall apart during an incident. So we see that we have solved this problem for a singular cloud provider.

But what about hybrid and multicloud environments? So the situation is not as bleak as it sounds. And let me talk to you about four of the solutions or tools that exist out there today that you can leverage to solve this problem.

So first is an easier approach that I'm trying out right now. Let your AWS service manager or systems manager manage your onremise workloads. Right.

So your VMs and your servers which are hosted on premise you can have your uh onremise instance be managed using AWS systems manager and terraform. So now when you you will have an uh sort of agent or an SSM installed in your data center when a new virtual image is coming up. This would be able to this would be identified by that virtual manager and then AWS system manager would be able [Music] to control it or you know operate it or manage it very similar to how it manages an EC2 instance in the cloud.

There is another tool which is called a keyless. So a keyless removes the need for secret zero entirely through its uh something called as a keyless universal identity feature which is packaged between the within the a keyless vaultless platform. So in this setup every machine can identify other machine in the network to ensure that the data received is authentic.

A keyless uses its own plug-in right to allow the vault and environment to interact in a secure fashion. The process begins with a starter token which is created by a human employee. So that is the first token is created by a human employee that is used once to authenticate the plug-in.

From there a keyless issues its own token and begins authenticating applications. The token is replaced by a new one in the next use for a specified amount of time. Whenever a new entity is registered under this system, it inherits the identity and token of the original identity or original entity.

This constant cycle of temporary rotating identity tokens is a secure alternative to use uh instead of using a single secret zero. So it's pretty smart uh in my point of view. Obviously you'll have to uh see if it makes sense for your organization because obviously you need a plug-in.

Uh there is a learning curve here. But if you are interested in solving this problem uh it's it's worth looking at a demo from you uh from them. Another tool that I have recently come across and testing out or hope to test out very soon is called ambit.

So they claim to have built the industry's first workload IM platform, right? Uh I looked at some of the people who are involved in this startup. Super impressed and they are super smart people.

So I'm I'm hoping that their claims would have some standing behind it or some weight behind it. So as I said they claims to have built the first workload IM platform to safely allow your applications to connect to any services or applications or APIs that they need. So this is very much what we have talked about their solution is based upon identities instead of secrets.

They give you a simple secure and central way to provide policy based access instead of needing to manage low-level secrets. So they claim that that we should be thinking of them as an octa but for workloads. So like octa is for identities or human identity.

Similarly they act like an octa for workloads. So instead of user to workload access ambit manages workload to workload access. It's pretty smart I would say.

Uh ambit is designed to be the workload identity provider. They broker between workload identity and service credentials using policies to control access. They do this across trust boundaries giving you a consistent implementation everywhere you operate.

So just a quick note, I am sadly not sponsored by either AWS, a keyless or ambit and you would have to validate some of the claims but this is what I see. So I believe how ambit very very smart way of doing it. Now how successful they are uh would really depend upon you carrying out a P but I think they suggest or are proposing a 30-minute demo and if you are in this space maybe worth your time and then comes the giants the Hashior Vault uh the the leader in this space for quite some time now and and still going very very strong.

So for cloud infra AWS we have seen uh we have similar solutions for others but even their conjure solutions works great. So if you have a cloud infrastructure or multi cloud infrastructure or if there is a workload identity store conjure the hashior solution works absolutely fantastic. Again I'm not sponsored by Hashi Corp and I'm pretty sad about it.

I want to actually test out hashior core with ambit. So conjure with ambit and see if ambit can act as a uh workload identity store or identity provider and then conjure can work across uh different data centers and different cloud providers. So I will keep you all updated as I carry out this PC's.

Uh I do need to have access to all of these tools. So I'm still trying to figure out how to set up that environment but if I can figure it out I will let you all know but Hashiko do also provide some other or proposes some other solutions uh to solve this problem. Uh there are like three solution approaches that they propose.

Authentication to the vault could happen via underlying platform identities. So using cloud provider IM roles or Kubernetes services or accounts, right? The second solution that they propose is by bypassing authentication entirely and using a token provided directly to the application which is not great in my point of view.

I don't think this is a solution even worth mentioning but they do. And then you know there is a new solution that they are proposing which is leveraging PKI and Eggme. So they do have a ton of uh features which I looked at in in their platform identities of how can you integrate.

So you can integrate via app role or or alleloud or AWS Azure cloud foundry github and jot or ydc or keraros and kubernetes and whatnot. So they do have a ton of integration patterns for platform identities. uh so I would strongly advise if you're going this route uh use jot uh pattern and just quickly oath flow is mostly used for users but jot flow can be used for machines with relative ease right so they have this blick flow about jot authentication which is um I would say simpler than the traditional oid or oath flow and vault only needs to verify the jot signature against the public key of the issuer and uh there are other features like bound claims that you could use to narrow down the specific entry uh like you know which pipeline has access which project has access uh reference etc.

So it al sounds great uh in theory but the problem still remains the same as to what is platform identity and how do you give access to specific pipelines without doing a ton of pre-work. Their app ro method uh is not even worth mentioning. It's like uh you know app role generates role ids or secret ids.

Role ID is like a username and secret ID is like a password and then they uh send this to the app via two different channels. So this is good but you know this pro this solution has so many problems associated with it because other apps in the same image can get hold of that secret. Right?

Uh and who is embedding that app role within the image? This goes away when there is a restart. Right?

So you have to embed that once again and it is absolutely a maintenance nightmare. You cannot scale your solution using app role method. And then there is acme method which is relatively better.

So you might have heard this about automated certificate management right acme is a protocol uh I think that was created by let's encrypt and it automates your uh certificate life cycle uh management. So you can use vault as an acme server in this solution and and you can really deep dive into this. Uh they do provide a solution architecture using uh acme and do propose it as a secret zero for flow.

So uh if the platform ID is available I would say use it right for Hashi Corp. Uh platform ID is available you can use jaw token and that would work great. That's an absolutely fabulous uh problem uh fabulous solution as long as you have the identity of the platform established.

you can use their jot integration to solve the the secret zero problem and that is where I feel like you know establishing the platform identity could be actually delegated to ambit uh which could be used as workload IDP and then we can use conjure or uh other hashore vault to use jot integration patterns then if the PKI is available uh platform ID is not available right I would suggest uh you and you know automate it with Acme and use it. If you cannot do it, you would then have to probably come back to some sort of you know passing the buck down the line uh that solutions would be implemented they are being implemented today. I've seen more than uh 60% of the places implementing it passing the buck because let's say there is an application right an application connects to the database.

So the question that I ask my team or the executives ask my team are is the application storing the credentials of database and the answer is a resounding no because the database credentials are being stored in the vault but the question which is not being asked is how is your application authenticating to vault especially during the bootstrap time right so so we did discuss some solutions or problems around it So what sets modern workload IM platform apart is their ability to automate the access life cycle of secrets completely and securely. Right? You need to think about scale because regardless of scale every new workload should automatically receive a unique identity with access managed according to predefined policy.

So, so in an ideal world and and we have been able to achieve it in some of the cases as soon as an application comes up an identity should be established and you should have some sort of conditional access policies or behavioral analytics behind the scenes as well. That means you know uh similarly like how you create conditional access policies for your user in Microsoft or Entra right uh based on your time of day or locations or device and postures your devices will have their own behavior as well and we could carry out behavioral analytics for workload. So there is an entire policy based access control engine that can stand behind the vault.

So something comes up It has an established identity, right? That identity is known to the vault. It can come to the vault.

Vault can have a policy based engine behind it based upon uh you know the different policies that we write or it also could have a behavioral analytics engine behind it. It could identify the risk posture and based upon that the access could be provided or the secrets could be provided to your applications right with relative ease. So I would stop here.

Uh we'll probably deep dive a little more into some of these solutions when I could uh I could finish up creating my own lab. And also there are some other things that I would like to discuss. But we'll stop here for now.

Thank you for listening. Uh I hope you found it as fascinating as I do. Um if you are trying to sleep, have a good night.

Uh if you are in the gym, uh I hope your workout was good. If you are driving, drive safe. And if you are just making fun of me, I hope you had a good laugh.

Uh you can always reach out to me via email or LinkedIn. Uh my email ID is the identity navigator@gmail. com.

And until next time, this is Rohit, your identity navigator.