I've seen many environments that any developer has the ability to push code and to trigger a pipeline that uses production secrets. Without even needing the approval of the pull request or anything else, you directly push code to a pull request. It triggers the pipeline automatically.
You have inside tons of secrets to production environment. Maybe the pipeline itself runs on an agent that has access to production itself, and it can allow attackers to directly affect the core business of an organization in order to cause financial damage or stuff like that. Hey everyone, Yaniv Hoffman here back with another video and today I'm actually excited.
I'm hosting someone that I've been following for some time. He's a security researcher with vast experience in incident response in SOC, application security, and he's now passionate and researching a lot around continuous integration, continuous development. He's also a public speaker, spoke in DEFCON and many other significant cyber security events.
And today we are going to speak about code security, continuous integration, continuous development. Some project he's passionate about. He contributes into GitHub.
Welcome, Assi Greenholtz. Thank you very much. Thank you very much.
My pleasure being here. Thank you very much for inviting me to show. I'm super excited to talk about cyber security, CICD security in particular.
And let's see where this goes. Definitely. So, Assi, I'm very excited to have you here, distinguished engineer researcher like you.
And actually, I reached you following a post you did and some of your followers did in LinkedIn about a project in GitHub around code security, more specifically continuous integration, continuous development. So maybe for the newer generation cyber security enthusiasts that are joining this show, can you tell us a bit what is continuous integration, continuous development or CICD? What are the threats around?
And then we'll reach out to look in more depth what you have done in your project and how others can, you know, leverage and learn from it. So CICD, it's a term that refers to development methodology, basically means CI continuous integration. It's one process of developing, building, testing, basically doing unit testing, stuff like that, to develop code together as a team and continuous deployment or delivery, which are two different methodologies.
We are talking about the way that you take your artifact of code, whatever it is, and you deploy it or deliver it to the production environment and all the processes that are related to it. So when we're talking about CICD in general, we're talking about the whole development process from writing a single line of code up to deploying it to production server in the cloud or wherever you run your code. Those processes, of course, are very important for organizations and because they're so important, they also attract attackers and it's an area in cyber security which doesn't have very high awareness.
Not a lot of people in the cyber security area know about security issues related to this area and because of that and because this process involves in the development and deployment of products. Maybe it's like applications for phones, if it's a web application, some other kinds of programs because it's part of the business, the core business of many, many organizations. It can also allow attackers away inside the organization in many, many different techniques that we'll talk about probably later.
And it can allow attackers to directly affect the core business of an organization, maybe by reaching its production environment through the development process or interfering with the development processes of the company in order to cause financial damage or stuff like that. Maybe still some intellectual property, not from production environment, but from the development environment, which is usually less secure and less main focus of the defenders of organizations and so on. So this is a quick introduction to the world of CICD security.
Yeah, I agree. And I think, as you said, many times it overlooked. It's a subject that not too many are speaking.
Yes, there are many discussions around, you know, code security and the coders need to understand better security and secure definitely their code. But, you know, when I'm looking myself at CICD, it's a very delicate, rapid cycle. Time to market is very important.
You want to release your application very fast. As we speak, probably our application on the mobile phone are getting updated, right? And time is key here.
And for doing or reaching this efficiency, CICD was actually invented in a way. And if someone can penetrate into this cycle, infect the code, inject some malicious code, this can be a significant impact on the product, on the company reputation, harm to users, etc. So I saw you put a lot of you created new project around that in GitHub with a lot of information.
Maybe can you share some details about that? It's called the awesome CICD attacks. The idea for this project came from a daily routine I have since I've started being in the cyber security industry.
I'm reading every day. I have an app I use it's called feedly. It's basically RSS feed manager.
I follow many, many blogs, websites, different sources. And each day I read about all the new things in the cyber security realm. And I decided to create a project to create all the best resources I've read about in the last three years about CICD security.
Basically, three years ago, I started researching CICD. And I think I initially made this project as a map for myself. When I do offensive CICD security research to have like a menu of all the possible different attack techniques and tools, which I read of and I know and I know And to make it easier for me to start new projects or do new research.
So this is how this project was made. And I still updated update this project on a weekly basis with new articles and I put very high effort on only adding only high quality resources and stuff which are actually practical and can be used to learn or to do offensive CICD security research. Very cool.
So maybe we can you can show us some example about the techniques or maybe or the offensive tools or the cases. Yeah, so the techniques I split them to four groups. Maybe pick some of my favorite topics.
So from the first one, which is publicly exposed, exposed sensitive data. One of the posts I like the most is this one cloud quarry. They detail the process of how they found public am I's am I is basically a image of a virtual machine.
It's a format of AWS, you can create your virtual machine image and then use it to create new easy to instances and virtual machines. And you can also make the image public. Now when you create an image you if you use some credentials to download or install programs or stuff like that, it will stay inside the image.
Those are the secrets that you use. We can see the statistics of how many publicly exposed images there are in each region or see Total of three million images. Now, a main technical difficulty in this research was to analyze this huge amount of data.
So what they did here, they first reduced the number of image they want to analyze by filtering them through different categories. And eventually they had to actually analyze small amount of images, but it still took a very large effort because you need to use a lot of storage and a lot of processing power to go through all the files and all those virtual machine images because each one has probably like hundreds of thousands of files, something like that. So it's 3.
1 million exposed images, anyone legitimate or malicious person can reach out to. Yeah, and maybe manipulate or change. Yeah, and that's only like one type of service that can be exposed on AWS, the other articles about many other services that have potentially publicly exposed data.
And I think the reason I put this as the first title is, I think that's in a lot of cases, the first initial access vector that attackers in most cases use to hack organizations. We've seen a lot of different incidents in the world that started from publicly exposed secrets in many forms. I think every security professional knows that it's a bad idea to have a secretly publicly exposed.
And we have a lot of tools, open source tools to scan for secrets, all the types of secrets. We have tools to verify in real time for secret. It's actually active.
Still, we see a lot of secrets exposed in many different ways or services or stuff like that. And it's still a real issue for organizations today. And it actually creates a very easy entry point for attackers to breach major organizations by exploiting mistakes of individuals doing something by accident in the organization.
So what is called like, we know as a misconfiguration, which is maybe one of the most common. Yeah, the human human error. It's the basic issue that we face.
Okay, so what do you show in this research? Okay, you show that 3. 1 million amazing number exposes, you know, images in this related AWS.
They detail how they analyze these images and the bottom line. I think we'll see it somewhere here. In your terminology, when you say secret, what what do you mean just to make sure that the viewers are?
Secret is a credential to authenticate your system, whichever system it is, maybe a SaaS, maybe something internal. Here we can see the summary of the results that they found. We can see 2 million generic API keys.
You can see 100,000 AWS keys, a lot of secrets that they found. It's crazy from just this basic research, basic idea of scanning publicly exposed AMIs images. You find so many secrets and you can do it.
I read about researchers, research like that in all the platforms we know today. People found secrets in Docker Hub in now AI is very trending. So adding face contains also secrets, every possible service that can contain secrets in some way or form.
You'll find publicly exposed secrets. That's my conclusion. Regardless of the environment, of course.
I don't know how people will put secrets in very creative ways everywhere. So what in your mind if when you are, you know, consulting to organizations or speaking to to peers or developers? What is your best practice in such cases?
What they need to be aware of or how they can, you know, ensure they didn't do any misconfiguration that can lead to such an exposure? Yeah, so I think the strategy, the best strategy is a multilayered strategy. We need as an organization, we need to do multiple things to reduce the risk of the secret exposure.
One of them is to implement a rotation mechanisms or policies, make secrets. Every secret that is being generated, make it expirable in some way or another. Also, limit the people who can create secrets.
That's all about just limiting the attack surface, the possible ways that people can make mistakes. There are also like services which scan the Internet for you to find exposed secrets and stuff like that. But the most effective ways to prevent those issues is by creating policies inside your organization and enforcing those policies in order to make sure less people can generate those secrets.
Those people that can generate the secrets will generate secrets with very fine permissions, only the permissions that are needed and not something, I don't know, like you create an AWS secret with full admin permissions over the account and stuff like that. Also, limit where those secrets can go eventually. Many people inside that I saw used contractors or stuff like that.
So contractors usually use computers which are not part of the organization's network and are not monitored or governed in any way. So it usually leaks easier by their mistake doing, I don't know, like, for example, we had the snowflake incident not too long ago. Most of the companies breached there.
It turns out they used contractors and those contractors used their personal computer, which they also used it to download probably cracked software or some software that isn't legitimate, that contained the info stealers. And those third parties stole the secrets, probably sold them on the dark market. And somehow they got to different attackers that use them and exploited them.
I think this is the common case because if you have a network organization, like if you are a big enterprise and you have good policies to prevent leakage of data, you probably won't face those issues. Yes, definitely an interesting. So if we go back to the to the project, I see this is definitely probably one one topic out of many that I see here.
Right. It's not only although I see other ones with secrets. Yeah, there are very interesting ways, very original ways.
For example, this one about hitting GitHub commits turns out someone found a way to retrieve secrets from deleted commits. So you can't actually see the commit itself in the commit history of the repository. But there is still a way to get this deleted commit.
You know, there's always innovative ways hackers will find vulnerable ways to, I don't know, grab your secrets or get into your systems. There's always something that you don't know, I think, in this area. You need to be ready for anything.
What I love is you categorize it very nicely, you know, publicly expose sensitive data to maybe revealing the secrets themselves, the credential. And I see also the initial code execution that you can monitor and maybe manipulate the commits or action, right, as part of the CD. So each one also contain a repository of information about about how to to exploit or how to.
Each one of these articles is a world of itself, very complicated techniques, very long blog posts. And yes, as you said, I believe in simplicity, so I try to make as a small number of categories as I can. For example, if you look at similar like models, we have the MITRE attack framework, which has a lot of tactics.
I think it's very confusing if you can drill down and condense those categories to less titles. I think it's more simple for people to to search for and to understand what they're looking for and find the right data that they want. What else do we have here?
I saw in the list that you have like offensive tools or these are tools that can expose these scenarios, for example. Yeah, so these tools are kind of offensive frameworks like similar to, let's say, an interpreter. But we all know.
So we have Azure DevOps attack toolkit. We have some awesome tools to investigate Github profiles, to have some web services to search Github using Regex. Jenkins attacks attack framework also.
So tools which I use very commonly. Most of them, some of them are good, you know, for once in a once in a while, you need maybe to use them. But I found those tools very useful or maybe also inspirational.
If you want to create something of yourself, you want to get some ideas of what other people do. Those are also great, great tools. I think for someone that is interested in safety security, this is a great way to begin your journey and to understand how to attack those infrastructures.
Definitely. And you know, I'm now looking at it. It's not only at the beginning when I thought about it, it was more also for developers.
But not only, you know, if you are an ethical hacker, a pen tester, right? Cyber security researchers and people that are going after bug bounties or capture the flags, etc. It's super relevant and maybe overlook the subject that.
Yeah, yeah. So one one example I'm hearing a lot from friends in the industry that doing a penetration testing for organizations. A lot of them say when they start also do penetration testing for the environments and not just the classical penetration testing that is usually done.
They they're just amazed at how easy to hack those organizations through their CCD because probably because of the low awareness to these issues. And you can see it also on cyber conferences. For example, last year on DEFCON.
There's only the I think my my talk was the only talk that talked about up success security in practice. All the others, you know, very good talks, but different areas. But this area doesn't have much attention.
So organizations also don't put much attention in defending those areas. Right. I see it for my my work.
You know, it's mainly how to protect the infrastructure and networking. But the world changed, not only because COVID moved us to more online, but applications started to transit into the private cloud, the public cloud, the multi cloud. So you have now many applications scattered everywhere.
So how do you gain visibility? How do you manage it? How do you secure it?
The attack threat landscape became or surface became much wider. And this is definitely something that is overlooked and probably a very big common vector of attack that will only grow till it will get the right attention. Yeah, I think also in the market right now, there are not there are some tools that give you very good abilities to defend these areas.
But still, if you think as a application security engineer in any organization, you're still missing a lot of tools that can make your life much, much easier. Before my last job before being a security researcher, I was a security application security architect, basically a security engineer. And I've worked with development teams on protecting the development processes and so on.
It's so complicated, this area of protecting a company that now develops some kind of application doesn't matter which type of application you have so many areas and different issues you face as a security engineer. And you're most of the times you're under budgeted and understaffed and you don't have enough manpower to you know, maybe it's not possible to sit in every meeting about a new feature or a new epic that's being developed. I can say you know for sure that most in most of the times the developers that are developing those features, never read anything about security.
Most of the time, some of them, some of them know about security, but most of the developers don't know and most of the times even don't care about security. And that's where most issues come out of. Indeed, very insightful.
Asin, I'm happy we are raising the awareness for that. And I'm looking here on the case study. Maybe you have a case study that you can go follow up with us on the stages that were used to exploit or compromise the secret or impact.
So one very interesting attack method which was published, I think it's three years ago. I think it's one of the best bloggers I've read. It's by Alex Beerson.
He basically found a new attack vector that he called dependency confusion. And you can see here he used it to hack very big companies like Apple, Microsoft, and a lot of other companies. I can just tell you before we begin diving into the technique itself that even though it was published three years ago.
And it's something that most of the people that are involved in application security or CCD security know. It's still possible to explore this issue. Many organizations are still vulnerable to this issue.
You can say an old technique in cybersecurity terms. So what is dependency confusion? So let's say there's a big company that has an open source project.
For example, let's say Microsoft. And inside the open source project, we have a file that declares dependencies of that project. Now you can analyze those files and you can reveal which dependencies are public dependencies that are hosted on a public repository like npmjs.
com. And which ones are internal dependencies that are all hosted in internal registries of the particular companies because you can't find them on the Internet. So what Alex did is look for those internal dependencies.
And he looked if the name of the internal dependency is taken on the public registries. Why did he look for that? Because let's say I'm now building this project in order to create the artifact that runs the project.
Maybe create a docker image that contains the project and then deploy it to a production environment. The first thing I need to do is install all the dependencies. Now, if I'm using a misconfigured client, let's say a client using its default configuration.
And I now try to install this project, which uses internal dependencies. The package manager client will try to look for the package in the public registries because it doesn't know of my internal registry because I didn't configure it. It's a default configuration.
So when the name of the package is not taken, the name of the internal package is not taken on the public registry and the misconfigured client, which is configured by default. So if someone didn't configure it correctly for the project, tries to fetch the package. If I register this package using malicious code, it will actually cause the user that is installing this package or maybe the CCD pipeline that is installing this package to download and execute it probably during it.
Maybe if it's during the installation stage, which is possible, or during the execution stage running a unit test, unit test or stuff like that. So he found many, many instances of this issue. He registered those names in the public registries.
He managed to execute code in many, many major organizations. I saw this attack in real life in many organizations I've worked with. So it's a very common issue and it has a very simple solution.
Each internal package that you register in your internal registry, just create the same package, even an empty one in the public registry and that's it. That's how you stop the attack. You have this attack in many kind of flavors in different languages and so on.
But I gave here an example of JavaScript. And we have a very complicated diagram here basically explains the same issue. But this is the basic gist of the attack and you can see that by doing a very simple move as an attacker to register a package, you can execute code in major organizations, which is crazy.
It's very simple attack, which you can then use to do other stuff, even less, has less awareness than the security itself is the post-exploitation stage of the attacks. So most of the attacks that are being published today about the security, we see that attackers are achieving a way to execute code in organizations and environments. And what do they do each time?
They steal secrets from the pipelines or from computers. They steal source code. And that's basically most of the times.
But I think the more complicated attacks, which we don't see in the news, maybe because they cause a lot of damage or maybe because the defenders weren't able to detect because they're more sophisticated. I have much, much more high potential of impacting the organization because when I'm running inside a CCD pipeline, I can do much more than just steal secrets. I can also try to elevate privileges, escalate privileges by doing different methods.
I can move laterally around the network because usually those CCD pipelines have very high network accessibility to services they need to use during the testing or deployment steps. I can give one basic example. We have this product that's called Artifactory, which is a package manager used by many, many organizations to create those internal package registries.
And in many organizations, the configuration is that the whole registry has read permission for anonymous users or even for any authenticated user has read permissions to all the package of the organization. So what an attacker can do if it's able to run code inside the CCD pipeline and it sees a connection to this Artifactory instance, you can download the entire instance, send it out to the internet to a C2 server. The first thing you can do, you can steal all the source code of the company without even having access to the source code management system, just the Artifactory repository, which also contains source code.
It can also maybe sometimes have write permissions to maybe create code that propagates through the organization inside the Artifactory repository. And as we've seen earlier, probably some of those artifacts have secrets inside them. So the attacker can also move, can move to another step in the attack and elevate his privileges by finding another secrets in the artifacts and then connecting to a different service and continuing the attack, so on and so on.
This is just one example. There are a lot of examples that can be done in the post exploitation stage of the CCD environments. And this one is also not a very, it's not an area that has high awareness of.
Usually people think of when they think about CCD security, they think about stealing secrets and that's it, but it has much more, much more of it. So I can give another example. Most of the times when those development environments are described in a security context, people are creating graphs, which are pretty simple.
You have source code management system, which is connected to a build system. Let's say Jenkins that is connected to a Jenkins and you have an Artifact Registry, Artifactory or NPMJS or whatever it is, and you have the cloud. And that's it, very simple process, but that's not actually the reality.
From research I did, I found a lot of systems are connected to this process. I can give some examples. Databases are usually connected to CCD systems, logging systems, secret managers.
We have all the cloud providers have vaults, some kind of vaults. They don't seem related to CCD. For example, let's say Jira.
You can automatically create tickets to Jira, send slacks. So a lot of systems which you don't think they're directly connected to a CCD pipeline are actually connected and they have secrets and attackers can use them to move laterally inside your organization. If we talk about slacks, one technique can be if you get the ability to execute code in a pipeline and you find a secret for slack, you can maybe use it to send someone a message to grant you more permissions in a particular account and exploit also the human factor.
You can do a lot of things, really a lot of things to make those attacks much more complicated and cause a lot more damage and harm organizations. No, it's amazing. I'm in this business more than 25 years and thinking, you know, my expertise is infrastructure application.
But definitely, and I'm looking a lot about excessive permission, many things you say they're misconfiguration. But indeed, the CICD, this area, the sophistication, the ability to get to get the entry point into the organization is overlooked in many, many cases. Even, you know, I'm not thinking about it too much.
You open my eyes here and it's very similar to all the known APTs we know of. I also read about a lot of I read about all of the attacks that happened in the world. They have I had this kind of obsession to read about any new APT that came out and to read the technical details.
I think most of the attacks started by phishing. And I think in this context of CICD, you can also start attacks in the form of phishing. Basically, you need to phish one developer account to get access to an environment, which usually organization think it's it's difficult to compromise one developer.
But if you're a big organization that has thousands of developers, we know the statistics of phishing. You'll probably be able to phish at least one developer, if not more. And also in those development environments and developers usually have very broad permissions and they're usually not correctly managed.
So if you get an access as one developer, I've seen many environments that any developer has an ability to push code and to trigger a pipeline that uses production secrets without even without even needing the approval of the pull request or anything else. You directly push code to a pull request. It triggers the pipeline automatically.
You have inside tons of secrets to production environment. Maybe the pipeline itself runs on an agent that has access to production. It runs also maybe there are also scenarios where the agents that run those CICD pipelines are used for multiple builds.
So they're not ephemeral. They're not deleted after each build. And you can also access other projects that are running in parallel to your project.
It's crazy. You have a lot of different ways to spread. You scale from there and spread from there.
Definitely. So I see. Thanks a lot.
Very insightful. Everyone that are interested. And this is not only for developers, guys, it's for security researchers, pen tester and everyone that is interested in cloud in operation in new ways.
And I'm saying legally. We are doing only legal things. We are not supporting illegal activities.
Can go to the GitHub project of ASI. We'll share the link in the description on the screen. And now there's a huge amount of repository.
Very simple to to to understand because it categorized very nicely with the research, the articles, the tools, the use cases. And definitely try to contribute even more. I'm sure ASI will be more than happy.
Follow ASI as he's speaking in many events and many cases. Not only about the CICD is passionate about that, but doing many other things. I will leave also some links for ASI public speaking events like Defcon and others.
If you are not subscribed yet, please do. If you like the video, if you like ASI to participate again, to speak even more technically, to do a deep dive. Let us know.
I'm looking you ASI for future cooperation. And thank you very much. I see.
Really appreciate your time today. Thank you very much for hosting me. It was a great pleasure and a lot of fun talking about the service security.
You need to do it more. Thank you. Thank you.
And for everyone. See you in the next video.
