DEF CON 32 - SQL Injection Isn't Dead Smuggling Queries at the Protocol Level - Paul Gerste

welcome everybody to SQL injection isn't dead smuggling queries at the protocol level or how I like to call it SQL injection lower decks because usually with SQL injection you're at the high level like at the bridge where the captain and the high ranking officers are and they're speaking their query language and their business language but today we're going to do go down to the lower decks where the mechanics are and where they shift around the bits and bites so to hook you at the beginning I have a small teaser this code snippit is a small

HTTP request Handler written in go and it just takes a user ID from the body and uses it in a prepared statement to select the user and if you think everything's fine here there's no way for skl injection then you should stay because I'm going to prove you wrong and at a high level in this talk we're going to look at how applications talk to their databases over the network and how attackers can inject in that connection and what can go wrong and also how prevalent the problem is a bit about me I'm Paul I'm

a vulnerability researcher at sonar's R&D team and if you don't know sonar we're the home of clean code so we help developers Write Clean code which also means secure code and that's why we look at a lot of code and uh try to find stuff there so for the rough outline first we're going to look at the idea of all of this research and then we'll go into the main part of the talk which is is attacking the database wire protocols and we're going to focus on post grass and mongodb here then we'll put things

into perspective uh by looking at the real world applicability of the research and then give you some ideas for future re future research and draw some takeaways so in short the idea is to do request smuggling but for binary protocols and of course for this I have to mention James kettle's HTP DC attacks where you BAS basically cause a disagreement between two systems over the end of a HTTP request for example between a reverse proxy and an upstream backend and if they don't agree on where a message ends or a HTTP request in this case

then the attacker can do some bad things and the root causes of this were many fold but some examples are text parsing where one system would only accept the exact word chunked and the other one would also allow a tapped character in front uh or they were logical where one system used the content length header and the other used the transfer encoding and then things were off again but what about other protocols and specifically what about binary protocols uh where do they have M message boundaries how do they do it um some of them have

delimiters where for example in null terminated strings the null bite is the delimiter that tells the end of the string or you might have length Fields uh and there's an entirely entire family of protocols that are called tlv or type length value protocols because I have a type field and then a length field and then the rest of the message is the value so how could we desync in these protocols uh for the limiters it sounds quite easy you just have to find a way to put a delimiter where it's not supposed to be in

a value of course it's not that simple but at least you directly have an idea of what you could do uh but for length Fields I was stuck for a moment and thought about okay you have a length field it has a fixed size it has a fixed location in the message so what could go wrong there and then I thought okay maybe some indianness issues or maybe uh integer overflows and I wanted to go deeper into that so for that I first looked at the Modern web application landscape where are binary protocols used there

so at the center you always have the application and it talks to databases or caches uh storages message cues structured logging and so on and I wanted to focus on connection between the application and the database uh and that was because I thought that if I would find something it would have a great applicability because every almost every web app has a database um then there's a big potential severity because databases are high value targets because they either contain interesting data like personal information or relevant data to the application for example used for authentication and

then also the exploitability should be quite good because what web app has a database but doesn't let the user query something from that database right so we have kind of guaranteed user input so let's start in and look at some uh database wire protocols and for that I have a small overview of four uh popular databases and how their messages in their protocols look like so for postgress this is how a message looks like uh you have a type byte then a 4 byte length and then develop which in this case is just uh the

string that's an SQL statement for MySQL it looks kind of similar your first have the length it's only three bytes but then you have a one bite sequence which would allow you to send multiple packets that would be reassembled to one big message at the end and there Al of course Al you have the value for redis which is not exactly a binary protocol but their docs kind of read like this uh because they talk about oh here the have this bite so I wanted to include it anyways there you first have a type bite

then you have a length but it's not a fixed size field it's a decimal integer that has to be pared then you have a delimiter and then the value and then another delimiter and finally for mongodb um it's a little bit longer you first have the message length again a 4 byte field then you have a little bit more metadata for example the op code is like the message type and then the value is a little bit more of a complicated binary structure called B on so let's start with postgress uh again this is how

a packet looks there one byte type identifier 4 byte integer length field and then the value so I thought about okay 4 byte integer Fields you can fit quite big numbers in there like 2 to the^ of 32 minus 1 which is roughly the size of 4 GB but maybe what if we could somehow make a query that's bigger than that how do actually libraries handle this um this kind of too large thing and for that I started to look into code of postgress client libraries and I found the first bug this is the code

of a library called PGX it's written in go and this function is supposed to encode the content of a of a message into a larger message buffer so first it writes the message type in this case it's a B for a bind message then it saves the offset where it should later write the size to then it adds all of the actual data of the message and finally it writes the size and to do that it first slices the message buffer to the exact size of what is included in the length for example not the

type bite then it takes the length of this sliced buffer and here the length is of type int which in most modern systems is a 64-bit integer but then this is truncated down explicitly to an in32 so we have a problem here because if the length was bigger than what can be expressed in an in32 we now have a truncation and the number becomes very small but then the library still sends all of the data after that so then if the database has to parse this message it will just see a very small length and

only parse a very small message and then try to parse whatever is coming after that as more messages but in fact it's data from the first message so let's look at this again with example messenges um so here we have a very small one we have a four by value and then the length becomes eight because the size of the length field itself is also included in the length so it's a length of eight this is how the largest possible message looks like you can see the length is filled to the brim with hex FF

because the value is just almost 4 gigabytes of Ace but still everything's good because the application can Express this length in the four bytes and so the database will parse the same size but if we make it just a little bit larger we can see that the truncation happens because the most significant bit is not represented in the length anymore so the length is just four which means this is a tiny message and then after that in the connection there come some garbage A's that the database can't really understand but if the attacker chooses not

A's but some bites that the database can understand they can craft an injected message here so let's zoom out again and look at this again this is the small message all is good this is the large message that does not cause a truncation so both the application and the data base see it and pars it the same way but if we make the size a little bit too big the application thinks it's writing this big number but actually because of the truncation bug it writes a small number so when the database pares it it sees

one small message and then garbage or if the payload is cleverly crafted it sees more messages that can contain attacker controlled SQL statements like this one that adds a new admin user so the impact is quite High here um you can inject entire SQL statements you're not limited to Union based or subquery based SQL injection like with the very classic ones it's more like if you have stack queries enabled so you can just finish the first statement that you're injecting into and then have a whole another one that can be an insert even if the

other one was a select and so on so this means the attacker can essentially read write and delete all the data in the database with the permissions of the application and if the database is configured insecurely this can already lead to code execution um or you just have all of the data from the database which is bad enough the only thing that's a little bit less convenient than with classic SQL injections is that data exfiltration is a little bit uh less straightforward because if you're injecting more messages into the connection then the application will still

only process the answer for the first message so you don't directly get the result of your injected message where you maybe select all the password reset tokens of users um so you have to go uh a different route of maybe doing a sub cury and then using an insert to put this data you want to exfiltrate another table and then use business logic of the application to access the data so it still works just a little less convenient but how does this look in the real world uh some of you might have already noticed that

if we control all of these A's that we have in a message why can't we just directly put any SQL statement in the first place why do we need to have the Overflow and you would be correct because this was just the simplification so let's look at how this actually looks um this is a two line code snippet uh the ID here is user controlled in this case it's a very small one and then we have our query you might recognize it from the teaser code snippet that uses the ID to select a user from

the database and the resulting packet or message that's sent over the wire looks like this we have our type bite then a small length and then we have the value with a trailing null bite uh all is fine here but now if the attacker makes the user controlled ID very large for a gigabyte then suddenly the length in the packet is smaller than before even then even though we send more data so we see that the truncation happened and now when the database tries to parse this packet um it will see okay it's a length

of 38 bytes so I'm going to parse until that and after that it will try to parse the next message so the attacker has to put their cleverly crafted payLo at that exact offset in this big buffer of a but how does the attacker know this offset um the offset depends on the query that they are injecting into and uh especially on the injection point in the query and the length of the whole query so if you know these things you can just calculate the offset and you're done but maybe you're lazy and don't want

to do the math or you don't know the query because it's kind of a blackbox test um so what can you do then and of course the naive solution is to Brute Force the offsets uh but unfortunately you for each try you have to send four gigabytes which is slow even on local setups and on like production setups it creates a lot of noise and risks Deni of service because it's a lot of data so can we make this more reliable and the first technique uh I tried out here was to borrow something from binary

exploitation from 20 plus years ago which is called a knob slat and mapped to this problem we just use a lot of very small message at the beginning of the buffer so that when we hit the start of one of these messages uh everything's fine because the message is parsed and then the next one and then the next one and so on until at the very end the big message with the actual Atta SQL payload is poed and executed but if we hit any other bite um the database will cause a fatal error because it

cannot par there and it will close the connection between the application and the database but usually that's not too bad because applications have connection pools and they just reopen connections if they get closed so let's visualize this this is the smallest possible message in the posterous wire protocol so we repeat this a lot of times and in the end we have our actual payload so if the offset happens to align with the first or the fifth and so on buy it then we're good but if it hits one of the other biets then there's this

fatal parsing error and the connection is closed so if that happens we can try again and now we just prepended one petting bite and then if it if the offset happened to align with the second bite and so on then now this attempt would be lucky otherwise we' just try again and again and again until we're lucky at the last try uh so this means that after a maximum of five attempts uh the attack is successful because each try has a 20% chance of success and we can repeat the attack but still in the worst

case we have to send 5 Time 4 GB which can still be a lot so maybe we can make it even better and for this I looked at what was holding us back with the last strategy and here it was the length bites because those were not valid type bites so not a valid start of a message so maybe we can make them valid type bites and I call this technique trampolines and what we do here is first we just use the Q which is a valid Tye bite for each bite uh because this is

also uh some big length it's hex 51 51 and so on and uh we can see down below the whole Atta uh payload and uh we'll see how things are parsed when the offset hits certain uh certain bites here so if it hits the very beginning uh the database parles it like this first Q is the type bite and then qqqq is the length of the first message so the database will parse the following whatever qqqq means as a number bytes and try to run this as a query and of course this will cause like

a semantic syntax error but this does not close the connection so the datab base will just complain with error response but continue paing the next message so this is like jumping over this many bytes so that's the trampoline right there and where we land after that we have a preconstructed message that has the exact size that then does another jump that ends up at right at the uh end of the attacker payload where the actual SQL uh statement message is so if we would have hit any of the other bites things look slightly different but

not too different here the length is pared as Q SSS the S is also a very valid type bite uh now it jumps over some more bytes and it lands at a different preconstructed message that again has the exact size that it does the right jump to land at the end but does this actually work like we plan to do and unfortunately uh after I tried to implement this I noticed well postgress has a logical maximum message size next to the actual size that you can possibly fit into four bytes and this is just under

1 Gigabyte so the first bite of the length can never be larger than hex 3F so that's bad because with a q that doesn't work so what if we just use 3F as the type and all of the length bites then also doesn't work because x3f is not a valid type bite and also none below that at least not for messages from the application to the database um so I had to use a compromise here which is an alternating pattern and with this uh now every second bite is a valid type bite so a valid

start of a message so so H we if we have hit any of those uh bites we're good because everything is pars okay and then we can do our trampolines like I just showed and if we hit any of the other bites like the null bite then we get the error and the connection is closed but still we can try again so after a maximum of two attempts we are successful um which is much better than the initial brute force all right let's see how many vulnerable libraries there were that we found and and this

is just a selection we looked at more but uh these are only the ones that were potentially vulnerable so for a go we tested four libraries and we found all of them to be vulnerable and also exploitable and I'm going to go into exploitability in a minute unfortunately only one of them uh fixed the issues we reported and the others did not so they are still vulnerable in the latest version for C there was one Library we looked at it was vulnerable and exploitable but unfortunately they fix and also backported it to older branches of

the library uh so that's great to see and then for Java and Java Script uh we tested a bunch of libraries and a lot of them were in theory vulnerable they had overflows without the right checks but they were not exploitable uh and this was most of the times due to language limitations where for example you cannot have large enough strings or buffers that would be more than 4 GB so their length would never overflow the 4 by length field and we're going to look at into these language differences later so short disclosure timeline for

transparency we sent out the advisories in February this year then PGX fixed in March n pgsql fixed in May and for a PG and PG driver the maintainer first responded to our advisory but then stopped and didn't Implement fixes until now and for a PQ we couldn't even reach the maintainers we tried to do it via their GitHub made an issue or made a poll request with the fix but they got no attention unfortunately so now let's go from vulnerable libraries to exploitable Applications so first we have a set of applications that use these libraries

or have used the libraries in a vulnerable version um and these for example include grafana or G or S thing but we weren't able to confirm if there's an exploitable path here so they might be but we haven't proven it yet and then there's a set of applications that we found to be vulnerable in a non-default configuration for example metam or G and then there was also The Sweet Spot which is applications that are vulnerable in the default config and our example here was Harbor and harbor is a container registry it's a cncf graduate project

which means it's quite mature and people are using it uh it's also apparently part of the mare tanzu cinas but we don't look into more of that so we don't know if that's exploitable in that scenario as well well um but in its default version uh it's vulnerable in the default configuration and also pre-authentication which is quite bad so they fixed this in version 2.11 which is the latest one uh by updating their dependency so if you have a Harware instance you definitely should update to the latest version and I'm going to try to demo

the Harware exploit now I'm going to try to do it live I also have a backup video um but let's see so first I will try to log in as the attacker to show that there should be no uh attacker user as you can see the login doesn't work so now if we start the exploit and I hope this works over the unstable internet um we have to wait a bit so there we see the query that the exploit is sending this is an insert query that will insert the attacka user into the database so

then we could log in in and as we can see it's sending a lot of bytes 4 gigabytes so it takes some time um and if it's Works successfully then we should be able to log in and have administrative privileges in the Harare application so that we then could send uh or mess with the containers in there so we saw an error which is expected here so if I now try to log in again we should log in successfully I hope okay it doesn't seem to work I have to switch to the backup video and

there on the left now we try to log in again and there we can see we can now log in and have admin privileges all right that was postgress now also let's look at mongodb to show that it's not a postgress and go only problem so to look back this is what a message in mongodb looks like we have a 4 by length field this time it's little Indian then we have two metadata fields that are not relevant here and then we have the op code which is like the message type and then the value

is a complex binary structure that's called bon which I think stands for binary Json and it's some nested data uh that's serialized to tlv sections so they also have lengths again and to directly look at some code it's a little bit more but I'm going to walk you through it we first have uh a line here that takes the content bytes so it takes the query and puts it into this B Bon representation then it computes the total length of the message by adding the header length and all and so on and this is saved

as a type usze variable again on most modern systems this is a 64-bit integer and then again this is truncated explicitly to an INT 32 because that what fits into the field so the developer had to cut it down here but of course this is the same issue as before if the length was larger what can than what can be expressed here um we have our truncation and things go wrong the exact same way as it did for post dress so for crafting a payload for mongodb it's a little bit more involved because we have

to avoid bad bites uh the payload uh must be valid utf8 because the strings are serialized as utf8 to the wire so we cannot create any byes that are invalid utf8 but the problem is the message type that we need to write in order to craft the right injected message is hex dd7 and that's always invalid utf8 so how can we do that and also size Fields inside the message could become some bite values that would also be inv valid utfa so we have to be careful there as well and the solution was to just

use other metadata of the message we're injecting into to create these bite values and I have an example for that so this is a normal mongodb query it just queries for a movie based on a few attributes and this is serialized to the following Bon document in the beginning we have in blue the length of the whole document then in red we have a type uh identifier that says the next key value pair has a string value then in yellow we have the uh key which in this case is title with a the trailing null

bite then in blue again we have the length of the following string and then in green we have the actual string with a terminating null bite then we have the next key value Pairs and so on and at the end we have another null bite that shows the end of the document so if we want to have the hex uh sequence dd7 anywhere in here we cannot use the yellow and green Parts because they have to be AV valid utf8 but what we can do is for example use the length of a string and just

make the string the length that will when serialized to bytes will be the exact bite sequence we need so here we need to subtract one because there's a appended null bite but then we have the dd7 there and with this trick now we can craft the entire injected message I cannot go into all of this here because we don't have time but this is the trick you need to be able to do it so we also looked at libraries here and uh we found the official rust mongodb client library to be vulnerable and exploitable uh

we sent out our advisory also in February and they fixed in March so if you use this especially if you use it in production which I doubt because it's rust but then you should update to 2.8.2 okay let's now take a step back uh we've seen all the cool bugs and the exploitation but is this actually applicable to the real world uh we've seen the demo but maybe this was just a lucky punch so let's talk about constraints and especially about the elephant in the room which here weighs 4 gbt because a lot of you

probably are thinking well can we actually send 4 gigabytes into all the applications aren't apps limiting input sizes and you would be right now there's a lot of common protections like default body size limits of your favorite web framework or maybe Json and Forum decode maximum sizes or you have your reverse proxy that's limiting request size and and so on and I also encountered this and thought about some potential bypasses here so the first idea is that sometimes end points are just unprotected maybe your framework doesn't have a default size limit and also sometimes the

developers explicitly disable the limits this was the case with Harbor for example they have an engine X reverse proxy which usually has a 1 Megabyte request limit but they explicitly disabled it I guess because they need to receive big docka images so they need more uh request size but this also meant that the whole API is now not limited anymore and the go framework that's used under the hood of the server did not limit this so we could send our big payload the next trick would be to use compression uh this on the one hand

allows you to send your attacker payload much quicker because you can compress it down and don't have to send 4 GB over the internet and also there can be logic bucks where maybe your reverse proxy or your framework first checks the request size but does the decompression after that so the check is essentially useless because your compressed data will inflate a lot and you can craft a big uh payload that after decompression still causes the Overflow so for engine X for example this works because engine X does not look at the body it doesn't try

to decompress it it just looks at the raw size of the request so if your back end does decompression then you could bypass this and also for example the JavaScript fastify framework does both checks the body size and also does the decompression but it does it in the wrong order so you can bypass the limit then you might also have websockets uh support on the server of your target uh this is cool because websockets also in theory uh supports compression it's not always enabled but it can help you but even without that a single web

ciruit request can be super large I I think it's a 64-bit integer uh the size field so you can definitely send 4 gab messages with that and maybe U the middle Wares and filters that are present in your framework don't apply to each individual web circuit message so maybe there's no limit there then you could also use alternate body types for example a multipod form instead of adjacent body and maybe the limits are different there or the developer forgot to configure a limit so this could be another bypass and then this one which I like

the most is a little bit more creative and open which I call server site creation which is you just find other means of bringing a big string to the server uh without sending it in your request and one example is ssrf where you point uh some functionality of the server to an attacka server which then returns a very large response and if this one gets saved into the database or used in a query it will also cause the overlow and this for example worked in Gog where you have a web hooks feature uh but beyond

the ssrf you might also have templating or a translation that you could use to inflate a string and this all depends on the business logic and the features of the server so that's where you can get creative to find bypasses all right now that we've seen how we might bypass limitations of Frameworks and servers we also need to look at languages we already seen some differences at the vulnerable Library so we need to see how well languages handle large payloads like how big can strings and buffers be and also what about integer overflows does every

language have silent integer overflows or maybe some languages are safe so for the large payloads we can see that go Python and rust out of the languages we we looked at uh have large enough string and buffer sizes that uh you can send for gigabytes and they can handle it for Java uh you're out of luck and we also saw See the s seen this with uh unexploitable libraries in Java and that's because uh strings and buffers here are backed by arrays which have integer indexes so you cannot index more than I think yeah two

to the power 31 here for C the string size is not big enough but the buffer size is so if you have uh multiple strings that you control then you can still end up with a buffer that becomes bigger than than 4 GB and for JavaScript it totally depends on the implementation for node strings are much smaller compared to the other languages but buffers can be large enough for integer overflows we can see that uh for addition overflows which just means you add two integers um we can see that in go Java and C CP

a silent integer overflow happens so it will just wrap around for JavaScript uh there will be no overflow things will just get more inaccurate because it's JavaScript and they use floats under the hood and in Python you have arbitrarily sized integers so you cannot overflow that and in Rust it depends on how you build your project if you build it in debug mode the compiler will emit a check that checks for overflows but in a release build it will not do that and for serialization overflows which uh basically is the case when you have a

standard function in your language that allows you to write a integer to a fixed size field um does this function check for overflows or not and in all of the type save languages this does not does not apply because the types that the function can receive uh cannot overflow uh the size field or the the field that it gets serialized to uh but this this just means that the developers now have to do the check before casting to this type and we've seen that this leads to bugs in uh our go and rust examples in

JavaScript it's again Implement implementation dependent uh I think in no JS it does the check but in bun or Dino it might not do it and in Python the struck. pack function actually does a check if you pass it too large integer it will throw an error if you try to serialize it to a twoo small field so to summarize the real world applicability we can say that in a lot of times we can send large payloads past Li uh the limits um for silent inter integer overflows or truncations uh we also seen that in

many languages it works or the developers do the mistake for the language and we also seen that you can definitely exploit some real world applications with this all right now let's look at what is there more to do in this kind of research for binary protocol smuggling but first I have to give a small disclaimer here please don't go out and send 4 GB to every API endpoint in your buck Bounty scope because you will crash systems you will get blocked and you will ruin a devops engineer day because they have to look at the

monitoring alert so don't do it please um but speaking of this it would be cool to actually have a non-invasive detection mechanism uh in whitebox test it's okay you can make your local testing environment and just hammer it with gigabytes of data but of course in Black box tests that doesn't work so if we could find a methodology to find for example a fingerprint of a database client lbrary being used and we would know that this library is vulnerable to these kind of texts then we could either directly report it or only do the large

payload then and if would this would be a tool it would be great uh to Aid you in in pentests and so on and of course uh it would be awesome to research a lot of more stuff uh more protocols like there's more databases than we stuff than the stuff we looked at today and there's all of the other systems caches message cues and so on that the application has to talk to we can also try to find more DN techniques for example look into protocols that use the limiters and not size fields and we

can also find more large payload message which is uh ways to bypass the filters of a framework for example to bypass the request size limits uh or maybe we can find generic ways of making a server craft a very large string um maybe there's some way to do this and of course all of the stuff we looked at today had four byte length Fields but what about 2 byte length Fields this would be much easier to exploit sending 65 kiloby instead of 4 gab uh is much easier you don't have the hustle with all the

uh limitations and I already already started to look into this a little bit so uh stay tuned for more to come in the future but feel free to also go out and check this yourself so let's wrap things up uh I have a few takeaways and the first one is that I think integer overflows are still relevant in memory safe languages of course in C if you have an integer overflow uh it's easily easy to get some memory corruption with it so uh that's done but in memory safe languages um the worst case that can

happen is a array out of bounds or whatever but with these data attacks uh in the uh binary protocols that we've seen today uh now we have a new way of exploiting these and I think developers kind of forgot about integer overflows and maybe researches as well at least in the memory safe languages then we also seen that uh it's entirely feasible to send large amounts of data even with uh protections because there's many ways how you can buy bypass these protections and finally to get back to the title SQL injection isn't dead of course

it was never dead at Le at least not in the real world we've seen a lot of uh real world applications uh with SQL injections but on paper you could say uh developers have the right tools to write secure SQL based applications they have parameterized queries they have query Builders they have orm so if they use all of this and the ation is safe so if you can't hack it you just have to go a level deeper thank you [Applause]