I Paid $500 for Devin AI (So You Don't Have To)

Look I've been known to have a not great opinion on Devon Ai and cognition I think this is very misleading that they have some amazing thing but they're not letting anybody use it they probably don't have something as amazing as they're saying they do oh if you're so proud that you raised $21 million and you have an AI software engineer couldn't you either use some of those $21 million or your AI software engineer To build a website that's not just black text on a white background with almost nothing there but to my surprise they actually

did release a product so let's give it a try give it a fair shot see if it's any good so this is the website for Devon so already it looks a lot better than it used to so good step forward there and it says Devon is a collaborative AI teammate wait hold on does this follow my mouse these little little animations I don't know how well That shows up on camera for y'all or in the screen recording but that's actually that's pretty cool okay so anyways let's scroll down wait hold on this little the scroll

bar with this sort of moving as I scroll that's actually pretty good I don't like that I like this website so much but this is this is well made good job Devon or I don't know if Deon made this or just some front and engineer cognition made this but clean website all right so anyways I guess this is What the workspace looks like we'll see that in a minute little scroll bar over here wait okay I might have to take back my golf clap this doesn't work I can't click on this and scroll so I

just have to like slowly scroll through this to keep going to keep going down but okay oo that's a nice animation why is there a white box but ooh okay that's pretty clean all right now we're now we're down here there's some use cases so we can do code migration and refactor already I Like this better because last time it felt like they were pushing so hard that they've built an AI software engineer that can do a little bit of everything starting out with hey we can do code migration and refactor much better because frankly

that's something that they probably can do because most AI tools are getting pretty good at that those are the types of things that developers don't want to do and that AI can reliably do pretty well but anyways We have bug and backlock work data engineering and analysis app development personal assistant tasks and many others okay so they are still claiming it can do pretty much everything but I do like that at least the first things that's showcasing are things that AI tends to actually be better at okay let's see what else they say learn and

work together okay cool able to work with hundreds of tools all right fair enough and then build with Devon her industry Leaders choose to build with Devon I don't really like the way they did this because my mind wants to read the big text first and this text second even though it's above it and it makes like a weird mental thing when you read it but anyways they're claiming mongodb Microsoft ramp a bunch of compan are using Devon I don't love this because I feel like this is making implications that while what they're saying is

probably objectively true the Implications feel less true like is Microsoft using Devon maybe like somebody right like maybe somebody at Microsoft is testing it and whatnot but if you went to your average Microsoft engineer and say have you ever used Devon the answer is going to be no they've probably used GitHub co-pilot a lot and other tools that are approved within Microsoft but I can't imagine that Devon is approved to widely use across Microsoft and sort of this the Same for a lot of these companies but maybe I'm wrong maybe they're all using Devon a

ton I don't work at those companies but just sort of my hunch that this feels a tad misleading it's probably more so that industry leaders are choosing to try out Devon and maybe a few of them are actually using it but okay let's see actually customers what do they say we have this guy case study what did they say about Microsoft Devon is an absolutely amazing tool from Kevin Scott the CEO or CTO of Microsoft all right I'm curious when did Kevin Scott say this Kevin Scott all right it looks like he said it at

the speech Devon all right their product Devon is an absolutely amazing tool okay guess he did actually say it if you can imagine for yourself some of the most tedious things that you do as an engineer or a software developer Devon is a tool designed to help with those Tasks all right Azure it looks like they're building Devon on Azure so a little bit of a conflict of interest Devon is a client of Microsoft is the way I'm interpreting this but okay maybe Microsoft likes Devon lumos we try to make low cost reversible decisions really

quickly and Devon makes more decisions that type of decision and Devon makes more decisions that type of what does that mean I don't know if this is a typo on the website or what he said And I'm misunderstanding it but anyways I can work on stuff and kick off Devon tasks in parallel it's cool to see PR after PR come in okay open C Deon has really redefined what we thought was possible especially with recent slack integration whether it's debugging prototyping or front end updates Deon is constantly stepping up and consistently improving is it open

see the like the nftd marketplace but okay this guy's the CEO opy so it does seem like some people Are using Devon so let's give it a shot how much does Devon cost pricing choose a plan that's right for you all right so we have Enterprise custom pricing and team $500 a month that's a lot this video has no sponsor for what it's worth so I appreciate you all watching but I guess we're going to pay $500 a month to be fair if Devon does the things that they claim Devon can do $500 a month

is not that bad of a price tag right if it can actually do what they say I'm not Confident that that's going to be the case and most of their competitors are free or very cheap so 500 a month feels steep but at the same time if it's meant for big corporations it's not really meant for individual use I guess it makes a little bit of sense so okay let's get started so I'm going to create an account and all that and I'll get back to youall when I've wasted $500 I mean spent $500 that's

I'm here on the checkout page you can see it's $500 like They say but interesting note here it does say Devon additional usage price varies $2 per ACU I'm assuming that's some kind of credit unit so just note that you can go over whatever amount of credits they've allocated to you and you can actually end up spending a lot more so be careful if you do actually decide to pay for this but anyways subscribe $500 all right so actually here it says in addition to your monthly subscription we ask that you set a monthly additional

Usage budget I like that they do this immediately after having you pay they have you set a budget of what is the max that you can charge I'm going to put $0 because I don't want to spend extra money so I've spent enough on this video so let's leave it at that okay so we have zero extra acus and it did say I think you get 250 by default yeah so you have 2 50 I don't know exactly what an Acu actually is but hopefully we can stay under 250 if I do need more I

guess I'll pay for it all right we'd love to share tips for working together book an optional onboarding session with a cognition team member we'll give you 50 free acus after the call I don't think I need that finally here are some important things to know about working with me I work best when you give me detailed junior level tasks tell me how to test my work break down the tasks longer than 3 hours does this mean like things that would take me 3 hours or Things that would take you 3 hours cuz I don't

know what takes Devon 3 hours but okay provide clear requirements help improve my knowledge base okay good tasks small frontend bugs or edge cases creating first draft of back luck work try handing me tasks from your to-do list at the start of the day and targeted code refractors tries using Devon ID extension for vs code and forks okay start using Devon hey there Connor like any new engineer I'll need help With my onboarding let's get things set up so I can start contributing connect GitHub all right connect to GitHub all then you can connect slack

I don't actually really want to do this so I'm just going to skip slack invite teammates it's just me skip set up your first Repository so let's use the tetris react repository which is one I made in another video where we used react to build a clone of Tetris and it's a good like decently sized repository that I Think also has a lot of room for improvements so we can see if Devon's able to improve it for us so let's set up this repo all right so I don't know what exactly is happening here so

we have like vs code here is this this is just the repository I guess yeah it is and then we have just like a browser all right so clone is it it's already cloned I'm kind of confused does it want me to do that for it I guess it wants me to just do this npm install I'm kind of Confused what exactly it it wants me to do here but I guess that's what it wants me to do just install the dependencies like that I I don't know I don't understand why it doesn't figure out

how to do this on its own K 10 vulnerabilities but maybe we can have Devon fix these vulnerabilities for four moderate five high one critical okay let's say I've installed dependencies I don't think there are any lent or test on here so this is fine we don't we Don't need to worry about these things cuz I'm not going to be pushing to the repository I don't think delivery teach Devon about working with this repository tell Devon how to verify and submit change to this repository Deon will always see these instructions in working in Tetris react

advise on how to run test formatting lenting commands PR format expectations window wait for CI okay let me let me take a minute to read this and get back to it all right so I Gave it some pretty basic instructions I said to testler code just run npm run Dev and manually try the code in the browser in a local Dev server and then I said there are currently no tests in the repo because there's not maybe I'll have Deon actually try to make some tests for us that could be interesting then the repo uses

eslint so I said do run es lint before making a pull request and create all updates in a new branch and make PRS let's say save okay let's do Finish and finish setup don't mind me I'll just wait all right so that took like a minute or two but that's fine and it gives me this example prompt to just ask Devon to verify that it can access the repo run lint and open a dummy draft PR so let's actually just try that so I'm going to run this okay so now we're here okay congrats on

starting your first session I'm excited to work with you here are some things to know about working with Me try tagging at Deon and slack I'm not going to do that I can make draft PRS of backlog work you can hand off Tech debt to me from vs codex okay we're not going to use the vs code extension either just going to do it in here all right so it's verifying his access to the repo running lint and creating a draft PR all right so I guess we we just watch Devon okay what are we

running here we're doing lots of things we have package Json we're reading that that's good all right We ran es lint it keeps saying Dev is working or awaiting your response but it's not actually asking for me to do anything so it's just working I guess and we are updating the readme okay I like this PR we're making we're just adding to the readme some instructions it's good and we're creating a branch what do we call it update read me I guess we added a PR description okay oh here we go created PR oh we're

still going I guess we're checking that it's Not failing any tests or something created a draft poll request writing local development Devon is working or waiting a response R oh okay here we go I've have successfully completed all of the requested verifications repository access lint check draft PR creation all right so let's review the PR okay here we go docs add local development instructions add local development instructions to read me changes added development section to read me added a Step-by-step instructions for local setup all right files changed fair enough seems good to me this PR

draft poll request can't be merged so why is this a draft PLL request I've never actually used a poll request on this repository but okay let's approve it and is it going to allow me does Devon have to make this not a draft I'm not sure so let's go here all right so let's say I approved the pr can you merge it please Devon is thinking I'll merge the Approved PR right away okay all right so it's actually going to run into an issue here so I guess I don't it doesn't have access to merge

the pr I'm not allowed to merge directly into the main branch as per my security restrictions this is a safety measure put in place by my creators could you merge the pr yourself the changes have been approved and are ready to be merged all right so it's still a draft I believe so I'm going to tell Devon the pr is a draft PR how do I Merge that let's see if it can tell me how to do it I'm actually not 100% sure I don't really use the GitHub UI I'm not sure if I can

make somebody else's draft PR like not a draft or I don't even fully know what the draft PR thing is to be honest I've marked the pr is ready for review so let's go back here okay so now we can actually merge it so let's say merge pull request confirm changes and successfully merged cool and we can delete this branch that Devon created Let's go back to the code so now we should have this and we have this pull request from Devon AI integration bot all right so I'm going to start a new session what

I wanted to do now is to try to fix the security vulnerabilities that npm told me I had so the repo has secur issues when you run npm install please fix all of these and create a PR that's ready for review and let's run this and let's see what Devon can do all right so we ran npm audit fix and Revealed 10 vulnerabilities including one critical and severity issues all which can be fixed with npm audit fix I'll now check the package Jon to understand the current dependency structure inversion constraints before testing can be done

with npm runev okay all right so came up with a plan run npm audit fix review package Json and then verify changes by running the app locally run es lint and create a PR for review and it's now executing the plan It's kind of cool that you can just like watch it do this like is this quicker than me doing it myself no but it's kind of fun to watch so I'm kind of I'm kind of enjoying this I don't like when it does stuff it doesn't always like auto scroll like I'm having to go

to my mouse and scroll down so that we can see what it's doing I don't I don't know why it's doing that but that's okay I'm curious to what degree it's able to test this and it's kind of my fault that there's Just no tests in this repository but I'm curious to what degree it's able to test this just by like going to the browser and making sure it works because how does it know what exactly it working means but it's kind of my fault for having a repository with no tests in it and not

really having good instructions for what it working means all right so we did es lint which passed and then we did npm randev is it actually doing anything to test that it's actually Working or is it just like making sure that it's able to run Dev okay it is on the tetris page so this is the game I'm curious is it going to play the game are we going to play Tetris or we just going to look at it what we going to do Devon that is kind of cool that it was able to actually

navigate to the the website okay so we viewed the console to make sure there's no issues there are we going to click new game and actually play the game we are not it does note a 404 error for the favicon valid I don't think we have one I'm curious if it's going to try to fix that or if it's just going to leave it alone or if it's going to ask me about it curious what it's going to do all right so we are pushing changes all right so Devon made a PR for what it's

worth it says that this is at all right so the pr is ready for review for what it's worth this is at 12:59 and initially I started this at 12:54 so it took 5 minutes to do this which I could Have been doing other things while this was doing it but this also would have taking me all of 45 seconds to do myself but anyways let's review this poll request all right so we fixed known vulnerabilities the following updates were made critical so we updated Babble Traverse High we updated a bunch of other stuff and

moderate more updates okay so we basically just updated stuff files changed is going to be just package lock why was package.json not Updated I'm curious if we load the diff maybe it doesn't need to be but I would have thought we would have updated package.json as well maybe I'm wrong though I actually don't know the answer to that so what I'm actually going to do is can you confirm that npm install no longer has the issues so before I merge the pr I'm going to ask it to confirm in this way okay interesting so there

were some deprecation warnings uh but it says they don't affect security or Functionality of the application so I guess it ignored them and the other vulnerabilities are gone okay so let's go ahead and merge this PR for Devon so merge poll request and I suppose we don't need to update the package Json so makes sense all right so so far so good with Devon now this repository has no tests in it like I said so let's ask Devin to write some tests so let's say can you add some tests to the repo to to ensure

that the Tetris app loads Properly and the game is playable and working as intended I'm curious how it goes about doing this it's going to have to decide what it's going to use for testing so if it's going to add some framework what it's going to use it's going to have to figure out what working as intended even means and what making sure the app loads properly means so I'm curious to see how deep it goes into testing and what specifically it decides to test but let's see how it does guys So it's still working

but I like this it says the read me provided minimum information so I had to check the package Chase on so Devon's kind of taking shots at my read me but that's okay Devon as long as you can make some tests for us we'll be happy all right so I've identified the main game components and hooks in the project structure which include core game logic in the hooks directory and UI components in the components directory I will examine the Core game logic files to determine the necessary function or functionality for testing okay this seems like

it's actually making good progress but I'm curious to see what this tests are actually going to look like all right so it's got a plan here so it says set up v test and react testing library and add basic app loading test app. test.ts and core game logic tests use tet. test and use Tetris board test and focus on critical game mechanics peace movement Collision detection and scoring test changes by running npm run test locally and create a PR with new testing infrastructure okay this seems reasonable to me so let's see this actually ends up

working so this is interesting so it's creating some tests and it looks like the testing Suite that it created are actually failing so it'll be interesting to see is it able to figure out what it's done wrong and verify how to fix the tests or is it Going to get sort of stuck in some kind of infinite Loop which I've seen like chat GPT and some other tools do before so let's see it's able to fix this interesting so I've added the test files and made some initial test runs I noticed a development environment issue

with the V react plugin that's unrelated to our test changes this doesn't affect the functionality of our test but you may want to look into the development environment set up later so I'm curious It doesn't really tell me what this is I guess I could ask but I don't think this is the case it it does work so I don't know what it's saying is the the issues here but we'll let it finish doing its thing all right it looks like it's still failing some tests and we've been running this for a while so let's

see if it again is able to fix its test to account for whatever isn't working and here's where it seems like we get into a bit of almost an infinite Loop it looks Like it's still failing the same tests as far as I can tell but we'll let Devon keep trying to plug away this and maybe Devon will figure out how to fix these tests but it seems like it keeps updating this one part of the code to try to get whatever this test is doing to actually pass but it's not really being able to

do it so we'll see if it asks me some questions or if it's ever able to fix it or what it decides to do all right so we are still in infinite Loop territory here where Devon keeps running into essentially the same issues and trying to fix that same issue and it keeps making this code more and more and more complicated and it looks like it finally decided to change not just the testing file but rather the actual file so it looks like just adding a bunch of console logs to try to debug so it's

almost debugging humanlike but I am like 98% sure that this code actually works it would be funny if Devon found an Actual error in the code itself and its tests were actually working but the code itself is wrong that would be interesting but I don't think that's the case I think it's just writing a bad test but we'll see if it's able to figure this out so I do like that it's doing some like debugging here rather than just infinitely trying random things until it works because that clearly wasn't working we'll see if it's able

to figure it out all right so it's Finished one thing that kind of annoys me is it says it created a R I'll let you know when it's ready for review and then the UI says review and comment which implies to me oh it's ready for review but that's not the case it's not ready for review until it does like this next step which is just a little bit odd but okay I've completed adding tests and this includes app loading tests verifies the app renders without crashing checks for presence of kyi elements ensures the Game

board is present in initial State then we have game logic tests as well the tests are implemented using Vest I think that's how you pronounce I've actually never used the uh library and then react testing Library providing a solid foundation for ensuring the app's functionality all tests are passing locally all right so let's review this PR let's see what it actually did files changed n okay package Json so we added testing commands and we added Dependencies fair enough Let's uh Zoom this in a bunch so you all can see it so we have our main

testing file so render Tetris app without crashing render the app and okay we're just checking that these things are in the document fair enough then we have the used Tetris test so we start with a score of zero that's a fair test start with a non-playing state initializes the game State when starting a new game result okay current okay this seems reasonable and then the Game board has correct Dimensions all right so these are reasonable tests I would not call these comprehensive in any regard but a reasonable way to test use Tetris then we have

use Tetris board so we get an empty board has Collision so this is where it was having a lot a lot of issues so let's see it creates the board of correct Dimensions has collisions detects bottom wall Collision so we get an empty board we have the eye shape I don't know why we're only Testing the eye shape here so it seems like we should be testing other shapes but okay I'll leave it and it's got some comments and then Collision has collisions expect Collision to be true okay so this comment is kind of convoluted

and this code is a little bit too and we're only testing the ey shaped Tetris block but I actually do think this is functionally correct as far as I can tell and then detect a left wall Collision again we're just using the eye Block right wall Collision same thing and then returns false for valid position so what returns false oh has collisions okay fair enough this isn't super useful because it's only checking one valid position and it should probably check a bunch of positions but fair enough and we're also not checking that the game is

actually playing like that time is working correctly we're sort of just checking when we have a board in a certain state that it is Returning the correct information but these are better than having no tests so I guess it's definitely a good start and we could probably ask it to make more of those tests all right so then we have a roll of grid on our board which I suppose is fine it's not what I asked it to be doing and I don't really like the idea of putting little things into a PR that are

not related to what the pr is itself some people will do that like some real software Engineers do that Personally to me this deserves a separate PR even if it's just a oneliner because here as a code reviewer I come here to review the fact that we are adding tests I'm not here to review the fact that we added a rooll attribute to our div here I'll leave it it's fine and then we updated use Tetris board and I don't understand why so this this was working before so let's see what exactly it did so

before we had has Collision is false and I wrote this code years ago so I don't really remember it but okay fine what did it change all right so it made a ton of updates to use Tetris board which is code I already had and it didn't really explain to me why it made these updates I don't really know what these updates are and if I was a code reviewer in a real team I would reject this PR because this PR is AD test for t react app it is not fix logic of Tetris react

app it is not change logic like I don't know is this telling me that this Logic was incorrect and we fixed it is it telling me that this is more efficient maybe it's because it's trying to return early is it telling me that it fixed some Edge case I don't know because I I don't know what exactly the purpose of this is and as a code reviewer I should just be looking at it to review that the change you made is the change you said you made not I'm going to come in here and try

to see if this logic is correct and this would Take me a lot of time to look at this and try to figure out is this new logic correct or not and then we have setup test that's fine and the config for the testing which also seems fine but this is where I'm confused so let's come back to Devin and let's ask why did you change the existing hook file all right so it's gone back to the actual website and it looks like it's going to try to test the game behavior in the browser to

see if the original logic correctly Handled bottom wall detection which it did okay so it looks like we're actually playing the game here which is interesting although this isn't like live so I can't actually see it updating and I'm curious does Deon see it actually updating well okay we're we're done with that so I don't know what Devon did or didn't see it wasn't able to like show me like the animation of the gam playing we'll see if it figures out whether or not the original Implementation was correct or not which I'm pretty certain it

was but maybe it wasn't so maybe Devon found a bug in the code okay so now we're at upon reviewing the I block shape definition it was determined that the test case for the bottom wall Collision was incorrect as it did not account for the shapes structure the original Collision detection logic is functioning correctly so essentially I asked Devon to make a test and what Devon did instead was it Made a test the test was failing and it decided the test must be right so I'm going to go change the code to make the test

pass but the code was already correct it was the test that it wrote that was incorrect so it created its own issue and in attempt to fix the issue it created it created new issues it broke the existing code and made a poll request and said look all the tests are passing when it's the test that it just wrote and it wrote them without really Properly verifying that these tests were actually working all right so it reverted its changes and now it's back to writing tests again and we're back to these tests not passing but

we'll see if it's able to this time correctly write the tests that are based on the logic we already have that is sound working logic all right so we're back into this thing again where we do this basically infinite Loop of changing these tests until it works and it feels like it's Kind of just doing a guess and check so it's just like changing things seeing if the tests pass when they don't it changes things again and it's just going to keep doing this until either the test pass or it gets back to where it

was before where it decided that the original logic was incorrect when the original logic as far as I can tell is not incorrect I wonder does Devon have like a point where it just gives up or is it just is it just going to keep Doing this forever cuz it's been at this for a while where it just seems to just be kind of changing these numbers up over and over again until it gets one that that actually passes the tests and I'm not even sure what exactly it's doing incorrectly to be honest but clearly

this this ain't it this ain't working like this is literal just guess and check coding where it's just trying over and over and over again with the different numbers until it works and What's scary about that is even if it finds some version of this that passes the tests it clearly doesn't understand the code it's written to a deep enough level to where it's able to logically figure out what these numbers should be and because of that how am I supposed to trust that these tests are aren't actually doing what they're supposed to be doing

right so I think it's it's once again come to the conclusion that the original logic was incorrect which would Be interesting maybe it is correct that the game has been broken this whole time although I wrote this code years ago lots of people have used the code lots of people have watched the video and followed along with it and nobody has ever told me it didn't work so I'm not very confident in the fact that it doesn't work but if Devon actually found a bug that'd be pretty impressive so let's see one eternity later all

right so we updated the pr so let's take a Look at this PR now to see what exactly we did all right so we have package Json the test the used Tetris test again used Tetris Board Test looks like we have a more detailed comment here I like this comment a little bit better this is easier to sort of understand there's a weird extra space here which just like nitpick I don't know why there's a space right there but anyways comment looks a little better I don't like that we have console logs left in but

it's a test so I suppose that's fine and then we still have this thing that we changed we have console logs now that we added to use Tetris board so I would have already rejected this PR now because we shouldn't have console logs being added and it looks like we're once again changing the actual logic all right so let's come back to Devon one more time and let's say did we change the logic in the hook if so why and can we remove all console logs from the code so at this Point it feels like

I'm having to baby this a little bit and it feels like I'm having to do more work than I would have had to do to just write the test myself because I don't know if I can trust that these tests work or that they don't work because it's changing Ching actual logic of the code not just writing tests and it doesn't seem very confident in the test itself and the fact that it did it like 100 times I I don't know where that went but it did it like a 100 times Going back and forth

of writing the test test don't work write new test test don't work so I don't know if I can trust this or not I don't know if the changes it made work or not and then it's also doing things like leaving console logs in the code which if you've worked in a real code base is pretty much always a thing you don't want to be do it wait so here it says yes we modified the Collision detection to check all shape rows instead of Filtering them which fixed the bottom wall Collision test but why are

we trying to fix the Collision test we should be if we need to fixing the logic and then writing a test or we should be writing a test that is actually testing the logic we already have and it says I'll remove the console logs uh but then it said here we found that the changes to has Collision logic were unnecessary so it's reverting them again so I'm kind of confused do we think that the changes Were necessary were they not necessary are we reverting them are we not reverting them and I don't fully understand what

exactly we're doing here so it says we're going to revert Collision detection logic back to the original file remove the console logs and run the test locally but I have a feeling when it reverts these changes the tests are once again not going to pass and we're going to end up doing the same thing over and over again but let's See what happens so once again the tests are failing in the same way they've been failing because either the test don't work or it's correct in updating the logic but I don't think it is but

we'll let it try this one more time see what it comes up with and if it still thinks it needs to change the logic I'm going to try the change to logic to see if the game still works or not but what I'm I'm feeling is happening here is that these tests are just not correct and not tests That I would ever be able to actually merge and realistically by now I could have just written in the code for the test instead of sitting here and doing this with Devon but it's kind of fun so

it's got that going for it and I guess I could be doing other things in the background while it does this uh so not all bad all right we once again have an updated PR let's open it up files changed eight see what we did so we updated package Json that is fine why Did we change oh we just added a comma there okay and then we have app test f fine use Tetris test seems fine use Tetris board test again seems fine there's a lot going on here but I'm going to assume if the

test are passing it's probably correct I don't know why it feels the need to add this accessibility attribute it's a good thing to have it probably should be here maybe uh but it's not what I asked it to do and I don't really like like I said Putting things in the pr that don't belong but that's fine okay so it seems like now it has successfully gotten the test to pass without the need of these extra things which makes me wonder why was it trying to sort of Gaslight me into thinking that my logic was

incorrect initially but it seems like finally find a way to figure it out so that's good all right so I asked it one more time to just verify that the tests are passing it does say they are but Then there's this like thing that's not passing it says it's due to a v plug-in configuration issue so I'm going to ask it to fix that first because I don't know why it would push a new test that isn't going to pass so we're still not completely there yet so let's say fix the V configuration issue so

all tests pass okay so this is interesting it says it's a version issue which is probably correct it says it's searching for a solution online to address the issue so It's actually like just looking up the answer I guess on like stack Overflow or something which I mean reasonable if it's able to successfully do that that's cool okay so according to best practices I need your permission before updating any package versions would you like me to update to the latest version try a different solution or keep the test as is and handle the environment let's

do option one interesting so it actually asked me first do you want me to update This so I do all right so it fixed the issue we had and now it's finding new issues with the tests that it told me were correct so it looks like screen. get by class name is not an actual function so now it's doing get by roll instead and finding the grid roll that way which I guess is fine but why is it just like hallucinating and making up functions and when it finds out it doesn't exist rather than finding

the name of the function it was looking for It just finds a different function to use instead but that's fine it is a way to get that element so it seems to be working and we have three tests passing now all right so let's see I would assume it's going to update the pr and we should be good to go finally on these tests okay so we once again have the pr to look at files changed so looks like should be all good here nothing crazy it's just the same tests and looks like we just

updated those versions so let's Finally merge this whole request confirm merge and now we should have tests so that's pretty cool and if you're curious the total time so it started at 104 and we finished at 152 so it took almost 50 minutes about 48 minutes to go from asking it to add tests to having the tests seemingly work although they're not very comprehensive so is that good I don't know it's decent it's not like it was super efficient it probably wasn't more efficient than me just doing it Myself and I did have to sort

of hold its hand a lot like I couldn't have just for the last 50 minutes been doing something else because I would have had to come back to this a lot so I couldn't really just have this running in the background but it was ultimately able to add some tests and eventually figure out the mistakes it made and fix those mistakes so that's good so next I want to see how well it can just make some UI changes so let's say improve the look of The UI by making it more modern feeling let's just see

if it's able to figure out how to write some CSS in this repository and how to actually make things look a little bit nicer all right so this is the UI we currently have so there's tons of places here where this can be made to look prettier so let's see what it comes up with I'm kind of curious what exactly it does but I think it's pretty low hanging fruit to make this look nicer so hopefully it's able To figure something out hold on hold on Devon's installing Tailwind okay send a message don't use Tailwind

or any new libraries slf Frameworks follow the style of the existing repo I I don't know why it decided when I maybe it's like modern CSS that means Tailwind but like that's not what I asked it to do I want it to work in the style of the repository so I'll interrupt it here okay so it's going back to using vanilla CSS without adding any new libraries and Maybe the the use of the word modern did imply right like modern code which often times people associate with using Frameworks and libraries but this repo was written

with vanilla CSS mostly just because it was made for the purpose of a video demonstration but I would think it should just follow the practices of the existing library or of the existing repo and if it wants to make a major change like introducing Tailwind it should have asked me first rather than just saying Okay time to add Tailwind because I don't know that seems like a odd choice for it to make but anyways now it is back to updating the CSS again so hopefully it's able to figure something out okay so this is where

we're at so far the button and the text I think looks better I don't know how I feel about this board that it's made interesting looking to say the least it's not awful but it looks a little bit strange I I'll say that but I'm curious Is it going to look at this itself and say okay that part looks good that part looks strange let me update these things or is it going to look at this and say all right it's fine and just move on like does it have any way to actually visually look

at this and understand what looks good and what doesn't okay we done more things now the tetris board is like black with this black border I I don't like this direction I like the previous one also the new game button is Now like super wide for some reason which feels a little bit awkward but maybe it's nice I don't know I don't mind that I think this font it's using is kind of nice it hasn't made huge changes by any means though I don't know though the more I look at this this dark theme I

don't think I like this black border maybe I do I the dark theme's actually kind of I'm curious what this looks like with the game actually started because I don't know the more I Look at this the dark theme's kind of it's kind of nice it's got a vibe to it for sure I don't know let me know do you all like the dark theme for Tetris right interesting so it added a bunch of CSS animations which I'm kind of worried that these animations might actually break the game but we'll see it might not I'm

kind of curious so we'll see what it's able to do here exactly this does look kind of cool though and I kind of like this uh so we already had this Where it shows like the next blocks coming up but it put like this little border thing around it and it looks kind of nice all right so we can review the changes that made files changed all it changed was our CSS file which is good because that means it didn't make changes we didn't ask for it's also using some CSS variables which is cool so

what I do want to do though is actually test that this actually works so I'm going to load this manually and Test it in my own browser and make sure that we haven't broken the game all right so this is the version of the game that Devon gave us where we now have this black dark mode version I don't know how I feel about that it's kind of cool kind of not okay we got nice little hover animation on the button as well so let's start the game and okay so this does seem seems to

be working still I don't think it's broken the game at all so let's try putting this down here it Is doing this thing now where the size of this is not constant so it's causing the board of the game itself to to sort of move although the fact that it didn't break the game I would say it's impressive although score hasn't gone up cuz I haven't scored any points that'll do it all right let's let's make sure the score is still working see what happens with that and then I'll also see what happens when the

game ends okay so the score went up now let's see if we Can lose very quickly and let's see what happens when we do all right yeah it is super annoying that it's causing the board to move all right game over score 100 new game cool so that seems to be working okay so I'm going to say there's an issue where the board moves left and right because the size of the upcoming blocks section is not constant keep the board in the middle of the screen without it moving and keep the upcoming blocks a constant

size without Overflowing it all right so Devon was able to supposedly fix what it did so let's try it again and see if it's working now all right so we're back here again new game okay so it did not move the board when I did that and let's see as the blocks come yeah it seems like our board is staying in the same spot now I wonder it hasn't broken the sliding nope that works too cool it seems like everything's working I don't know what the point of the animations Necessarily was like it doesn't seem

like those do a whole lot if anything so I don't I don't know that we actually need whatever animations it wrote but other than that it seems like this is pretty good and maybe the animations are doing something it's hard to tell exactly so I'm going to look at the animations and I'll see if they seem like they're doing anything or if it's just pointless animations that Deon wrote All right so I looked to morth These they're supposed to do something when you clear a row when a block appears and like a flash when the

game ends but none of them seem to be doing much I can't tell if they're working or not but they're definitely not doing a whole lot so I'm going to ask Devon to remove those and then we'll go ahead and merge the pr so let's say let's remove the animations leave everything else as is all right so we updated the pr again it says it removed the animations so we Can come here click the code cool so it looks like the animations are gone I'm not seeing anything animation related here that it added cool so

I think we are good to go on this one so we can go ahead and merge this poll request as well for the modern UI overhaul as it calls it all right so now one final test I want to see if it can add its own feature to the game so what I'm going to ask is add a feature to the game to keep track of high scores for the users Computer so we're not going to be adding these to like some database in the cloud or something we just want to add them to the

user's computer preferably it's going to use something like Local Host but we'll see of what decisions it makes so let's run this and see if it's able to figure out a way to do that okay so it says I'll add high score tracking functionality to the Tetris game it says it's using local storage so I think that is the best way to do this so that's Good and it is now working on a plan to do it I do find it kind of funny that every time every time it tries to run the application it

runs npm run lint and npm runev and it always gets told that there's not a script called lint and yet it just seems just seems to keep going instead of trying to uh actually run the linter which maybe we should have npn npm run lint or maybe it should run on dev or I don't know maybe there should be a way to do this but it just Hallucinates that this npm script exists and when it doesn't it just keeps going instead of finding a way to run the lunter which kind of entertaining to me but

either way not a big deal okay so now it looks like we are getting a a page not found kind of curious how it managed to do this but we'll see if it Finds Its its way around and figures out how to fix this okay so I don't even know what this page notot found is I don't know what FRP is maybe it's Something that Devon uses but anyways says it's made the changes I'm experiencing some environment setup issues that prevent me from testing the changes in the browser would you like me to proceed with

creating the pr or would you prefer to help set up the testing environment first so it it asked me a question but it doesn't seem to be waiting for my answer it just like is creating a PR and it's Devon did you want me to answer a question or do you Want me to look at your your PR or I don't know what you want from me Devon all right well it says it created a PR I don't know if it works Deon doesn't know if it works because Devon couldn't see it but Devon decided

I wasn't able to test the changes in the browser you may want to test locally so at least it's telling me that but okay so let's say try testing again and write some unit tests to verify it works as well interesting point here it says it's Using this mt4 class for spacing which I think might be a Tailwind thing but we have Tailwind in this application so at minimum this UI it thinks it made is not going to look like it thinks it does so we'll see if it's able to figure that out when it

retries it cuz clearly they're having some issue and this looks like it might be a Devon issue not a this repo issue because I have no idea what FRP even is I'm curious also it says it suggested update to knowledge Let's see what this says okay so it's saying to update the fact that there's no test unit test should be written to verify functionality that's valid initially there were no tests and Devon wrote the tests so sure let's accept this knowledge update oh we got like confetti or something okay fair enough close that okay so

it looks like it's still having this issue the server is powered by FR FRP I'm gonna see what exactly this is and see if I can help it Figure out how to fix this but I have the impression that this is a Devon issue not a this repo issue because the repo does work okay so it looks I I don't like this thing where it asks me if it wants me to do something and then it just like keeps doing other stuff but it looks like it's having it it hasn't run npm install so maybe

that's the issue so uh yes you can run npm install I don't know why it's telling me I should run npm install to set up the Testing environment because Devon like this is this is your testing environment you are the software engineer you should know to run in PM install but maybe that should be in like the knowledge thing I told it I guess I maybe didn't tell it to run in PM install but I don't know it seems like that should be implied and the other things we did with it it was able to

just know it's supposed to do that so I guess that's the issue hopefully at least okay so it ran in Pam Install and ran the tests and it learned that its tests are not working so there's that okay so we still have one test that's failing so let's see is it able to fix this one so it's saying it's going to force game over with a game take but that's not going to do okay but it's changing it to result that current is playing equals false so this would force game over state but I don't

know that doing this I don't remember the implementation I don't know that doing This is going to force all of the game over code to run by just manually changing this but we'll see cuz I don't think we're like listening for this to change I think it's on game tick if it's over then all the game over stuff is going to happen so I don't think this is going to work yet it did not work we'll see if it finds a uh another way to force the game overate but I actually think that's a somewhat

difficult problem to figure out in a test so we'll See if it is able to do it or not so it looks like it made another change clear exist okay I guess this is sort of uh forcing game over and doing some like mocking as well interesting okay well we'll take a deeper look at that in a minute but it seems like it's finding some solution to have some kind of testing hopefully all right so this is an interesting little hack it seems to have found it says starting a new game while one is in

progress triggers game Over so this is I guess how it's forcing a game overstate interesting I think that that maybe works I don't know I haven't I haven't looked at this code in a long time but maybe that'll work I don't love that though because it's very hacky and I could see this hack no longer changing or no longer working because of some change down the line it's going to break this test that shouldn't break because of that change but it's a I guess potentially decent Solution if it does work but I don't even know

if that's going to work and no it doesn't seem like it did so now it's doing more mocking to to try to get the game to end all right tests are still failing but I do like that Deon seems to have an ability to fix some of its mistakes and sort of continue to iterate whereas with a lot of AI tools if you just tell it hey this is the mistake you made it's not really able to fix that mistake and it'll just keep doing the Same thing over and over again which we've seen Devon

do the same thing over and over again don't get me wrong but it does seem to eventually sort of make some progress even if it is sort of guess and check coding at times which isn't the best but all right so we've been at this for a while and you can see Devon has made a lot of attempts tests are still failing it looks like it's trying to mock out timers now which can be a tricky thing to get right but it's Still not able to solve it it's been doing this for almost 10 minutes

now but again it's something we could have on the background one working on other stuff so it's not that big of a deal how long this is taking I again don't love the guess and check coding because it makes me far less confident in these Solutions it's giving me but maybe it'll eventually find a way to figure this out all right so it's gone back and forth on updating these tests about 100 times Maybe not literally 100 but a bunch it took almost 15 minutes to do this 13 minutes or so but it seems to

have finally updated the pr so let's take a look at this PR see what it did add high scores feature files changed six okay so in app. TSX we instead of just having start game we now have a high scores component I don't I think this class name is something from Tailwind which we're not using so that's that's not going to work and then let's see we have The test I'm just going to assume these tests probably work because they were passing okay so show high scores shows no high scores yet when there are no

scores okay and then we have the high scores component test which has some local storage clearing before and after the scores no high scores yet high scores in descending order which is good displays the at most 10 high scores that's a good test to have and then applies class name prop correctly okay And then we have I I don't really know the the point of checking that that can have a prop is but okay it's fine and then we have the used Tetris test which it added a bunch of mocking for which which okay I

think this is probably a reasonable way to handle this although there's something something to be said for if you do so much mocking granted we want unit tests to test individual units but this test is meant to test sort of the entire functionality of the game and If you mock out all of the functionality of the game and then test that the functionality now works in this like super simplified mocked version I don't know that you're actually testing all that much but regardless is test there and then we have the high scores component which it's

not really following the style we have in the rest of the code base and then we have this class name thing which I suppose is fine I don't love that but okay and then we Have used Tetris it updated for the high scores and yeah so this class name is not going to do anything because we aren't using Tailwind so I'm going to have it fix that and use CSS and I'm also going to tell it to follow the style of other components in the repo so let's say follow the style of other say the

coding style the coding style of other components in the repo we also are not using Tailwind make sure this this looks good using vanilla CSS all right So you can see what it did it added this high scores down here but you can't even really read it because it has white text on a white background so I'm curious to see is it going to figure out that this is a big accessibility concern or is it going to fix it or not okay so it did something but it's still white so let's see can it figure

out that we have white text on a white background or is it not really able to understand what it's looking at okay so it's running into an Issue now that a test is failing it seems like a few but one of them is the fact that it's checking it there was a test that was checking that you can add a class name to high scores which was think a pointless test and also a pointless prop and it removed that prop so now it needs to figure out that this test is pointless and remove it so

we'll see if it figures that out as well okay so it added the prop back rather than fixing the test so let's say we don't Need the class name since we don't use it remove that prop and the test for it it's also still doing con scores instead of using the function format that we use for all the other components I'm also going to say also instead of using const and react. FC use the same coding style that other components in the repo use so I don't know if it's going to be able to understand

I just gave it two different requests and do both of them or if it's Going to only do one or if it like is this overriding this request or is it going to know that it needs to do both of those things but we'll see okay it looks like it's doing both so it's kind of combined the two requests into one thing so hopefully this ends up working it does have knowledge suggestions again so let's see what these are use vanilla CSS for styling components not Tailwind fair enough follow the coding style of existing components

that seems like Something it should just always do but fair enough that's good knowledge I guess only right test for functionality that is actually being used I mean that's I suppose these are are fine but I don't know why you would have functionality that's not being used to begin with but sure I'll approve this actually let's see can I edit this and let's not have this part but we can have this part okay and I like that we get confetti it makes me feel like I did Something good okay so it removed the prop and

it went to just function high scores but we still have the interface for props even though we're not using that so I don't know what the point of that is but maybe it'll figure out to remove it or maybe it's just something I'm going to have to tell it in code review which seems to be a theme of this a lot that you have to really walk it through the code review we're still trying to run npm run lint which is not Going to work oh wait it did I've discovered there's no lint script so

I guess it's finally discovered that although it was able earlier to figure out how to run the lunter but it has many times done this and just went right over it but this time it's decided to actually run the linter and we're also running the test and we have a failing test so it looks like this test is calling a function that doesn't exist all right so now we're removing this Interface so that's good and then we are maybe fixing this test it doesn't seem like it fixes it but we'll see yeah so still having

testing issues and I would imagine this means we're once again going to be in this very long Loop that it seems to constantly get itself in of trying to fix its test it seems to struggle a lot with writing tests that actually work at least in this repository I've given it I'm not even sure I understand how it managed to Break these tests but hopefully it's able to to fix them cuz I thought they were passing a minute ago and I I gave it pretty simple instructions of just like updating coding style and now we

are back to failing tests again but hopefully it's able to fix it and give me a complete and working PR come on on Devon you can do it Devon I believe in you wonder why they named him Devon like was there like a big board meeting of what do we name this guy and like what Other names came up and how did they come up with Devon or does it like mean something does it like stand for something is it just like let's give it a human sounding name I didn't know that there was much

more frustrating than not being able to get test to pass but I think something more frustrating is watching an AI not be able to get its test to pass something about it I feel uh helpless like there's nothing I can really do to help Devon at this point Cuz I don't understand these tests I didn't write them and I don't know what's wrong but I just have to watch it and watch it fail over and over and over again until finally at some point it's going to either give up or do something crazy or tell

me it's finally fix fixed it but it's just like an infinite Loop of of hoping it finally figured it out and then it didn't so then you have to watch it again I guess you don't have to watch it I could go elsewhere but I feel Invested in watching it $500 invested to be specific all right we are now 11 and 1 12 minutes into Devin trying over and over again to fix the the test it wrote and I thought it had this passing at one point although maybe it didn't I don't even know at

this point um but Devon keeps on going I guess that's the benefit of a AI software engineer is it doesn't get frustrated it just keeps doing the same thing so it's got that going for it I like how every time it Does this too it like edits the the comments but it doesn't really like a lot of times it feels like it doesn't necessarily even need to although I guess in some cases it does but like it'll just like add and remove random comments there's not a great example here but it's quite entertaining I'm also

pretty sure that we've added and removed the same import at least a few times at this point I'm pretty sure I've seen this already AB boarding after Running 10,000 timers assuming an infinite Loop okay so Devon has created an infinite Loop of timers which is fun I don't I don't really know how it managed to do that but good for Devon is this the future of software engineering just watching AI attempt to write code and never figure out how how to do it the future is now it's interesting it runs get DEF a lot to

like find the changes it made itself which I guess makes sense but it's also like have you Forgotten what you did which I guess if I did this many attempts at the same thing I had at some point forget what changes I made and what changes were already there oh what do you know the tests still don't pass who would have thought could have swarn these tests were going to pass at the 16th minute I have an idea for cognition if they want an extra Revenue Source they make Devon free but as you use Devon

you can bet on how long it's going to take Devon to do Something and whether or not Devon is going to be able to do the thing like you could have like an over under of 30 times that tries to rewrite the testing file until it figures it out an idea we could have a bit of a premium but a a betting model associated with it that's some some good free business advice is going to drive me insane or it already has I'm not sure like part of me wants to tell it to just delete

this test so we can move on but for science I want to Figure out if it ever figures this out like it seems to find some kind of solution to one problem and then get back to a problem it previously had solve that problem and create the same new problem again and it's just kind of like in this Loop of effectively trying the same same things over and over again I think because like I've seen this error result. current. move down is not a function like 100 times at this point and it has solved it

before like it's Gotten past this line of code before but then it creates the issue again we got to be at some kind of like Devon doing the same thing over and over again world record right this can't be normal I hope but at some point you would think because it does have the ability to stop and like ask me questions at what point is it going to say okay I don't know how to do this and it's just going to ask me to either fix it for it or ask me for advice or ask

me if it's okay to delete It or give some options like at what point does it give up on this or does it never and if it never does do we just is this how you use up all of those uh credits you were giving or whatever is it just going to do this forever and then run up the bill although it does have like a limit of how many credits it can use per uh instruction I think so I guess it can't run up the bill that high I don't know what that did I

tried to I wanted to see how many like of these Token things we've used so I hit this evaluate the session button but it didn't seem to seem to do anything wait do we have the Test passing oh my we might have done it Deon Deon might have done it oh we have es lint warnings though can it fix the es lint issues without causing the test to fail again I actually I don't even know if it did all right we've got an update to the pr it took a while but we got here all

right so now we have a high scores in here and Got some tests not going to go through all those right now but we have tests that supposedly work maybe WR a lot of stuff for the test and then we have high scores function high scores get high scores Let's get high scores okay this comes from the hook wait but we're not we're not actually using the hook we're just okay a little odd but high scores so I don't know why this this is in a hook file if it's not like requiring us to use

the hook but that's all right I Guess it does call the file in in the hook okay and then we have some CSS all right I'm going to try and run it see if this works and we'll get back to Devon okay so we have the high scores here it is white text on white so it's very hard to read but it is there it says no high scores yet and let's try playing the game and let's see if we're able to get a high score so first of all if I just don't even score

is my high score going to be zero or am I going to not have a High score so let's see how quickly we can lose this all right so my high score is zero all right let's play again let's get a new high score I'm just going to get score of something actually not trying to win here we can just do this do that all right now I have a score of 100 and let's just lose the game and see if we have our high score yep we have a high score of 100 now perfect

but we do have this issue of white text so let's see if it can fix that and then I think We're probably good to go on this PR the text is the text for high scores is white on a white background change the text color so we can see it so it's once again running into this issue where it has like this page not found thing which I I don't know if this is uh because it once again hasn't run npm install or or if this is some Devon related thing because we got this air

earlier and I think npm install fixed it but it might also just be something related to like The Devon servers I'm not not entirely sure all right so it says it fixed it although I never actually saw it check because it just ran into that issue so I'm going to check this manually and we'll see if it actually fixed it okay so it changed the ENT entire background of the website to be black which I don't hate we have like true dark mode going on now and it made this text all white which is interesting

uh so I actually kind of like this and yeah this this Seems to be working and also you noticed it had my high scores saved from the previous time so if I refresh the page we can see we still have the high score so it is using Local Host successfully so I do think this is a working PR and we can finally actually merge this all right merge pole request confirm merge all right so what do I think of Devon now that I've paid $500 to use it is it worth $500 I don't know it

it could be for some certain people some certain Organizations but it sort of felt like I had an intern so I don't feel like this is a junior software engineer I feel like I had an intern and this intern was like a first year college student at best but also has some like oddly deep knowledge about some random things so they're like a special firste college student but they also have sort of like a Ron bergundy esque vibe to him where like Ron bergundy will read whatever's on the teleprompter and Devon will kind Of do

whatever you tell Devon to do but it won't really think about what it's doing very much and it just kind of does the thing and like oh I fixed the test but really it fixed the test by breaking the code so it feels like it really needs to have its handheld and you need to give it super super Specific Instructions of what you want it to actually do and it's not like you can just have it doing stuff in the background while you work on other stuff Like you kind of can but you're going to

have to be going back and forth so much that I think the context switching much like having an intern is going to make you less productive on on your own work because of the fact that you're having to do so much handholding with Devon and ultimately I just don't know how much I can trust the code it wres like I feel like I have to spend a lot of time going over that code and almost reimplementing it myself because it doesn't feel like It's adequately testing its own work before putting up a PR unlike how

a real software engineer would now if you're working in maybe like a more backend and logic focused repository versus this frontend repository where it's easier to have good robust testing then maybe that won't be as much of an issue but in my experience here I felt like I had to go and manually test everything which is just creating a lot of work and a lot of these things I feel like I could have Just done on my own much quicker so do I think it's a scam no do I think it's worth $500 also no

at least in most cases but if it's worth $500 to you then by all means spend your money on Devon it does work to some extent you'll just have to be very very careful with it and be very careful with what tasks you actually give it and now there are a lot of other companies also making big strides with AI so if you want me to look at some of the other products do Let me know and then take a look at this video over here where I actually went all the way to Vegas to

see some of Amazon's big new releases as they relate to AI as well as a few other pretty interesting things for us as Developers