for a long time I would start up my computer open my code editor and run my daily scripts manually and watch as they requested transformed and move the data around I needed it worked fine but I knew there was a better way I just assumed it was complicated but it's not in this video I'm going to show you the easiest way to run your python code on a cloudvps how to set up and manage that server how to get your code on there including how to handle Python's virtual environments and how to run it automatically at the intervals that you need but before we even think about running some code though make sure you add this one thing that if you don't you'll have a world of pain trying to figure out what's going on when your code stops working or even knowing that it's doing what it's supposed to do I'm talking about logging and it's vital to go through your script and add in any sensible logging output where it makes sense I usually put it in places the script could fail and include a start and finish line so I know the whole thing executed properly I'm not going to go into logging here in detail but here's a simple logging config that will have a day a Tim stamp and a logging level output for you if you're struggling a good starting rule is to replace any print statements you made with loging doino so for the server I host all my code on digital ocean but you can of course use any VPS provider that you prefer if you want to do exactly what I am though there's a link to do below that will have some free credit attached to it something else that is mandatory when running web scrapers in the cloud is is a set of high quality proxies this video is sponsored by proxy scrape these are the proxies that I will use for this project and also the ones that I've been using personally in all my other projects too we have access to highquality secure fast and ethically sourced proxies that are going to allow us to run code on a shared VPS and not get instantly blocked there's 10 million plus proxies in the pool to use all with unlimited concurrent sessions from countries all over the globe enabling us to scrape quickly and efficiently I'd recommend the residential proxies as these are the best option for beating any antibot protection on sites and with auto rotation or sticky sessions this is the simplest but most effective way to avoid our projects being blocked and allow us access to the data we need it's only one line of code to add and then we let proxy scrape handle the rest from there and any traffic you purchase is yours to use whenever as it doesn't ever expire which is really nice but there's definitely a case for data center proxies though and proxy scrape has you covered there too unlimited brandwidth 99% up time no rate limit and IP authentication makes them a great option for The Right Use case so if this all sounds good to you go ahead and check out proxy scrape at the link in the description below okay let's get this code running in the cloud so once you've created your digital ocean account and logged in you want to come to create I'm going to create a new droplet select the location obviously that's nearest to you we want Ubuntu latest version we can drop this down to the regular SSD and $6 a month I'm going to use a password and then type in your password that you want to use here and then create your droplet you can change the name if you want to this is going to create everything now it's worth noting here that this is obviously a Linux machine you need to have a basic understanding of Linux commands and we're going to be using them in this video if you run Windows I'd highly recommend that you install and get used to using WSL the subsystem for Linux it will serve you well for years and years to come and it's definitely worth learning so if you want to do that you need to follow along if you use Mac you'll be fine so this is finished now I'll have an IP address I'm going to hit copy and I'm going to come over to my terminal I'm going to use SSH I'm going to do root at and paste in the IP address this might not be fully booted up yet but I'm hit yes here and I'm going to type in my password system is booting up okay and now we are in now the first thing that I'm going to do that's just going to make my life a little bit easier is I'm going to install a couple of things because at the moment this is a very sort of raw terminal and there's not a lot of things you can do with it especially when you're used to you know highly configured Linux terminal so what I'm going to do is I'm going to do suo app install zsh again you don't have to do any of this but it makes your life easier get this installed then I'm going to copy this string over from my other screen this is from the om oh my zsh it's just an extra shell and this is the install script I'm going to install this I'm going to set it to my default now you could end up with a little bit of a problem um with things not working properly so what you want to do is is export X term like this you see I've got some issues like so now we have access to all of the commands so I'm going to do pseudo AP install this just makes my life much easier I'm just going to install neovim again I need a code editor because we have to do some mild code editing again this is all optional if you want to use Nano and you don't need zsh or anything like that you can just skip this section but you know this is a droplet that you can to have for a while you want to make it comfortable to use so now I'm going to use neovim to edit my zsh config zshrc and I'm going to come to the bottom and I'm going to create this xterm export here so now I'm going to save and exit out of this which just means whenever I log in I can clear the screen I have uh you know I can go back up etc etc so if you don't want to do any of that that's fine pick up from here if you do and do sudo app update because we want to update our system this is very important because you know security etc etc so pseudo apt upgrade tells us that's 37 packages so I'm going to let this run and I'll I'll come back to you once it's done so we're going to reboot this later we're going to install a few other things first so again pseudo apt install and we're going to want to do python 3- pip because we want to install pip on our machine so we can you know install python packages and then python 312- VMV and then that should be it this is just basically so we can create our virtual environments and then use pip to install packages into those virtual environments okay so that's done so now I'm going to reboot the machine just type reboot and it's going to close my connection and I'm just going to wait a few seconds and then try and reconnect uh it's probably strictly not needed but always worth doing and we're back in and everything's working so from here we want to create a folder for our code so I'm going to do CD into my home directory with the squiggly little line we're already in it I'm going to do make the projects and I'm going to call this Alpine and then projects like so I'm going CD into it this is where we're going to put this spec specific project obviously you can have multiple ones on here we are going to handle virtual environments don't worry python 3- MV MV VMV and we'll create our environment and we'll be ready to push our code onto into this folder so let's activate it Source VMV bin activate and we're ready to go here now there's a few different ways that you can push code onto your droplet the best option is using git GitHub because obviously when you want to update your code you can update it you can edit it on your local machine update your get repo and then pull it back down however I'm going to show you a different way we're just going to copy the code directly with SCP I think it's called secure copy from our from our host machine to our Target machine it's very easy very quick and you know it's nice and simple to do again if you're going to do this a lot use git it's going to be much better for you I'm going come back over to my actual desktop we'll do SCP and the files we want to copy over APK and our requirements. txt this is the actual list of the packages we need to install you could do it manually I've just done it in a text file now we do our our root at and I need the UR the IP address for this again paste that in there then we do colon then we do our squiggly line projects and Alpine like so and it's going to ask us for the password and it's done so if I come back to our uh droplet we now have those two files here and we can do pep 3 install the requirements. TX oh we need to do uh - R and this is going to install everything that I need for this project to run and to confirm that I'm going to do Python 3 and I'm just going to make sure that it runs and it is and we can see right away that they have all of this nice logging here which is going to be saved for us so I'm going to stop this because there's one other thing that we're going to need to do and that's to add in our proxy string if we come to the actual uh file and I go to proxy we can see here that I'm saving the proxy into a um environment variable on this droplet now you can put your proxy string directly into the code if you want to I wouldn't do that I'd keep it as an environment variable personally that's how I manage all of mine now to do that we want to come back to our zsh file so I'm going to CD back out here because it's going to be in this directory and we'll do MVM cshr C Now to create a virtual Environ to create a environment variable well we already created this one we use export and then we give it the name and I called mine sticky proxy because I'm using one of the ones that auto rotates um every 5 minutes I think I said well 3 minutes and then you put the proxy string in here I'm just going to put mine in and obviously I don't want to show you it because this is my actual proxy that I I do use so now that's done you need to Source the file so we'll do Source uh zsh LC which basically kind of just refreshes this file and now to show the string just so you can see that it working you just type dollar sign and then whatever you called it and it should give you this string back up here he says there's no such file or directory that's because I was trying to do something with it but I'm just verifying that it's there so now we can go back to our project file over here activate this again and we're good to go now when we want to run this we're going to use KRON I'm going to talk about that in just a second but first of all I'm going to create a shell script to actually run this um code now you don't have to do this but I've just found this is to be the easiest way and what I mean is that it's basically you know A bash script that's going to go to the directory it's going to activate the virtual environment it's going to run the code and then it's going to deactivate the virtual environment and this is a really nice way so if you run multiple projects that have different virtual environments because they have different requirements you can just create a shell script for each one bash script and then have it run via Chron and it's all nice and separate so I'm going to create a new file and we'll call this one Alpine Dosh and we need to do a couple of things here we need to start this off with our bash script so we will do the shebang line and then we do bin bash like so bash then we want to say that our path is going to be equal to and I'm just going to type this out um this is just so it kind of knows where to find um bash so it can actually work and these are a few different places that it can be which is why they're separated by a colon uh I think if you copy this string you should be absolutely fine so what we want to do now is we want to activate the virtual environment now we to do that we did source so we want to do actually what we'll do first is we'll CD into the directory because we'll go into our project Direct directory so we'll do CD and we'll go projects Alpine and from here we want to Source VMV bin activate then we want to run the python file so we do Python 3 it was AP ptk I think py and then we want to deactivate deactivate the virtual environment so I think that should be fine so what I'm going to do is I'm going to CD back here into my home folder I'm going to deactivate it oh one other thing I forgot to do I need to make this executable CH dot plus X to make it executable projects Alpine Alpine Dosh cool so now we can just run this file and we should start to see our script running and this just means you know you can run this we can call this script from anywhere we are in our in our um uh code in our in our file system so when KRON runs it it runs it it's going to find the right directory it's going to run it and our codee's going to run it's going to activate the virtual environment and then it will deactivate it at the end and everything will work fine for us so now that we know that that's going to work what we're going to do is we're going to do Chron tab Das e now uh it's not going to give me the access I don't know if I can run this with oh yeah NE there we go perfect all right so this is our Chrome tab so I'm going to come to the bottom of this file and we're going to create a new line now I'm going to show share with you a website called Chron tab.
Guru where you can choose the time so the each star is representation of when the code is going to run I'm going to put star star star star star for the moment this is going to run it every minute because we want to see that it's actually working so what we're going to do is we will go our F star and then we'll do our project folder so projects Alpine and Alpine sh like so and then because our output is going to standard out instead of having that log file for you know each one we can just pipe it using the double arrows into a new file so we'll do projects Alpine SL Alpine dolog and we last thing we need to do is this two into and one I think it's to do with the error logging I don't know exactly all I know is that you need to do this right so now that that's done I'm going to hit save I'm going to come out of this and we're going to check a different file now real quick so we're going to check our VAR system log so I'm going to do this we want to do I don't know if it's not in this folder oh yeah VAR uh log CIS log so if we go to the bottom of this file we can see here that our Chron tab has now been updated we updated it to run every minute so what I want to do is basically refresh this or you know have a look at this file every minute and just check to see that it is I don't want to edit it though it's running so I'm going to do cat V log CIS log this is just so I can see that it's working so we can see uh when did I execute it when did I change it at 14501 so in about a minute or so it will run and we'll see that it had indeed run every minute and we'll see that it did indeed run every minute and we go we can see that at 14511 it's run our project so let's go to our project folder CD um projects Alpine and we should have Alpine dolog and if we cat out Alpine dolog it's going to have some information now obviously it didn't get all the way through before you know it's it's got to a certain point but we don't want to run this every minute CU we know it takes longer than a minute but we can see that it is now running and it is now working so the last thing I want to do is Chron tab d e again so let's update this so it runs 0 4 this is going to run it at 4:00 a. m. every morning so all you need to do is now just leave this and Away you go it's done now I know that I've covered quite a lot in this and we've been going for probably about 20 minutes but I think I've been talking through as much of it as quickly as as in detail and as quickly as I can but you know once you know what you're doing getting this set up to run some code takes a couple of minutes at best and it's super easy to do it's very reliable in fact it's more reliable than running your own Home Server which I actually do as well because you know there's no if you have a power cut at home it goes off if if something goes wrong with your Hardware at home you need to fix it but having you know paying $6 a month to run you know multiple scripts in the cloud on digital Ocean or similar you know it's a no-brainer it's up 99.
