everybody welcome um so just appreciate everybody joining today for what is the last session one of the two last sessions here that you can choose from today and hopefully today's been a valuable rewarding experience for everybody i'm going to take a few minutes and talk a little bit about how the session is going to work although for those that have been in the other sessions this won't be entirely unfamiliar and i'll also introduce our two speakers ed robinson and lisa spory um just a reminder that if you have questions throughout the session you can ask
them using the q a feature and at the end of the session we'll sort of go through those together um and please you know as as we do this um be mindful of anything that you might want to ask and feel free to ask as much as you want so um i'll introduce quickly are two speakers so ed is an information assistance professional with 26 years of experience in the industry and 16 of those in international development he manages a software development and deployment team designing and maintaining technical solutions for projects in low resource environments
in the healthcare education and agriculture sectors and and when you get your session started feel free to or your section started feel free to add to that and lisa i have a pleasure working with a little bit here at bao systems where i work and lisa has over 20 years of experience in federal i.t consulting and software development and has successfully led software development projects at dhs the department of homeland security department of defense and pioneered the application of agile analytics in the federal space before she came and joined us here at bao she served
as the vice president at a technology and i.t solutions company that delivers mission control solutions and digital modernization services to the federal government and now as the cto here at bao systems she is responsible for overseeing the delivery of a portfolio of um lots of different programs across all of our different clients and implementing agile application development devsecops and data analytics and lisa please feel free to add anything i may have missed there so in this session we are going to talk about dhis2 hosting requirements and how these have grown in complexity alongside the expansion
and adoption of dhis2 so you'll have the opportunity to learn more about next generation options to support a variety of different hosting needs from using docker for containerization to advanced infrastructure design for maximum security and compliance and we may also talk a little bit about edge computing compliance and turnkey interoperability and both of you please feel free again to add anything i may have missed um i think with that i will hand it over to ed to start and lisa will take over about midway through thank you hey everyone thanks for joining the session i'm
just going to get my slide deck up give me a sec to share my screen here and then i think we should be good there we go right there we go um right as mentioned i'm ed robinson uh i'm technical director at project balance i've been with project balance since about 2016 although i've worked with the team for many years originally starting in about 2004 when i entered the international development space in 2004 i joined with a company called futures group and worked on a project called age relief which was my initial foray into international
development and my first experience with dhs2 was actually in south africa on a couple of projects that that used dhs2 back when it still ran on ms axis and of course it's evolved a lot since then um so this is really just a a brief discussion about um dhs2 and docker in production and it's specifically a use case that we've encountered through one of our clients um and just to briefly dip into the the water on on docker and what it is i think it's just important maybe you know for those who uh are familiar
with the name but may not be familiar as much with the technology it's really just a software platform for developing and deploying applications in lightweight virtual environments called containers it's it's a form of virtualization effectively so talking about virtualization everybody you know in the tech uh field tends to be familiar with virtual machines and so i did a quick side-by-side comparison really virtual machines versus containers and in the container context here we're specifically referring to docker um but with virtual machines there are two types there's a type one which uh your your hypervisor which handles
the virtualization the guest os and and your applications uh runs directly on top of your infrastructure and type two you would typically run an operating system and you'd run a hypervisor under that and you would run your virtual machines under the of course top one is uh generally more efficient because you don't have a host os um and of course type two well we all run them from time to time on our own uh machines especially developers we tend to run a whether it be virtual box or whatever hypervisor you you prefer you may have
a couple of virtual machines running underneath there and of course containers similar although the um the container virtualization happens uh at os level so effectively these are both forms of virtualization and two of the most common so virtual machine is really hardware level virtualization the hardware itself is actually virtualized along with the operating system and the software on it virtual machine really provides machine level isolation and each vm is an independent emulated machine with the separate operating system drivers etc virtual machines are flexible but obviously more effort to configure and administer their independent machines effectively
with their own resources to configure cpu ram storage etc and they require more resources they include hardware configurations and an operating system and applications on top of that whereas containers and and docker in particular their os or application level virtualization objects so containers provide process and application isolation which is great each application can only interact with what is necessary for its performance job and that provides better isolation and only required libraries for the application need to be included in the containers which makes them lightweight so both virtual machines and containers can host multiple applications but
typically a single container will be deployed for for each type of application and i say type very loosely for instance you would still have tomcat and java typically hosted together uh you might have apache and php hosted together in a docker container so getting back to our used case um starting off with the old environment so i worked with with the team on a project at national level that kind of inherited an old system from the previous iteration of the same project the project had a very inefficient and difficult to maintain system they they had
two servers sitting at an isp they had an aggregate and a tracker server and there was a firewall slash router and then at the project office another firewall router and a donor and test instance of dhs2 they're two different domain names to access the two different systems and the firewalls used network address translation so the two servers were accessed on different ports which made things very difficult to manage and so the requirements in country were first of all in country data stories and this is something that we come across fairly regularly you know it you
you very often uh find that uh on a particular project you you may have a ministry of health tell you that they don't want their data um posted out of country um you know it's you can really have a strong case for using a higher tier service provider that provides all sorts of infrastructure services and security but sometimes it's just really difficult to to convince the people who make decisions that uh that that that is a secure environment uh and you find that occasionally having data stored in country is a requirement of a project whether
it's aggregate or or personal information sometimes there really is just that resistance to to using a a cloud-based provider and that's what we found in this instance and what we had was co-location hosting at the local service provider so a um actually a local cell phone company they they had a a data room with uh servers and we could we could rent space on the rack from them and we provided the equipment and linked it up and of course the other need was to have multiple dhs2 instances we needed two production instances and two qa
instances and we need to be able to improve efficiency with deployments for for testing and new program setup and so the project replaced the two aging servers of more robust machines 32 and 64 gigs of ram new generation xeon cpus and larger storage and we just took that as the perfect opportunity to redesign the hosting architecture now if we go back a little with uh with dhs too uh this is really the the typical um old style uh setup for single server environments you would you'd have your infrastructure you'd install your operating system and you'd
have all your your applications really you know running on on your operating system and only one dhs2 tomcat instance was typically run because of the dhs2 home environment variable uh that sort of dictated that and then of course you know the current traditional method i call it traditional is just you know the old school way of installing all your applications under your main operating system now you um you you can support multiple instances on the same machine through the you know tomcat instance create method um all components are deployed under the same operating system um
and all os and application processes execute within the same space and occupy the same file system now that is typically a a higher security risk we've actually seen scenarios where servers are not well maintained no um security updates have been you know done over time and the most common thing to be compromised is tomcat um we've we've had a couple of instances with tomcat because it's exposed maybe there's no reverse proxy or whatever the case is but tomcat gets exposed and is compromised and we've even seen malware loaded onto servers and uh you know running
in the background that way and of course access to the file system where servers have been set up badly with uh um with root uh root user running all your major services uh and then of course with one of those services compromised uh there's root access to the file system so the case for containers and really it's three simple reasons they provide great security there's a need for isolation you can reduce the tax surface area by running containerized applications so you know a software vulnerability will only expose the container contents and then of course for
practical reasons running multiple instances of an application on a single server that's a great uh reason and then of course efficiency this is my personal favorite ability to deploy new instance quick new instances quickly it really is a lot easier to deploy once you have the system set up and and so this is really what our production server architecture looks like currently incorporating docker so we have our router connect against the outside world on the servers we we have our hardware operating system we're running ubuntu 1804 and then docker containers are our databases tomcat and
java and we have additional applications so we're running a couple of apache and php applications you'll notice that the tomcat each tomcat instance has its own db and i'll discuss a little bit about this in a minute um there are people who advocate against running uh databases and postgres postgre sql uh in production on docker um and then we use openvpn to prevent exposure of sensitive ports we only expose our port 80 and 443 for https and http so our reverse proxy our nginx uh handles all the um input output via the the router and
it assigns the traffic to the different servers accordingly depending on the euro url access which clean things up really nicely um but with openvpn what we we do is we expose openvpn uh to outside world and we have certificate-based access to a vpn which allows us access internally to the to the operating system level and allows us to connect to our postgresql servers directly and obviously ssh into the server and do all of that our stuff um so right now the benefits of docker uh immense quick and easy deployment of new instances as little as
five to 15 minutes for complete dhs2 instance from scratch now um that includes configuration and you know maybe up to 20 minutes if you include image download we have independent management of existing instances we can take an instance down we can take just the tomcat portion down do something with the sql database bring the top cap portion up which is great and then of course a unified component management tool so using using the docker management tools it's a unified system to be able to manage your your instances and of course improved security and segregation of
the different processes and we use docker volumes and and that's really to separate data and configuration from the container this is a really cool feature of docker it allows you to have your custom files and and data live outside of the container in the the host operating system and be shared inside the container so things like dhs2. your tomcat web apps root directory the tomcat config file nginx config files anything that's customized within the container we we use docker volumes to sorry to host it on the operating system and the great thing about that is
it means we can deploy a new version of our tomcat um content sorry i'll just skip forward when i believe right sorry back to where we were um so it allows us to to deploy new versions of our tomcat and and other containers um without having to redo configuration uh etc that those files are they remain on the host operating system and they are then linked up inside that container and the new container runs uh fairly instantaneously there's not much to do there what else is this so i think really that's that's the crux of
it is is quick upgrades uh and quick deployment of new containers so in hindsight is there anything we do differently uh and and what lessons have we learned so for our production server i think what we may do is switch to a switch our sql server as a docker container uh we message that for a single uh sql instance installed in the host operating system on another machine we don't really have a unique postgres sequel version requirement per dhs2 installation and doing that will save ram running a single instance um storing postgresql data outside of
the content on the host machine using docker volumes doesn't give you any benefit really in terms of upgrades you will still have incompatibilities when you upgrade the docker container with major versions you basically have to do the same process of backup your database install the new version upgrade and restore your database and so for us that's something we we want to consider is is maybe moving to environment where we have a single sql installation on the host os rather than postgresql docker containers it also means one sql instance to maintain an upgrade and then lessons
learned so shared memory size uh that's something we came across uh there's a an out of memory area that crept up occasionally and we discovered that the default shared memory size for postgresql containers tends to be on the lower side and for the hs2 we had to up that to one gig which was fairly easy to do with that dash dash shm dash size equals 1gb and that result that individual error this is typically what we would probably do uh in future is have a database single instance hosted on the operating system and our various
tomcat and apache php containers connecting to the single host instance and still have our reverse proxy within a docker container and so really the benefit here is we have a single db instance lower memory footprint we have a single db instance to upgrade only disadvantage is all apps share the same postgresql version and configuration which may be a little less flexible and may be okay for your situation it may not be it all depends on your individual scenario and and that's really so i've got a few links here for further reading please do look up
project balance on linkedin there's a link in the uh in the powerpoint um and follow us in the in the coming week i'm going to release a blog post that takes you through our abductor setup from beginning to end um just a word on on that uh our docker setup didn't use any special tools like docker compose or anything we use the standard docker.io tools that come when that package is installed um and the reason for that really is it it gives you a great overview of the ecosystem and and when you set up a
a a a dhs2 instance from scratch like that you learn a lot about the system you understand it better there are some great command line tools available now provided by the university of oslo so if you look at the bottom link with dhs2 and docker you'll see information there and including uh some of those tools those great command line tools that you can use to stand up docker instances uh and then of course there's some some good reading on on docker itself from the the docker docs the the getting started guard at the docker compose
guide and that's it thank you please do pop your questions uh up and there is a poll um in the poll section so feel free to to take that poll all right thank you ed let me share my screen okay so um as scott said my name is lisa sworey i am the chief technology officer at bao i have been with bao all of three months now so um no heckling the new kid from the peanut gallery please um but uh as scott said um you know i am new to the um health and international
development space uh but i do come with about 22 years experience uh developing and supporting various applications with about 18 years focused specifically on data warehousing and analytic type use cases a little bit about bao for those who may not know bao is a company that really has deep roots in global health and international development and and has really branched out to become both a product and service company focused around data the team at bao possesses really deep knowledge both in subject matter expertise around public health and health information systems as well as extremely deep
technical expertise in terms of data science data engineering systems engineering and software engineering we provide as i said a variety of different services and product lines with really the central theme being focused on data we do data collection data management analytics and we have a variety of product lines centered around these core services in terms of data collection mechanisms such as our dharma platform our bao analytics platform to perform advanced analytics on health and hmis data and of course our dhis2 hosting services bao is is a technology company that really got started as a dhis2
hosting provider and consulting firm and over the years our aws cloud-based hostess hosting services has really been the longest standing uh product and service that we provide uh we are at the point now where within aws uh we have about 400 dhis2 servers that we are managing um and we have a variety of different tiers in terms of hosting plans that go just beyond running dhis2 in the cloud but providing more of a platform of a service in terms of supporting upgrades backups um and built-in redundancy and availability uh so again not just taking dhis2
throwing in the cloud or running it for you but providing more and end platform-as-a-service features so when we talk about um really what what are we talking about when we talk about next-generation hosting what does that mean um as our services and our user base has grown we're starting to see more sophisticated needs as we get more enterprise level dhis to customers coming to us the standard plans that we offer are really great for small organizations who are mostly self-supporting and they don't have complex data requirements or security requirements but as i said over time
we've really seen an increase in the scale and complexity not only of the dhis2 code base itself but also expanded requirements of our users talking about implementations in the hundreds and thousands of users very complicated analytical requirements customized applications and workflows and even a demand increasing for security and compliance requirements and support for that as ed touched on too there's also instances where uh there's countries that have data sovereignty laws that require that health information and pii stays within national borders and we've also run into situations where um you have either a very remote uh
installation and and poor infrastructure or you're trying to support a disaster relief effort that again is facing um very unstable infrastructure in terms of access to the cloud and and even just basic um power and and access so within the context of that we really started to look to docker to provide the foundation to get us where we knew that we need to be in terms of really offering that next generation of services to our customers and being able to expand our hosting service operating we also are looking to expand to include multi-cloud and in-country
computing use cases as i mentioned our instances right now that we're supporting are in aws but looking to be able to also provide that and branch out into azure we also really need consistency across the portfolio so that the users that we're supporting with 400 different servers and growing we need to have a standardized and consistent experience for our users and containerization really has been able to provide the foundation to provide these enhanced enterprise hosting solutions and be able to scale while providing high level of customer support and stability because we're doing things in a
repeatable way in a consistent way uh so what have we done so far at bao in relation to docker um so our systems team has built a bao systems docker dhis2 image that is a production ready image that has been open sourced and released to the public you can access it at github on the url on the slides baked into that are a lot of very highly configurable deployment options that are based upon our experience hosting dhis2 instances for our customers um this includes things like the ability to deploy auto-generated dhis2 config files depend on
other network services and initially initialization and non-interactive startup scripts we also in this docker image have support for horizontal scaling so as ed um showed on his diagrams where you had multiple contact tomcat servers running um our our image supports that as well um and we actually have um bo edge servers in sub-saharan africa uh running dhi's two docker containers um and again that that's um expanding out to the edge computing which i'll talk about in more detail momentarily so where do we go from here as we're looking to really be able to productionize this
just to help us manage all of our 400 instances expand out to the other clouds and work on the edge we're continuing within our systems engineering group to develop expanded features so for example we're working on creating docker sidebars for helper features so that we can have standardized helpers in terms of log shipping advanced monitoring and observability custom maintenance scripts again looking for that consistency on a lot of these cross-cutting concerns we also are now going to move forward in terms of being able to orchestrate we need to be able to orchestrate and manage the
containers and so we're going to be moving to kubernetes for container orchestration so that we can standardize deployment across the cloud offerings and across out to our bao edge servers and kubernetes is going to give us the ability to do that and support multiple platforms while maintaining that level of consistency once we have that then we need to operationalize it and deploy it where we're we're looking to um not only stand up new instances based upon this but go back and convert existing uh servers that we're supporting to our new containerized and orchestration platform if
anybody wants to know more about the bao dhis2 docker image that's been put out there there is a session a boot camp session tomorrow being hosted by our own alan ivey and chase freeman where they're going to go over and walk through learning how to install and use the docker desktop for dhis2 on your own computer set up a working environment and work through some use cases around that so great session if you're interested in diving into more of the technical details so bao enterprise hosting then as we talk about next generation hosting is really
that next level where we're looking to get as we expand out our our offerings of hosting um in a nutshell enterprise hosting is really a bundling of environments and hosting services um with the ultimate goal of really providing a higher level of technical touch to our customers that are enterprise level customers and so some of the features that are being baked into what we're packaging as enterprise dhis2 hosting include providing multiple environment support so that you can have development test stage training environments so that you can control the deployment of flow of code through through
a true pipeline including opt-ins for automated dhis2 patching and upgrades advanced monitoring tools and advanced server health monitoring features elastic sizing and auto scaling to be able to really expand and collapse with fluctuating demand as well as uh integration with some complementary bao products such as our dharma platform for data collection or analytics platform for doing advanced analytics and reporting on dhis2 data um also looking to include in that quarterly server health reports and really more of an engagement from an account management perspective around really working with our enterprise customers to look at the health
of the system look at the forecasting of sizing and demand and needs and really work hand-in-hand in collaboration around the budgeting and the planning around supporting an enterprise instance we're also looking at offering some advanced optional add-ons for people who need it um one of the biggest areas where we've seen that is in the area of security and compliance so some of the options that we're offering are um for example web application firewall deploying raf being able to set up and routinely run nessa scans appscan and then even going beyond to engage in a full
compliance as a service type model if there is an enterprise customer that needs to get through a very rigorous security audit such as fisma or fedramp also looking to be able to optionally engage in with customers who would have a need for really high availability infrastructure requirements where you need to have an immediate active disaster recovery solution that can immediately flip over with zero downtime um being acceptable um and the potential to engage for any kind of real true critical incidence response again servicing more of these enterprise customers who might have much higher levels of
mission criticality around their installation and deployment so why would this enterprise hosting enterprise dhis2 hosting be right for you i think if the word enterprise is in your system that might be a good a good tip off right there really anybody who's doing an enterprise or institutional level mis if you have a global level project with thousands and thousands of users if your dhis2 implementation is using complex workflows custom applications and really a lot of complexity that's requiring really stringent testing and control through dhis2 upgrades and other um consuming of server resources i mentioned if
you feel that you really need to have either high availability or there are extreme peaks in demand where there wouldn't be a need to have a hosting solution that is going to be able to expand and collapse as you go through um periods of um surge in terms of your user base so if there are times of the year where you have really increased demand for reporting or massive data imports happening and you need the ability to expand and collapse to adjust for those workflows enterprise dhis2 might be useful for that as well as anybody
who really needs to have more granularity and information about their server health so that they can really do much more um proactive and detailed planning and budgeting for future needs that would be another reason to look at moving to an enterprise dhis2 hosting uh so the edge um so what is what is bao edge and what is edge computing um edge computing is really um a paradigm where you're bringing computing services closer to the consumers or data sources rather than trying to bring all of that processing and all of that data up to the cloud
it typically can be done and and used in industries like manufacturing robotics ai and machine learning where you have either massive amounts of data that's being collected at the edge at with sensor data for example that needs to be processed and there's there's limitations and reasons why you wouldn't want to try to complete that processing and bring it all the way back up to the cloud so it's layering out where the data is reciting where processing is happening that most makes sense really given the reality of um infrastructure and your ecosystem and the data bao
edge is taking this concept then and bringing dhis to hosting out to the edge and um in this situation it's more to address the um issues around data sovereignty and the stability of infrastructure and internet access than necessarily um high volume like ai and machine learning data processing but the concepts are the same so we envision supporting different edge deployment scenarios a very simple scenario is really again driven by data sovereignty uh you have pii data and health data being collected in dhis2 and basically that data is not allowed to leave the borders of the
country so it is not practical it is not legal to be able to take that data and host it in a cloud instance of dhis2 in that deployment scenario you would have a bao edge device that would stay out in country and serve as the in-country repository for that data that's a simple use case there are potential for there to be hybrid options though as i talked about with more traditional when you look at just edge computing where there's layers of processing that's happening at different levels so there can be deployments of bao edge where
you would have that patient level uh personally identifiable data living and staying at the edge in a bao edge box and patient level data that's being aggregated and obfuscated basically the pii removed from it being pushed up to a central cloud-based dhis2 instance or other data store that would be doing the the individual level collection in country but allowing there to be higher level reporting in the cloud for the roll ups and the aggregate level data and information another use case we envisioned would be where the actual data and the dhis2 instance is running in
country but it may be allowable that a backup could be stored on a secure cloud server that would not be accessed by end users but that would give a really great um easy and cost effective way to manage backups because that is one of the challenge with the edge and when you're working on edge computing unless you're going to have physically separated and other device ready and accepting backups you really need to have a backup solution and so a great way to do that if it's allowed is to be able to send those backups up
to the cloud but not actually run any dhis2 instances in the cloud um and other use cases that we look out in um to the future would be again i talked about there being sensory data being collected um and pushing that to the edge in terms of being able to aggregate very low level sensor type data to very um cheap edge devices that's then being aggregated and sent upstream ultimately to be consolidated for analysis in a data store that may live in the cloud so um just to summarize why would you necessarily want to use
edge what are some of the use cases around that definitely data sovereignty is a big one if you have data that just cannot leave the country and cloud-based hosting is not an option edge is a good solution where you can keep the data in country and bao will manage that dhis2 instance remotely on your behalf so you're able to have a hosted instance without having to actually have that data leave the country another great use case or reason to look at expanding the edge would be for implementations where you really don't have stable internet connectivity
um and you're not going to be able to have and and be able to support a solution that requires consistent internet access to be constantly using a cloud-hosted solution um similarly it could just be latency issues so maybe maybe the um the internet connectivity is stable but it's very slow and it's just not feasible these would all be reasons to look at potentially leveraging an edge device either in a completely on-prem isolated instance or looking to do some kind of a hybrid solution to bring some of that data up to the cloud and that's all
i have um scott i will turn it back to you if there's any questions for ed and i thank you great and so far i only see one question in the q a if anyone has any others feel free to add them in there but i'll summarize bob has asked ed uh when you run the postgres sql on its own outside of containers how can you ensure that postgres sql doesn't greedily consume all their resources ah the hard questions for me yeah that's a that's a very good point uh you know pastor sequel itself doesn't
have any sort of mechanism to just say limit memory um it's not an easy thing to do short of tweaking uh you know a lot of the uh the work mem and other settings um it's it's a bit of a a trial and error game on that side you know for our situation what we'll probably end up doing uh is we'll split up the posterior sequel onto its own dedicated box and have uh the the main box running exclusively you know for docker um that's more than likely the the easiest way to deal with it
um having it outside of a container it does tend to um it does tend to be a bit of a resource um but absolutely there are things that can be done within postgre itself um but yeah it's a it's it's an ongoing challenge uh getting uh resources for postgres sequel tweet can be a bit of a nightmare i sometimes i don't have a clear answer for that let's apologize i'm sure you've probably experienced a lot of this yourself judging by the question yep see grips of course yeah so there's a there's a great opportunity i
guess we run out of time but to have the docker lxd discussion uh lxd is actually a great system and that's kind of where it all started for me and content containerization in fact um your scripts bob have been uh really really useful we've used them in trainings and uh it was a really great experience and of course the big difference i guess is is the fact that you your container is effectively virtualizing the operating system and uh and apps whereas docket is really just lightweight geared specifically at apps so there's a lot of goodness
to that obviously resources may be a little heavier but um you know there's definitely benefit to uh to contain rising things at the operating system level as well and and absolutely i encourage people to explore lxd as well um i have a look at bob scripts uh on github there's some there's some great stuff there and um i don't think there's any reason why we can't take another question so lisa there's one for you that says can you talk to what good environmental management discipline involves really what we're looking to do here is to be
able to standardize deployment standardized configurations through containerization really again it's to drive that consistency so that you're not looking for relying on manual discipline when you're managing various servers um i would say the goal is to automate wherever possible and that's really where we are trying to move to and looking to leverage containerization and orchestration to be able to handle that for us to have the orchestrator be able to manage the containers on our behalf and focus more on being able to monitor the services and the quality of service that we're providing to our customers
yeah i guess some of the just pop in there um i'm not sure but i assume uh you're referring to server maintenance as well so i mean you know what comes to mind when when i hear server maintenance is keeping things up to date really important software prioritize security updates don't don't slack off on those that really is important to keep software up to date today you know if you if you want to look out for a couple of cve bulletins out there and subscribe to them and look for all your major products on those
lists to see when new vulnerabilities have been identified uh and take action um but definitely make sure that you're doing regular at least security updates on your server and of course uh when you deploy always always always consider the principles of least privilege don't assign rights to um to users or um or processors uh that don't need those rights lock everything down um you know if there is a breach uh it will basically reduce the attack surface area and limit the damage um things do happen to the best of us i mean we're seeing people
in the news every day we saw microsoft recently you know huge companies with massive budgets and a massive cyber security budgets that are um you know having having to deal with breaches uh it's it's often not a case of if but when um so it's it's really important to to be prepared and of course backups um you know a number of times we've heard of uh organizations that have that have been compromised and you know whether it's ransomware or whatever the case is they've lost their data and they've lost their backups because they've all been
on the same system or the you know the backup is available from the compromised device or whatever the case is but their horror stories out there so definitely principles of least privilege and backup are really important pieces and it's an ongoing thing it's not just a case of set and forget so do check up on your servers check logs consider using third-party logging services uh datadog and others provide you with insight into your server with really nice interfaces be able to see how things are uh you know are going and uh and there are other
providers uh intruder dot r uh you know will give you insight into any vulnerabilities known vulnerabilities on your software platform etc but maintenance i think really um you know around securities is is probably the most important uh important aspect yeah that is a very important incentive to know what you have running where at what version right and that that's really what a lot of this is about i mean especially when you look into a lot of vulnerabilities being around library and dependencies um i remember the days when we used to do manual deployments and um
you know we were looking at file sizes to figure out what what version of a library do we have there right because it was so difficult to tell and to know was it put on every single server that we needed it to be and really trying to go through that troubleshooting and as you start to manage more and more instances of servers you can't be leaving that to guesswork right that that needs to be packaged together you need to know what versions of containers are running where um and yeah as as ed said that becomes
absolutely necessary in this world where security um vulnerabilities just fly all the time out of nowhere right and you have to respond so very quickly yeah exactly it happens overnight you know what works today you know it doesn't necessarily work tomorrow you may have vulnerability tomorrow it's a constant battle that is part of the job and i think it just needs to be put into the budget you know keeping system secure is everybody's work and it's you know i think that's uh i guess a talk for another day but security is everybody's problem uh you
know very often the the threat is on the inside and it may not be intentional it may not be malicious it may just be an accidental email that was you know clicked somebody fell for a cleverly designed phishing scam whatever the case is but those threats are very real um and please don't expose your sensitive ports on the internet i see it again and again at large organizations uh where ssh and other ports are are exposed of the internet with poor security uh you know maybe even password-based security etc at least have a vpn or
something um hide all your sensitive services behind that your rdps and things like that um and really there's there's good free software out there to to set up vpns there's no need to uh to to have systems uh be exposed like that i think if you just look at the logs of attempted um attempted breaches or attempted uh uh even just logins to services like ssh and rdp uh on any system you'll be quite shocked to see the the number i mean i've seen servers with thousands of attempted brute force attacks and on an ordinary
day to an rdp that port that's up now but it's like a red flag to uh to their doctors yeah so i'm coming to bao from a world where security compliance is the name of the game um you know i've been i've spent the last 15 years for example supporting department of homeland security where everything we did had to go through extremely rigorous security and accreditation processes one of the driving reasons why my application development teams um and and my systems teams we were pushing to go to um infrastructure as code and configure continuous integration
continuous deployment type environments is to be able to push this concept of compliance's code right um where the security compliance becomes part of your infrastructure and your code deployment pipelines you're running the security scans you're deploying hardened images you you have control inventory versioning over all of that with really that ultimate goal to being able to ensure programmatically and through that that infrastructure is code and continuous delivery and integration services that you are always in complete compliance i i spent a lot of time at uscis which is united states citizenship and immigration services and i
was able to get my teams to appoint through automation and through following these practices where we actually did we stopped going through an annual recertification process from a security perspective and we moved into a model that was called continuous approval to operate and that's basically because we got so automated to the point we were running our scans as part of our code deployments um we had the appropriate agents running where we needed to have we were tagging servers that were up all of that was automated to the level where security was able to say all
right we we're going to routinely look at your scans we're going to look at your security logs but we trust you to be constantly compliant and that really becomes the goal um i mean yeah we talked a lot about um standards and and services but yeah i mean to ed's point security is really um the crux in a lot of this thank you to both of you for your insights and your help and for the questions in the q a in the chat
