ROS Testing, Continuous Integration, and Deployment RIC Web Meeting

Welcome to everybody first I want to say thank you to our presenters you know it's always that they ask we mean when you ask somebody that to put together some slides and I attend one of these so I definitely appreciate it that people are willing to help it out and share their knowledge thank you to the Ross industrial consortia there are three of them now one in in the Americas where I am and then in Europe and we're just Starting one in the Asia Pacific region so they support these kinds of meetings and they support

members and so I wanted to thank them and I also want to thank the participants because without your interest we wouldn't do things like this so I really appreciate it unto the agenda so we have some really really good speakers and I'm really excited to hear what they have to say they truly are experts when it comes to Ross and and into continuous integration test and Deployment so I'm really excited to hear their presentations unfortunately Ian from rethink he he had a lot something last when it come up so we'll have a little bit more

time for other presenters and I do I'd be I I am a little bit sad that we're missing rethinks because I really think that they could have brought a lot when it comes to actually deploying a product and doing software updates and maintenance and those kinds of things so I might ask some of our other other presenters questions related to that because I really felt that's what we think was going to bring but they let's say we have a great list otherwise and I'll let I'll let as we go through the talks will introduce people

individually but very excited about the people who are presenting just some promotional things some upcoming event we have some Ross industrial training here in the States February 13th through 15th there's a link if people are interested consortium members please reach out to us and we can talk about the value of training and in my cost that you might have also there's going to be developers meeting February 14th this is an invite-only meeting but it's really meant for people who have a genuine interest in Ross industrial and and and actually contributing so if you have that

interest please reach out to me there's My email there and I'll definitely send you an invite we can talk about it but this is a little bit of a reboot of previous developer meetings so it'll be it'll be slightly different hopefully more efficient as my goal and then if you're interested in more events as always you can go on our website there and look up what events are going on all around the world now so it's great okay so why are we all here well every so often consortium members will come to us And say

hey I really I really want to know how to do X or Y you know is there you know is there anybody in the community who's really an expert on this I'm I could you know help and so if enough people come to us we decide to hold a web meeting like this because pretty efficient way of communicating this kind of information so that happened this time we actually had several people come to us and said they were interested and you know they Had done enough development with Ross and had achieved enough that they were

actually looking at deploying applications and so you know this this becomes this topic becomes very important if you want to put something out on the factory floor or put out a product or what have you there's three kind of sub bullets here right testing which was really just exercising code pass by executing test cases and within Ross there's a lot of Framework for this already G test unit test Ross test rock launch checked these are great tools but if you don't know they're there you might not use them and you're missing out on the benefit

so I'm hoping that at least some people touch on these and if not I'll I'll add in details certainly continuous integration is the next topic it's basically automated build and test for every software change and we for Doing this we use tools like Travis or Jenkins or bitbucket basically any time somebody changes something you build the software and you run all of your tests and if that all succeeds then that gives you some level of confidence that the change that was made didn't break anything this is actually very important when you're taking changes from around

the world from people you don't necessarily know or have no experience with continuous integration is very a Very powerful tool for forgiving your confidence and then deployment so this is kind of the last thing that happens but it's the methods of installing a robot software and whatever your production system is this isn't something that a lot of people do as I mentioned you know I I hoped rethink would would really expand upon this Wetty yeah I know I will come up in some of the presentations and so this is really Important about how do you

guarantee that whatever you ever software you built gets deployed in a standard way in in a traceable way so you know what's out there and there are tools for that as well what I would say those if you don't do these three things then you don't really have production ready code and you know in industrial you know robotics and particular integration I'm not sure we did all these things honestly and you know but as we're Moving to Ross and more powerful programming environment these things become readily available and there's no reason not to use them

honestly so ok so before we get into our our present presenters I just want to let you know that we will start a discourse topic after the meeting so once I have the video the meeting I'll post that on Ross discourse and I'll try and start some discussion topics and we can basically continue these discussions Long after this web meeting ends and that's what I look like just in case you wanted to know I'm going to bring up our presenter our first presenter is Tully foot from low srf and just remind to all presenters that

I'm going to hold you to your time so please please try and keep it between 10 and 15 minutes I know that's actually hard oh yeah thank you so I John asked me to talk on continuous integration I've seen filippo Kisan that Because of the time limits this is a four shortened version of the presentation that raw Dirk Thomas did at Roth Con this year particularly he covered what's available and build up a fedora including continuous integration package building and API documentation generation I'm specifically just going to focus on the continuous integration since we have

a time on that at the high level when you want to run community integration you're going to clone to Clone the sources get your dependencies build it test it and continuing on you would build packaging and run API documentation we're all going to cover up through the testing of this for this time so when we're putting this together we want to it's relatively simple you want to solve all your dependencies make sure the job is in a clean state and the way we do this is we use the docker container so that every job is

completely isolated and have no side Effects from past instances and load affect future instances we also for better isolation we use captain make isolated captain make is the most common tool we're slowly transitioning captain make isolated or catkin tools which also has an isolated paradigm but on the build farm we use captain make isolated and so it's all the cook we make sure to run the installation step to make sure that things work and then we generate X unit type results Which can be plotted over time and recorded into your continuous integration framework such as

Jenkins and develop things there are a couple of things that our current implementation does not cover well one is the a via of multiple packages per repository you end up with you can conflate their dependencies and we don't completely test the test versus build the tendency to cook declarations so I mentioned capping Meg versus Kakumei isolated when you're testing We strongly recommend cap and make isolated this will let you know whether or not you have some implicit dependencies that are not covered if you would run isolated you're much more likely to capture this you also

don't have to worry quite as much about the exported targets and a couple of other C makes level things that get in either interfere with your system and one thing we do for clarity's on run the Biltz With only the j1 it's not as fast but it ends up with a much cleaner console output which is actually the most important thing so that people know what to D but when they get there now what I'm talking about is mostly builds up across a torg we run it as a centralized service it would be possible for

you to do everything yourself but it's a lot of work and it's not as discoverable or as useful to the community and you know that when you download from our servers You're trusting us that we actually built it and then it's not you don't have to worry about verifying and trusting as many parties as you not to say that you can't do it especially if you do some of these things are valuable and you want to do it on private repositories I'll have more details later but the everything we do is open source and deployables

so it's not that you can't but in general having the community behind it is very valuable The way we work coordinate this is through the Ross distro file here are a couple of links and some examples of what the roster looks like if you've done any releases or anything to the Roth community or Rob Roth packages you will know what they look like but I'm not going into too much detail right now so this is what a develop job does I mentioned this with a higher level earlier but the most the flow is that in

the top left you have Your code repository and by default every hour we quote the build farm will clone your package or your repository and check if there's been any changes if there have been changes it will trade a docker instance install your build dependencies build your package and then install everything and verify that everything works if that works it will then continue on create a new Doctor instance install all the test dependencies build the tests run the Tests and generate your the xml output that external output will be fed into or Jenkins will pick

up that external output after Dockers darker execution is finished and based on the package that XML it will actually send an email to the maintainer of the package for which there is a warning or error if it passes no emails are sent it will also email people if the job goes back to green and it's happy and we also in addition email indicators we also email the anybody who Is committed since the last cable release our last fatal commit now this is valuable as we've been doing this for a long time on the raw spill

farm in the recent iteration however we've been able to add support for github pull request goes the exact is the same job on the right where it does the exact same build in the fall and tests and generate results but instead of sending the email out it will comment on a github pull request status and instead of being Polling the repository for changes every hour we register what a github webhook which means that when you open the pull request Jenkins will be no five and immediately run the build on the branch that you're working on

and not on the main branch there are a few limitations on the pull request right now we only support github and you have to either add rocks full request builder to access to your repository or you can actually set up the web hooks yourself Manually if you don't want to give Ross pull request alert access to do that you can do the same thing with other continuous integration providers for example Travis CI but for any release package we strongly recommend doing it on the main Roskilde farm for consistency so an important thing about continuous integration

is it needs to be reproducible and in particularly it needs to be reproducible locally so that you can debug it because it's really Very frustrating when you get an email from the build farm saying your package is failing this test and if you run the test locally it does not work or the test does not fail in the passes so to make sure that that doesn't happen the Roth's build farm packages all have scripts that will run exactly that same build test install build build test kept run tests and check the results this means that

for any job that runs on the farm you can reproduce it locally on a Machine and the other thing if the advantage of this is that you can actually run this on Travis or any other system where you want to run your continuous integration you can actually get these same scripts that are very principled and install things from scratch exactly has declared in standard way as I mentioned the with the continuous integration is all based on a repository so if you have changes in multiple repositories it's very hard to Run LOC completely automated continuous integration

do that end we have another solution which we call the pre-release job and what you can do with a pre-release job is you can select a number of repositories indicated in blue here that you're you want to test some version of and then you saying what packages they affect or might affect and you want to test against them which are the green double circled ones and the job will Then take that and expand it to say well what is it all the packages that are between the double circle green ones and the blue underlay ones

and pull them from source because they're going to have to be built because they're building on top of the chain is two blue ones and then it will backfill and say what are all the dependencies of all the source packages the blue and green and fill in the rest of the packages from the kind of Debian packages and this Allows you to test across multiple repositories and against higher level packages so for example if you're running a release of raw CPP on a specific branch that would go into the blue but then I want to

test to make sure that RV is still works I would declare that and then it was fill in all dependencies between raw PPP and arvid's from source using the default version upstream and then it will pull in all the dependencies of Those raw PPP all the intermediate dependencies as well as rviz and follows from binary and then running so just as a quick overview of some of the things we support we support a bunch of debian we support amd64 i 386 arm 64 and arm hf we don't turn everything on by default it will takes

a lot of energy and resources on the built arm and it also ends up with more emails that people actually care about one of the things we work really hard if did not have false Positive failures that means people stopped paying attention to alerts and that's quite bad we're also who we also have beginning support for our PMS but we're not actually building anything yet we love your support Ross and dojo have supported us on running the build farm and as I mentioned all of this is open source and deployable if you have different requirements

proprietary repositories that you want to run continuations testing on you can do that If you see the wicked are rapid or - build farm or the Ross build power repository on you have their instructions how to do that boss and clear path as well and testing and deploying things there's also a link here to Bosh's previous presentation at Ross Khan from two years ago if you're want to actually make something happen very shortly you can go if your packages are in the Ross build farm you can have a dock Entry to make sure it generates

the API Docs the source entry will turn on develop jobs by default you can also turn on pull request building the link provided earlier and then of course when you release packages they will go through the Debian build pipeline which I have to admit for timing and if you do get emails from Jenkins pay attention and if not ask if you don't know what to do you ask questions we can help you make sure it doesn't keep emailing you And I'd be starting or an all email here's a couple of troubleshooting tips that are put

together I'm not going to go over in detail for timing reasons and thank you if you want more information that's the link to the github repository which is the main entry point for reproducing these builds running them locally or applying your own builds are cool well thank you telly I think that was really helpful and informative and I can just kind of reiterate how useful This stuff is we use it in Ross industrial and kind of the level of support we get from the community when when things don't work is has been really nice so

appreciate that but I'd like to open up to questions so if you have a question you can either choose them mute yourself or type it in the chat and we'll answer it there so I tell me I'll ask you a question we also use Travis name we're off industrial but we use their own script so how how many People use your build scripts button Travis do you know or do you have an idea I don't really know the Roth industrial is the primary one I know of using Travis at this point also the boovie community

and I think there's a non-trivial overlap on that this developer base okay like get a couple of questions can you see him telly yeah Casilla publicly accessible repositories the public if you're using if you're using a deployed version of the raw Spill farm on a private version you don't need a public Ross district fork you may need to have specialties that are given to the given through the build farm I believe we now support the HTTP token-based access coverage question about coverage information from the in tests or off test arch CI integration does not currently

capture any coverage information what we love to do is have another extension to do coverage and Collect coverage and aggregate it over time I would actually suggest integrating that as a separate job that then will report separately and not to cross the unit test with the coverage but definitely we had a Everett engine that could be integrated helliya if somebody wanted to contribute something like that would they should they reach out to you first their order core somebody at OSF if you're interested in adding that I'd suggest Starting either ticket to discuss it or mailing

with something on the mailing list this course probably but we definitely would love to have that we we spent a lot of time on the build from getting to the current state and reworking and making it available for everybody and I've had to switch to other development priorities oh cool some of the already had the confirmation that from Philip that yeah you don't need to have a public roughness or for All right I think is next Thank You Tully I appreciate you waking up early with so next up is I think say Oh No thank

you hello everyone my name is Isaac Saito I'm a co-founder of a fork nonprofit consultancy in Japan and I'm also contributed into industrial laughing destroy projects these days so today I'm going to talk about industrial CI package it's actually a Roth package so you just hear these are heard Hollywood explains about the robbery of a bill torrent feature about the continuous integration so I'm trying to I haven't although I am prepared my material as to specifically to clarify the difference between the feature that was the wealth filled firm provides I'm going to try as much

as possible so just a quick recap of why we are why the constraints integration is useful so just there's a lot of concerns during the development so long which is like For example you want to say if I want to keep the system stable with your change your writing you don't want to break your system with a change and also make sure that the ops training packages change in app streams of the break your system also a technical debt is something you really want to avoid you want to find you want to find slugs or

potential reaches as early as possible similarly instead of projects procrastinating and ending up with a More future work so there could be many more concerns depending on their roles but these two absolutely you really want to avoid for sure in order to make sure that you're clear of changes not breaking a system to because you can run the deep recess of killing your posture because you obviously you don't want to make not you want to ensure that the Mafia feel also you want to run the unit camp and also the system test suite on your

near comp There so you really encourage you to find a deep camp in the event in the conference also you want to check there is no issues a form of you some of deployment and knowledge installation running these steps manually apparently that it's going to be really tedious especially yeses you have to do these tasks on a clean environment to make sure that the dependencies is fully a fully met so that the your software is not relying on something that is only Available on your machine and also if your software is supposed to run on

multiple multiple platforms like various operating systems for example in India Yahoo's check to check all the all the possible combinations so really it's going to be really I'm taking so there's absolutely no doubt that via automating you templates is so beneficial and that's where continuous integration comprehend so continuations can be can be useful in basically in Every development environment but it becomes particularly crucial if you're using open source software because open source software is is subject to change 24/7 throughout the year on Christmas Day or New Year's Eve or on the international point three-day or

not anything in any day so non-stop so you never know and and you never know what change in the upstream could break your system so it's just simply a good practice like Sean Just mentioned in the very beginning that is if you're not integrating the continuous in situation and along with the other methode you're not really sorry I forgot what what what exactly says you're not really doing the development like that so there are several software options there a server option software as an option Oh an umbrella cloud-based service options a firmer software on the

host machine that you have Mac use you Have a full control you can run it whatever timing you want to and in customize however you want much of it so features provided and some sauce software like Jenkins is known for having an extensive set of features so that you can get much more tailored for your father needs obvious downsides involve running a service officer on the old machine is the Scout maintenance you going to maintain the CI server software and on more Kalinka Is the observer machine itself so the cloud-based options frees you from setting

up and maintaining the server and everything and you just start using continuation leniency it's really up to your choice whether you use server software or cloud service so looking at the last few years the service called travesty eyes which is run by a German company in Germany these be one of the most favorite options Among wrong development as far as my job so now we have software or service that we can Wesley Wesley useful but still we're robot is not like IG guys though and even with us very nicely developed software there's still learning curve

so the same time and take some motivation and most likely you are exposed to some some shell scripting and a lot of software actually provide their own grammar so understandably robotics engineer and two pencils with these taps Away and and to stay away from using it even though they fully understand important social situations here they come see the inaccuracy accompanying very handy the idea is to reuse consolation for continuous integration system amount among among other countries so that the developer can be free from maintaining continuous integration and let's and focus their own robotics work so

it's basically set up just a shelf fixed configuration and It can be run at which can be run against actually the rock McAdoo's massive rock industrial packages and it's originally a fourth of the package called JSK travis which is created by the JFK lab after University of Tokyo so travesty is currently intended as a primary target platform but it can also be extended to other CI transfer observers with minimal addition la indigos Jade kinetic are currently Supportive and and fully as briefly mentioned about the pre-release testing it is very nice feature I totally agree with

him especially when you're if you're if you're a package that is supposed to be a built supposed to be released under the Ross field form one minor problem I found with the previous testing is that you have to setup the soccer environment on your own on four machine which doesn't take that long but still could waste the go mental heart Turtle for for visual arts engineers so industrial CI enables you to one pre-release testing without setting up anything at all and I might get a industrial see I can run on club or the Mogul ha

and it can be also integrated with you a private repository I know some quite a few number of private repositories are used to wipe again from a minimum minimum example of the computations that you need to add on Your palm tree in order to use in the field vs lgd if less than ten lines of the file and it's because it's trying to see I targeted if the format is is also what is trying to see I have to find if you look at the third third lines on the bottom its keep cloning the Industrial

di code into your CI server and at the line of the very bottom you just run a specific script and that's where all the all the continuous integration prophecies that are defining the mystery Of di get started so this is a layer of the entire system while industrial T Awards video response three and CI server your response rate sits on the left bottom this is where your software is maintained on bottom right developers write code on their own computers and assume them send them to the repository of the full recurrence or zone recommends the direct

Commission and the repository sends a request to the Associated CI server server or service To run CI job which is at the top left and then CI server fetches industrial CI by D cloning as I explained in the and London service with calling me so that's it everything on your ant can be private as long as the CI server can act as the github repository where industrial C is maintained so there were some questions about the user base it's hard to tell how many users or makaan frames are there to stop uses industry FBI although

there are some few years that I can get So to all the metrics page on the github palm tree sale what it says longer 53 clones per day on average over is it January along with 100 week loan for them but these fears are honestly so Lewis are not really meaningful to me well at least I can tell there are there are some users not 0 and also using the github beast the search feature I searched the repository that launched a thousand material CI and then there are 142 results they're all duplicates I went over

them manually and influenced some of them and I will roughly say at I would say roughly more than 50 with top three athletes mentioning or using the fruit VI so there are other options CI options are obviously and a Roth bill farm family explained so I would say it's the biggest difference might be a it's the industrial see I can explain it can be used from the private repository so there were the questions discussions just happen right now though but Unfortunately I couldn't share that part clearly so someone may be able to appreciate someone could

clarify that later but the full moral I understand industrial CI can be usable and a private Widow whereas while field forum needs to be you need to register your altering information public place on there yeah lock okay some other resources is available in the lost comm and five and I'd like To thank all the all the users who are getting feedback and also particularly like to thank Matthias from from from Curitiba who : carry coal making very natural CI and and lastly industrial product who gives me an opportunity to present and legislative package but any

questions that sounds great I think I appreciate it well we'll see if any questions come in you know I did want to talk a little bit about him in the beginning when when you started this one Of the main reasons that I was interested is that it it integrated the travel which integrated with github full request now that you know that oh it's RF support them as well it's a little bit less important but I will say that I use industrial CI across my public and private repos and and I do it even though they're

not formal Ross packages and so I think that's one benefit of an industrial CI is that you basically Enable Travis and and check in one script and you're off and running with continuous integration oh it looks like there's a couple questions for you so so well Trent was asking about let's see when you deploy capability and locally hosted get every so so he is basically saying if you have to host everything locally and he's using get lab can can use it and it looks like Matthias is saying that get lab support is time to I

would I would also add trends that you Can actually deploy both Travis and github locally you have to pay for them at that point but those are things you can do too and then a lot of the scripts that both Holly and Isaac presented our can be can be used privately I guess we do this at swery or well when I was at tufflips research we did a version about the lawsuit farm so there are other differences so first of all I really recommend everyone especially as I said if they're planning to release the Package

through the pill farms they we would encourage to use them off the part that comes to the service the reason their staff Industrial see I feel valuable is other than the private process public issues it's also customizable you can customize the pre profits post profit on even you can even have your custom ideas into the main script and also be very particularly hosting your quad 3 on the github as far as I understand correctly Our github has a one hour time limit for for each built which in and and Rossdale farm tends to take longer

in my experience because in because it is prioritized equity over its peak so that's what's actually over the problem with the mugak product so because they it feels it feels takes forever and you still want to use a CI so we ended up having their own industrial I don't know their arms or CI project so but still I'll really like to it will be a co a Free consult if you can control it into service so that's the yeah we get the better support though we will still look for the opportunity to how we can

come watch how we can contribute to the wrong compartment what we've done awesome alright thanks guys thanks so much we're going to we're going to move on to actually advil ater but uh is so iam for PJs read from Southwest Research Eddie I'll turn it over to you thanks John Hi I'm ed Bennett or I'm from Southwest Research Institute PJ prepared these slides Phillip Reed he's one of my co-workers and he's actually in chat today but he is home sick with the flu and doesn't have a voice so I'm going to be doing my best

PJ impression the work with you is with autonomous vehicles so full-size automotive scale robots and I'm going to talk a little bit about what we do for testing CI and deployment I hopefully I Can fill some of the gaps on deployment that we missed out on from resync today so quick overview I'm going talk about some conduce integration which totally covered wonderfully and Issac followed up with a lot of information but I want to just touch on how we use it testing I'm going to talk a little bit about and I'm going to spend some

time on deployment we've tried several different methods for deploying code to vehicles from sort of pure source deployments all The way up to our own build forms if I'm gonna spend some time on that this is contact information for PJ Reid who developed all these slides and I would call our resident expert on CI and deployment systems I think mentioned how roboticists don't like CI PJ is not a roboticist he'll probably kill me for saying that but he came out of software development and he looked at our team and was horrified and whipped us into

shape he's still looking at initiate and Any questions he's in chat right now today and you can feel free to contact him my contact information is not up here but I can post it in chat at the end of the presentation if anyone wants to get in touch with me so continuous integration there's a few options here one of them is in vessel CI which Issac just talked a lot about another one get labs CI which actually came up as a question in the last presentation we use gitlab internally For the same reason that Trent

asked about it our corporate overlords do not believe that github is a secure place to store anything so we post everything on completely internal get back servers gitlab is a product like github or bitbucket it's a git hosting service except you can host it entirely locally and we find that it's a little bit better than the enterprise github but it also comes with its own CI solutions the CI solution That comes with git lab is very similar to Travis both of them are very lightweight and or the configuration in the repository and we find that

as a result they're a lot easier to configure and interact with than something like say Jenkins or teamcity of course the gold standard which totally talked about earlier in Co SRS build form and if you're releasing packages publicly then you know you can't get around using the OSR so farm and you shouldn't it's the Best in the authoritative way to text and release your gear software one of the things that he touched on there is that it generates documentation and this is one of the best features of it if you have oxygenated code and if

you have any Ross messages in your package all the documentation on Ross org in the API documentation that's coming off a double form so it's sort of a free value ad if you want to use the OSR a closed form and we do have some 30 packages in our Group that we release publicly on github and build publicly on the OSR EXO form so I'm going to jump into testing a little bit here we don't do as much automated unit testing as we should and some of that is because automated unit tests don't work really

well for a lot of the things we run into unless we could set up a full hardware in the loop testing lab a lot of the stuff that concerns us we can't do automated we haven't figured out how I know there are Some really smart guys that companies like sex that are doing it but our approach to this is manual testing is still infinitely better than no testing at all so we run as many automated tests we can but we also expect that for significant changes we'll test things manually another option would be testing a

simulated environment like you see both and we have we work with some clients who do simulated testing in another simulation environment it's Incredibly valuable for them because they don't have access to the actual hardware we're working on but you do run into some problems with simulated testing the big one is there's a lot of overhead and setting up the test environment for specifically for outdoor robots which is what we do setting up the test environment where all you care about is I had an indoor environment and a robot on a cleaner surface is a lot

less overhead been creating an entire Gazebo world for a test track or an outdoor test environment so we found that getting that kind of environment set up for the kind of complex environments we work with is very difficult but for a lot of people working in robotics it's a really great solution so that you don't have to deal with hardware you don't have to deal with potentials breaking things or safety problems and also you can run vivo in in an automated fashion but we Don't have a lot of experience with it so a lot of

our testing because of the problems of safety and being able to test things repeatedly being able to test things without risking hardware or people is back house we have a lot of bad files that we've generated over the past few years of doing Ross development and if you have a bad file then you have sort of the authority on what your robot is seeing and doing so if we find that it's really useful for debugging Behaviors that haven't changed of course you can't forward simulate the changes that you make but it's really good for simulating

and testing perception so if you can set up all of your your launch files to be able to run headless where they don't run any of your sensor drivers and just run a bag file in their place and you create bag reporting scripts that store raw sensor data then it's really easy to create a test Line where you go out you collect a whole bunch of data and then you can spend the next few weeks developing off with that data where you're running the actual vehicle software with the actual vehicle data but you just sort

of drop the bag and as a as a mock sensor it's a quick plug for one of the things we've developed because of all of our our bag testing PJ spent some time the last year to developing a web-based tag database server the basic concept of the bag Database is if you have a lot of tags you need a place to put them and you need a way to find ones that match what you need for what you're developing on so for example we have hundreds of hundreds of bags and sometimes you need all the

bags that contain images because you want to train a classification algorithm so you can go through and grasp all the bags that contain images by searching by topic or for example maybe you want to train a specific Classification algorithm if you're working on something to attend Xtreme if you tagged those bags in the database server you could search by those tags so if you could collect a whole bunch of bags containing pedestrians or no pedestrian then you can use those bags and input to your training algorithms or your testing that's just a quick plug for

our bag database now the thing I want to spend the most time on here is deployment portly because the guys from Rethink couldn't come today probably because I feel like totally and Isaac really covered the CI part of side of things really well but we do have a lot of experience with deployment not as experienced as clear path to rethink but we have to offer some background the work v2 we develop a lot of almost exclusively we develop software for clients so the client comes to us with problem we work on problem we delivering software

and so it's not a deployment Problem of the scale that a clear path is working on or the Tully's working on where they have hundreds of customers or a community of thousands but it's a situation where we do need to deliver polish software that our clients can run without having up standing there telling them how to do it so we've gone through a sort of evolution on this process we started out with source on vehicle and this is probably what 90% of Ross researchers are doing you check out all Your code on your vehicle you

build it on your vehicle and then you run it on your vehicle there's a few things about this that make it attractive and a lot of things about it that make it bad so it's very simple to set up which is why a lot of people if you're doing a research project if you're in grad school and you just need to make it work this is what you go to you're guaranteed that your code is going to run on your video because it was built on the same Architecture you don't have to worry about that

but it's really difficult to coordinate once you have a team of more than one cowboy coating on the table becomes a real problem and knowing what you ran on the vehicle becomes a real problem so it's easy to lose track of changes when we were using this method and we used it for several years it was common practice for someone to come in notice that there was uncommitted code on the vehicle and then not know what to Do so the fact that it's really fast sort of gift swamped by the fact that you could lose

changes and that sometimes you're testing against code that you're not 100% sure it's the code you put there and once you completely wipe the vehicle and you come in every day it's a really bad way to develop so this is what we started out using the diagram on the slide that you have multiple people deploying to multiple vehicles multiple computers on the Vehicles it can get really ugly real fast so okay deploying source on vehicles that from there we moved on to a sort of hybrid solution where we didn't want to set up a build

form because those forms are scary and hard so instead we came up with actually I came up with a script where you build everything on your developer machines using catkin install it to an install space on your developer machine and then sync that Over to the vehicle this is a little bit more work to setup but it still allows you to very quickly iterate because you don't have to wait for a build form to build something build deafies install the Debian etc etc so the main advantage of this is very easy to keep track of

the changes the code that's running on the vehicle is always the code you just built the Co you just built it's always on your developer machines and so it makes it a Lot easier to coordinate between multiple people because every time someone goes to run codes on the machine they've removed all the other codes that other people have run disadvantage is well exactly the same thing if you have multiple people developing at the same time on the same vehicle they're not going to be able to easily share their code between each other without coordinating externally

like committing it to github and pulling and marking so It does slow you down a little bit but we've found that it makes it much easier to determine the truth of the code on the people and so it's a better method one other advantage is if your vehicle computers or storage limited you are only putting your installed artifacts on the vehicle so that means binaries launch files configuration you're not putting all your source code so actually I had to use this method on some of our projects because the vehicle I was Installing to didn't have

space for all our source code so deploying from source still better than building on target but still not as good as actually building software and distributing binaries if you're going to be distributing your software to applying you don't want to have to make them build the software themselves because then they have to maintain their own build environment and most of your clients probably don't want to do that next step up a single Debian Package you take all your software put it on a build server or just build it on your own machine and put it

into a single Debian it's a little bit of work to setup and depending on the size of your workspace that can take a long time to produce but it's really easy to track release versions it's really easy to keep a history of release versions and if you have a lot of repositories that are going into your release then you've got all them synchronized at one point In time so one of the projects I work on has about 188 packages spread across a couple dozen repositories keeping all those repositories in sync when you're deploying them to

a vehicle is a real short because you have to tag all of them sync all the tags make sure you check out all the right tags on the vehicle it's the math so having a single Debian that just installed exactly what you want is a necessity in some cases so this was our Interim step before we took the plunge so we are now running a full OS car filled farm locally at query that builds our code from our project gate lab servers generates Debian's in the same manner that the OS RS builds formed us which

is to say we have one for every package it's a complex setup but because of the work that telly and the guys at OS RF have done it's definitely doable so pga spent about a week setting up our build form we have one virtual machine That's acting as the head of the build farm that runs the web interface it's a Jenkins interface I think most people either probably familiar with it from the OS RF farm and we have two or three build slaves that are running on just beefy computers around the lab like I mentioned

we're running it entirely internally our Ross distro is hosted on get lab hour all of our code is hosted on kit lab and we use the same bloom process that is used for closely Releasing there's a the only thing is at the end where it makes a pull request on github you don't want to do that there's no way to automatically make those pull requests to get labs so you've got to do it manually other than that it's entirely the same as releasing to the OSR farm and you get the same output so you get

Debian's source debian for every package you can configure it for what architectures you care to build for you get documentation and so if you're Looking to create actual build products that you actually can distribute it's a really great option and we actually we've gotten to the point now where we can go to a vehicle with a fresher bun to install at our own again private apt repository and install our full software stack from Debian's using apt-get which is huge for us so it is a little bit of work but it makes it a lot smoother

to release and the other advantage here is you're in your own dog food so the stuff You're testing and you're working with is the stuff that you would be delivered to a client so it's a really good way to make sure that oh you know I actually did install everything and when I give them a Debian they're going to get the same thing that I've been testing with as opposed to my desk code which runs differently because I built it on my laptop or my dad packages which are different because they don't have everything installed

in the installed Version so I I kind of flew through that this is my last slide I thought I got the assigned by the seat of my pants and to open it up to questions and if anyone has any questions I'll try to answer them if not TJ's in chat and he can also answer any questions thanks ed that was that was really good I think that they covered a big hole in deployment and I certainly learned stuff that I didn't know I'm gonna I'm going to switch over presentations but uh I know this was

Important to some of the attendees so unless you explained it perfectly I hope they have some questions and if anyone has specific questions about what we had to do to set up our own build farm I'm not the expert on that but I'm sure PJ would be happy to answer those either online now or an offline discussion it looks like there are a couple of oh I've got a few questions here so I've got a question from Adam how did you ensure that package is built on your local Michigan correctly ran on the deployed system

the answer is we did uh-huh basically it was easily pretty obvious to us it's packages built on the local machine didn't run on the deployed system we were always running the same architecture between the deployment issue the build machine and the deployed machine which is key to that if you were running say armed on your your actual robot and had an i7 on your computer you have to set up tool change Which sounds like more trouble than I want to go through so we didn't have any particular method for for making sure that things were

running the same but we had pretty good faith because we were using the same architecture the same distribution between the tail question from Isaac about bag database asking you for any funds to run a cloud service for it we do not have plans at this time to run a cloud service for it but it is open Source on github and you can write with docker it's pretty easy to set up so and it's BSD so if anyone else would like to run a cloud service for it be our guest I believe there is also a

company that is running a similar cloud service for bag posting bed and searching but I don't know anything about them I think they give a presentation about calmness here Jeremy asks if you were trying doctor in your deployment it's true can you talk a Little about that so we do not really use docker in deployments on vehicles we use soccer for build environments so the RS reform uses docker extensively for consistent build environments as does geek FCI we have played with it a little for deployments on vehicle pj is done it more than i have

my experience is that docker is really really good at running a service and emily exposing it and ross was not designed for middle minimally exposed services so unless you run your Entire ross application every node inside the same docker container setting up your network configuration is either a bit of a hassle or practically defeats the purpose of dr pj just made a comment on the back database cloud service he said right now there's no authentication mechanism for sag database because it was designed to be running inside our network and before it could be used as

a cloud service that would be a feature that so you know for wordpress welcome All right I I really appreciate that ed and PJ PJ I hope yeah you get better but these guys really they really done an amazing job I've seen it work and they're they're a great resource so please people feel free to reach out I'm going to move on to our first what we're calling lightning lightning talks or discussions and these are just quick five minute presentations a couple people came to me and just wanted to put some stuff out there for

Discussion and Jeremy Adams from Intel agreat 'add in particular was interested about a mocking and Ross unit testing so I I'll pass it off to you so really you kind of pull the stole my thunder a little bit because that's almost entirely what my question is the opening slide but we'll go through these so my problem what's that I said my apologies that's okay it's a lightning talk right so my assumption is the most developers or writing and by developers I mean ones that were commonly pulling Debian packages from so those using the Ross build

server and those outputs and what my assumption is that most of those are writing unit tests and/or Ross tests but it looks like from my inspection that they're not actually covering much of the code and since we don't have a coverage tool it's likely we we don't really have the full understanding but and then other developers who are using this these code bases they may not be Familiar with unit testing or and or Ross testing or they may not just use it at all and and try to avoid the topic so my my kind of

question is why would you mock and in the Ross answers are right sorry Ross wiki it talks about using a library and a real library structure where you kind of make your library code agnostic to Ross and then you unit test that and incorporate that into the rest of Ross so I want to isolate the Ross code But I may have a I may have a package that's already been written and I don't relent it really want to refactor it so what I'm looking for is you know why where would it but we're going to

write mocks where would we write them where would we store them maybe we create them in the test directories for the Ross framework or the base packages or maybe we incorporate those into Debian source files I also noticed that in Ross answers there's been some Discussion about using Google mock and the notations there are that Google mock is not does not work well I'm not really understanding why but not working well with indigo and that now in kinetic I guess especially since it's on 1604 Ubuntu that it works better so that's where I'm kind of

leaving it and wanting some feedback so Jeremy if if I understand luck correctly you you would create mocks for all the Ross API calls and that would allow you To very efficiently run instead of running lost tests where you bring up a whole Ross system you can run things that look like unit tests which are lightweight and fast but you it would require mocking out all the Ross API is that is that what's really required well I think in my understanding especially with Google mock you can write a hybrid a hybrid mock class and I

don't know I haven't read too much into that but I think it's through inheritance and or Templating so the answer your question I don't think you have to mock out the entire Ross you know like every class that's written in the for Ross but I think just what you're using now the question could be the person writing the production code right the mock or does the person you know is it something that would be conveniently supplied by the developer of the API your calling and my I think it's a little bit of both so is

that kind of Answer your question about my question so it an else are you are you aware of anybody who's doing this like any rocks packages that are using Google mock in general not necessarily just you know for the Ross API it seems like from that last slide that if you like it I won't click on it here but if you look at that question that some people have actually started using mocks and especially in kinetic and I think that's the only used I've seen I checked out like probably a Half a dozen or so

different Ross github repos and examine the unit test and I didn't see any any use of G mock and C make list text or in the code itself well maybe maybe somebody out in the audience who's more familiar with Google mock and unit testing can chime in because I I can tell you why nothing exists it's because there are no tutorials and no examples of how to do it because as Isaac pointed out most robot assists are not not a you know not Software developers necessarily so if if anybody wants to chime in and has

some expertise here that'd be appreciated John a bit honest le bit on the compatibility and the issues of deploying team up is that G mock includes a version of G test internally which conflicts with the default version of G test we're deploying which is the problem with deploying it kinetic I'm not sure exactly I certainly know that from indigo It sounds like people have found workarounds for kinetics I believe there's still a inherent issue that G Mach is packaging Q test internally and there's some C make hoops that you have to jump through detect if

key Mach is installed use the G lock G test instead of the default G test so to your point about visibility or best practices I know that on the Roth's tutorial are sorry Ross wiki there are a handful of pages discussing you know writing good Unit tests and why you should do it but no real mention of mocking and you know as totally just stated there's that's probably a good reason why because of this conflict but going forward it seems like some people are starting to solve that problem and I'm wondering if some tutorials and

you know pushing that kind of best practices on the on the wiki would be helpful certainly I think that'd be good Python mocking some going on in the Python unit testing for quite a while until G mark came along there wasn't a good standard C++ one but now that it's hopefully available it's good practice to use one of the big difference between mocking is that it allows you to do integration tests that only have the dependencies of unit tests led do you I mean do you see the value in doing that like is it are

they lighter weights are they faster you know do you see that value yeah a mock test lets you isolate The you say okay I expect the behavior of the rest of system to this test only this system greatly decreases your the kind of C's on the external systems when you want to run it so you can actually run it up the unit test and say I would expect to receive six messages in this order in the sequence and the mock will provide that and then you you could test your unit on the unit under test

is much smaller and you don't get you don't get an issue that like if there's a bug in Ross you pee in it something happens over there it doesn't affect your test okay I was just thinking that the isolation that you you mentioned is is all the reason for both sides of the equation people writing api's and people using the api's to start trying to develop frameworks I guess the real question here is is how to construct it and get around this this conflict between G test and G Mach and also where to store the

Mach headers When we're done what's the best practice there yeah so Jeremy I I do think that the build is going to be that's going to be hard right you get we're going to have to work around it but I will say just from experience once you create a tutorial it's amazing like how I can kind of jump start or kickstart the the development that that you want or need and at the very least it kind of transfers the knowledge to two people and like I said since we're not software Developers when it comes to

testing that a knowledge transfer is kind of critical so yeah any any other last items before we move on no Florian's talk I don't have any I just look to the discourse to continue the conversation and I'll be sending that out probably within the day with the recording of a web meeting all right thanks so much Jeremy I really appreciate all the work you've you put in and behind the scenes and and then Your presentation I appreciate it no problem all right Orion last but certainly not least okay thanks Sean can you hear me yep

are you great okay very good so another short lightning talk about yeah the automated test framework we've developed so yeah as you all know there is a wide range of this boot of soft sediments going on in the rest community so that means that information about a Roth package is spread in a row squeaky In source code system but although in extra knowledge direct contact mailing list or wonders and there is already some tools in Ross for measuring code quality so first of all of course it was at this farm which does a good job

on compiling and yeah running unit tests but the unit tests mainly depends on the package maintainer so there's four sugar packages which have a good coverage but there is also a lot of packages which Have a very poor test coverage there's also things like the rock induction rating system which is based on the wiki but that other at the moment a manual effort with labeling that there is all the tools like github and rot index which gives you an hint of how active a package is still maintained or developed but there is no objective measure

for the runtime behavior on a high level point of view the problem a developer focuses is how can I select a component Which fits best to the other component I'm using to my robot to my environment environment to my infrastructure so that you to the reason that you can young have the same rough package with a slightly different configuration doing something completely different or having multiple packages offering same functionality like different drivers for the same hardware piece of hardware or different paths panel for like moving the mobile base somewhere and if you sum Up all

this and have just a small relation you easily end up in more dissolves or even tens of thousands of test cases which is of course impossible to to test and off to compare afterwards so we feel there's a need for forty to measure on the runtime behavior which has some young objective testing which means if I am in the place of selecting a path banner for my mobile robots I'd like to have some assistance to choose which path banner is the best for my Application or which configuration set fit that to my my robot in

my application so what we have developed is automatic testing benchmarking framework which generates performance indicators using common metrics on the API level what does this framework offers of a conditional equation service can either run on a champion school farm or on Travis and especially can be run as a hardware in the loop or simulation in the loop of test system it already Offers building blocks for gathering data on system level like CPU RAM usage of the node or of a deploy across work with rock network out of multiple nodes all the things like update frequency

or publishing frequency of the topic and all those TF stacking or frames and so on the third thing it offers is offers like a benchmarking tool set so if you have widely used throughout the face like move based interface or move proof or move it there can be a metric which Already ever lates like execution time accuracy path lengths and so on so you can easily compare different new space instances with each other like combination of past and localization and different configuration that's all composed out of the yo-oh based on our GIM server and then

on top of that you either connect to a real Hardware you connect to a simulation if that's available if both is not available then you can still work on Recorded data as we heard in the previous talk about the weasel and we might even need to add another blog here for mock up components so if you have neither Hardware no simulation or recorded data so you might just want to mock some wrong interfaces to continue these tests of the application and then on top of that there's a web-based aeration analyzer tool which lets you compare

the different test cases would be the other so that sells a very Short example how that comparison could look like so you test different combinations of robot specific environment specific and application specific configuration and then it's easy to charge our two to the legs the component which fits best to that specific combination so as a summary the goal was to enhance the transparency of the component quality the distributors of the first developments which takes the app which make use of metrics which Can be get at some runtime performance of a diploid raw system which helps

the user to little X a component or a component configuration set for the application so resolution is an automatic test framework which is available open source so you are welcome to try that out we're currently hosting a workshop in Stuttgart we will guide into that tomorrow and see how we can use that and applied to different use cases thank you thanks foreign but that Is a I realize there's a lot there that you packed in the five minutes we'll see if anybody has any questions but I assume you you've done you use you use this

on some combination of hardware and simulation at a PI's is that true okay so that the positive ovals to only test on a CI server but also to include like a really deployed system like a robot with multiple machines involved and really the deployed configuration so you can easily evaluate if like network of CPU load is spread in a good way on your system or which kind of combination makes your path planner tonight better path and let the robot arrive earlier ethical okay by the way I just I just looked up your your github site

and it has a lot of really good documentation for for helping deploy this there are a lot of pieces to this so that's that's really good I appreciate that if you do you intend on doing more development or is This kind of a done thing and now you're just going to use it as other at the moment is the I would call it a proof of concept implementation so it helped being used in a variety of use cases but it's not spread out to the to a wider audience yet but if that's if you figure

out that useful then we'll also continue development for that so yes if you had there in kind of for a couple of sentences say what's the main value you get from using ATF above and Beyond some of the other tools people have developed I guess can you can you get a summarize I true and on my level so the main purpose of the ATF as it is emitted now is to get metrics metadata from metrics which describes the behavior of the application and application meaning a deployed system on on real target combining different rough notes

so the same system as you would deploy to a customer we can tire a need test there and compare different Configuration set I like like high shares a comment you see that in most business yes but you have to possibly to specify robots specific configuration sets and Raman specific and applications specific environment sets and this would allow you to be obsessed same robot in different environments and check how the path men are can deal with different maps different for example yeah so kite was one of the first mercury users of that oh well PJ in

Edie this is maybe something you want to look at for for vehicle testing in Florian do you think it in like it would be useful for that oh I've already started to get a repository okay yeah it provides all you need to reply back files and then specify your test cases and get the metrics out of it but I'm a positive that you can apply that to raise yes all right well I appreciate Florian I appreciate you putting that together and I know the Time is short so I will encourage anybody who has more questions

to follow up with you offline and yeah so Florian was with our last presenter so we're kind of at the end I don't have much to add I'm not an expert in the space so I certainly appreciate everybody else climbing their expertise I will send out a mail to every all the participants with the recording and we'll continue this discussion on discourse and yeah just thanks everybody For participating and mental that'll be it for now thanks for your time