Wolt Tech Talks Podcast #10: Scaling Kotlin Backend Development

hello and welcome to W Tech talks podcast my name is Yu and today I'm joined with Andre nsky who is our cotlin competence lead today we will be talking about cotlin backend development at Walt and for those who have been listening our podcast before maybe you remember that the where first episode was named adopting goland that was recorded around two years ago this will be like a nice continu for that things have evolved quite a lot during the past couple of years today we will be talking about what kind of cotlin backend Services we have

at wal nowadays how's the developer experience especially what kind of intern internal tooling we have built uh both to increase the velocity and the quality of the back hands we built with goland and who would be a better person to talk this things through than Andre so Andre welcome to the show could you start by introducing yourself hello hello y I'm Andre the competence lead of C here at wal nice to meet you all great you mentioned competence lead role what does it consist of I guess it's not a industry standard role so to say

yeah definitely not a typical role you'd see in every company um so it consists of um a hiring aspect like for example uh one of the things we've been doing with Y by the way is um introducing a new backend assignment for our Engineers for our candidates um that's one of the aspects another aspect is knowledge sharing both internal and external and apart from that also of course helping the engineering community within the company by answering their questions or sharing uh best practices and stuff like that yeah I'm sure we will talk about the best

practices that you have kind of cified into tooling so that they are taken into use a a bit later on what's your own background in software engineering how how did you end up writing cotlin backend so I've been working in it for the past 14 years um initially started as a admin but then realized that I enjoy writing software more um been writing most backend initially in Java and then around 5 years ago I think maybe it would be six years already um started using cotlin and since then I've been writing cotlin exclusively so yeah

uh that's how it happens I'm sure we will get back to the Java versus Scotland down the line I think that's a one interesting aspect to cover but maybe let's proceed to the so to say basics of cotlin at w so what kind of scale are we talking about how many how many people are actively developing gin and how many backend Services we have at the moment so according to the stats we have in GitHub uh we have um more than 100 Services deployed and we have around 150 monthly contributors to our cing projects and

we also have uh around 30 of those are building our underap so yeah and we also have a cutl knowledge Channel where we share all our knowledge and there we have around 260 people I think at the moment yep so more than 150 cotlin all together like people developing goland out of w around 30 f focusing on Android and probably all the rest are backend focused and like mentioned in this podcast we will dive into the backend aspect of golin uh in which part of the organization golin is being used in the back end and

excellent if you can give some concrete examples of what kind of services it's its domain is build in gotland with yeah um I would say at this point coton is used all around bolt in all its parts like initially it started all in fintech and at the moment in fintech we have um cotton used in invoicing Service uh tax reporting integration with this tax authorities Payment Processing like all the inex stuff apart from that we also have cotton used in our retail platforms basically in Vault Market Inventory management uh item catalog pricing uh terms of

uh Merchant things we have Merchant on boarding back end written in cotlin we also have uh it's used in imaging Tech basically asset repository is in cotlin and asset service is also in cotlin uh our ODS are also seed by cotton services and it's get more and more used in consumer backends and quer domain so yeah all around world yeah good say so and in addition to cotlin maybe wor mentioning that the biggest backend languages that we have are Python and Scala Scala mainly used in the career domain and while python especially in the consumer

facing backends but cotlin is the one that has grown fastest during the recent years and I believe it starts to be on bar with by which used to be the biggest one considering backends so one could say that we have been ping on cotland during the past couple of years uh what would you say are the kind of selling points of cotlin for example compared to these alternative backend languages that we have had in use I guess one of the main reasons to start using cin was it step safety um it's probably it's biggest advantage

in comparison to python another reason was uh performance PN being an interpreted language has its heat on performance and in comparison to Scala ideas it was is ofuse and it's Java interrupt um yeah and also basically Talent availability because uh Engineers with Java background usually can adopt cing quite easily so it's a bit easier to find talents and also I guess especially recently some of the previously only or python focused developers who didn't even have any jvm background have been recently moving or started at developing golin in their teams and well I'm sure our internal

tooling is one of the reasons or one of the enablers for this I think we will touch that later on but maybe that's something about the like easy to adopt sorry apart from that um yeah there are some nice gu on how to just start with cotton there are cuton cons which is a set of practical examples to go through and get familiar with cutting there're also nice guides like uh cotlin for python Engineers so yeah I guess it all makes it easier for people to adopt cotlin what what would you say why not chva

we have cotlin and Scala well very well presented in our stack but why not Java kind of feels like if uh the company didn't have any prior Java development then there is no point in going with Java because uh cutl is Superior in many ways in terms of synx sugar and working these um multi trading is me more convenient in cotland thanks to Corin so yeah it's just more pleasant to use than Java I would say yeah I have read from somewhere that the better Java some some describe it that way I don't I don't

know if it's true or not okay let's dive a bit deeper could you share a bit details of the cotl stack we have so typically what kind of backend applications we have what are the kind of database aspects Communication web Frameworks and and the like yeah so in our case I guess we have pretty typical backend services which are in most cases are simply API to put some data into the database or R through the from the database so in terms of uh rest API or rather Frameworks we typically use spring framework or cater uh

some teams have experimented with other Frameworks but there are not so many of those um we also use corins quite heavily where we use Gator because that makes lots of sense we use Gradle as build tool and um some teams have adopted error TT for a more functional approach to programming in cing uh in terms of databases we mostly use pogress um and and to work with pogress we typically use Juke which is a very nice um sort of declarative library to work with a database not an RM like jpa or something like that and

to work with Monga we use official C driver um apart from that for API we also rely on JPC we use the um official JPC server provided by Google and for messaging there we use protuff and we also use cka quite heavily in our systems yep so I I guess it's very similar for all the backend language Lang langues so usually the consumer facing endpoints are some uh R Jon apis while grpc Proto Buffs are the preferred for service to service communication and or synchronous service to service communication and then C for where if you

don't need the response immediately or you're just informing that something happened for other services yeah how about the continuous integration and deployment side could you share some insights sure yep so uh we host all of our code on GitHub so to run our pipelines we use GitHub actions and we deploy everything to our case clusters in AWS um apart from that we also use our sort of custom made abstraction on top of terraform to configure our infrastructure uh and we typically try to have separate deployments for services uh having multiple performance profiles so for example

if you have a Kafka consumer in service and an HTTP API particularly create two separate deployments for that um yeah and of course it's our Engineers responsibility to configure infrastructure and maintain it so yeah full responsibility there yeah I believe it might be different in some other companies but yeah the uh we we have the infra themes but those are more as like supportive functions while usually the uh product teams are able to own their stuff from start to the long journey during the life cycle of the deployed application right you mentioned Gates maybe wor

mentioning that we're talking about kuer NES oh yeah just to be clear for all the listeners yes so that's the typical stack and the cicd kind of things we we discussed that botland is basically used uh in wide variety of domains in our organization so are there any differences between these domains considering how the stack looks like or is it that it's very similar in each s that the business problems are different based on which domain the back end is built for I'd say the stack is typically quite similar yet it's mostly that the business

programs are different because we all essentially solve the same issues right we want our service to be scalable reliable observable um so yeah I guess main differences are that for example some of our groups um prefer using spring framework more some other groups prefer using cater more but yeah there aren't any huge differences okay I think we could dive a bit deeper into the let's say stack and how it concretely looks like to develop codlin backends at w so I know for the past year or so you have been building this thing called cotlin starter

which is an internal project I know it has kind of revolutionized the developer experience of our codling back and Engineers so we could start by uh or if you could start by explaining the motivation behind this cotlin starter maybe that would be a good bridge to actually try to explain what it is sure so um before we had C Ser what we typically had to start a service um there were uh repository templates on diub there was one for let's say cator one for spring framework apart from that we had a few shared libraries like

a library for work stator providing authentication and some other more specific things uh there's our internal JPC library and some other uh internal libraries mostly group specific and um that provided us with some challenges uh so for example one of the issues with having a repository template is that you can't easily propagate changes to that template if you if you need to change some of the infrastructure there uh you'll basically have to create a PR to each and every service started from that repository right [Music] um so that was one of the issues another issue

is that without sort of an sdta like uh solution it's hard to propagate some best practices in terms of let's say I don't know consuming messages from kka or working with a database um and another was well just starting a new service some some of our Engineers like you have mentioned before are coming from python for example right so they might be unfamiliar with Gradle and well mastering Gradle could be challenging sometimes so that that set of challenges required a solution and coton star hopefully is the solution for that all right so uh starting kickstarting

new projects was mentioned SDK like thing was mentioned so could you try to explain on on High level what this cotlin starter is about what are the kind of essential core features of it so I would say that it's a sort of a SS arming knife for basically starting a new service uh that can provide you with all you need it's uh pretty simple to Serv with you just add a few lines of code to the gradal settings file select the features you want there for example maybe you want to have a uh rest API

and you want to have Ka consumer and the database then you select those features and it will automatically configure all the dependencies for you it will configure all the infrastructure and it will it can even generate an example code for you on how to use those features so yeah that's basically the way to to scale to scale services to scale best practices is it also usable in existing projects which are not kind of started from scratch with cotl starter in mind yeah it's uh given that it's a it's it's actually a set of Gradle plugins

and libraries with an umbrella top level plugin so if you want to start a new service you can use this umbrella plugin to configure everything for you if uh you want to use it in existing Services you can simply use either a library or one of the let's let's say smaller plugins where if we if you talk about the structures there uh let's say we have a database feature right that feature would have multiple artifacts one of the artifacts there would be the library itself that provides you these classes to work with the database and

given how often uh people were having dependency issues with bringing transitive dependencies I took a decision there to not bring any transitive dependencies with the libraries so the library itself doesn't bring anything transitively if you want to bring dependencies trans you can just apply a plugin for working with a database which will configure the dependencies for you and bring this library and make life a bit easier so thinking about the benefits the dependency management is clearly one kickstarting new Services is probably the biggest one and then adopting like uh Library kind of things to existing

services is so it's kind of a SDK in itself as well yeah and uh like one of the benefits there is that let's say if you started a service using C starter there basically just one dependency that you have to update so how we set it up is every week uh there is an automated dependency check in the coton star Repository which updates all the dependencies and then releases a new version of cotton starter so that all projects using the cotton starter can just update it and get all the recent versions of dependencies and apart

from that uh coton St also manages sort of packaging the service into a Docker image so it does that with jibb it's a Google plugin for packaging containers and with that we are able to even propagate uh jtk version updates to the underlying services or Downstream services so yeah it's quite cool could could you explain uh a bit more in detail like let's take the use case of kickstarting a new service what what is required from the user you talked about the CR settings for would it be that I include the features I want then

run CR build and then it spits out me like a full code repository with code examples for all the features that I would want to have well yeah there is a separate gr task uh called generate examples there okay yeah yeah yeah if you if you want to start a ser using example endpoints or example repositories you can just do that you can click generate examples and then just run grade build or Gradle run and it will start a working service for you locally um yeah one of the things it also does is manages containers

for you locally so if for example you need a Kafka container or database container it will also start everything up for you so you basically don't have to worry about anything need or regarding this uh autogenerated code examples if I understand this correctly is it like basically documentation on demand I want to have this feature please show me how it's done or how should I how should I use it is that the correct interpretation of this yeah exactly that was the idea behind it you know uh the problem I have is documentation in let's say

conf for other systems is that usually it gets outdated and it's kind of hard to keep it up to dat with the actual software package while this way uh actually testing the generated examples is part of the CI for C starter so it will never generate a non-working example for you which means that it's sort of documentation is always up to date with the actual software so yeah you always get up to date examples that you don't even have to copy from somewhere you can just generate them and run the service and everything works nice

yeah it hasn't been once or twice that I have faced a outdated documentation issue or let's say completely nonworking example in documentation issue so that sounds like a robust approach and then maybe a bit more about the SDK like things so the fle plugins what kind of features are supported there like what what kind of things uh people get out of the box when starting new Services um you get rest API layer like or HTTP API layer in ter in in form of Tater um you also get uh all the things you need to work

is a database be pogress or you get uh security stuff like uh our internal libraries for service to service authentication you get uh JPC uh you get Kafka Clans and Publishers and yeah apart from that I tried to put a heavy emphasis on observability so all all those modules all those plugins all those libraries are guaranteed to be instrumented and provide you this meaningful metrics so basically opt in to use a kfka consumer and you get data do metrics out of the box yeah exactly yeah yeah sounds super cool you you mentioned that Kor is

the one for the HTTP API layer um and previously we discussed that also spring is being used especially in some some teams or some product groups uh what this like a decisions decision of yours from the getg go to only support Kor would it be super tricky to support both how do you see it well with the current setup I would say it is technically possible to have Springs there as well but probably doesn't make much sense because spring overall offers a very similar functionality to what we have there in C starter although some things

are missing there for example spring wouldn't wouldn't be as straightforward I would say to use or as easy to include new features you would still have to tackle dependencies there in the Gradle Bild script and some other things and yeah it's just that um I myself actually like spring a lot I I gave some tech talks on Spring I contributed to Spring but uh working with Engineers here at w where people might come from python world or maybe some we we have quite I would say that quite uh quite a strong culture of internal mobility

within the company so uh people often can move from I know maybe working on frontend to work on backend and in those cases for people with no prior experience with spring I noticed that it could be quite uh quite hard for them to troubleshoot when something goes wrong and that was I think one of the main reasons to go for cater because it's way more simple and I've noticed that people have less troubles uh adopting it makes sense makes sense and I think uh godlin is not the only language in which we have this competing

web Frameworks battles going on and I think the situation kind of changes yearly we're uh not stuck with uh any framework from decade ago but rather trying to follow how the language communities evolve and keep ourselves up to dat so you cannot avoid ending in some debates about which framework to use both usually have their arguments against and uh for yeah usually there is no Silver Bullet there just a set of pros and cons and you have to pick something picking some one framework is usually better than trying to go for two in terms of

even building knowledge around that right because you don't have to split your knowledge between two different Frameworks EXA L and then you don't have the separate silos of framework X developers framework y developers and they cannot really easily uh contribute to each other's projects yeah exactly uh could we dive even a bit deeper so if I understand this codling starter uh in high level it's like you change something in the settings file you run some credle some command and then you get some changes reflected directly in your Source Tre being dependency updates or generated code

or whatever like how did you come up with all this what was your inspiration and well how did you implement this so my inspiration was just seeing people struggle is grad mostly uh I realized that there has to be a way to make things simple to make things declarative and it turns out it wasn't just my idea at the time as around the same time it seems like Fox at Dre brains came up with Emperor and FKS at Gradle came up with declarative Gradle so just just great mind thinks like uh and yeah so I

guess people were fed up these rad build scripts and needed something else and um well how it works is like I mentioned before the star is basically a set of grad plugins this an umbrella plugin that's configured through the settings file where you just when you select a feature uh it applies a relevant plugin based on that feature so that's pretty much it it just applies configuration based on what you picked and while the idea seems to be quite simple on the surface to be honest I haven't seen anyone doing anything like that before great

that sounds uh simple but very elegant uh how about how has our cotlin community taking taken this cotlin starter I believe it has been out for around six months or so so I believe there's already some adoption what what is your experience how people uh view it and how have they adopted it um I guess people were quite happy with being able to start a new service from with the time of starting a new service going down from a few days or a week to just a few hours that was nice and apart from that

yeah people were quite happy with how simple it is now I mean even I was quite impressed with how quick it was to uh start a new service back when we were working on our hackaton project basically we we got a working backend service in less than a day fully working like that's really nice I think and how has been the adoption are there already like services in production which were initially created with cotland starter yeah quite a few Services definitely more than 10 I uh there are I know at least five Services those were

started with Scot ster are definitely in production now and I know that there are way more in development at the moment moment so yeah I mean it's simply the simplest way to start a new service nowadays so there is no reason not to go with it right so already quite a bit of adoption going on how does the future look like do you have any plans on improving it further maybe introducing some uh addition crle plugins do you have some wish list somewhere where people can submit new ones or how does it work yeah people

well people actually contribute to it themselves usually when they need something that's uh the best way to move it forward I think U but apart from that yep some folks just create issues in the repository if they need some features um plans for the future are mostly revolved around our infra setup because at the moment when you start a new service with our internal developer portal called b. Dev uh it generates some of the infrastruct of the infra stuff for you but not everything yet and we are working on improving that and yeah there are

of course new Integrations on the plate for example AWS sdta integration so that it's way easier to work with AWS and maybe some other things now now that you mentioned w. death so uh I could maybe briefly explain it's uh backstage powered internal developer portal so our internal name for that is W dog death and yep that provides kind of a additional convenience layer for a cotlin starter especially when it comes to starting new services so could you elaborate a bit like how's the developer experience when you combine godling starter with this world. def uh

for kickstarting a new service these days so it's incredibly simple nowadays it's sort of in a way similar to what you have with springb initializer basically you just go to this web portal and pick the features you want to have you select if you want to have an HTTP if you want to connect to the database and all the things I mentioned before and then you just click next and it creates everything for you it creates the service repository it creates uh the infra definition for your new service and some other PO requests related to

Security in our infra and that's basically once you merge all the PRS it creates you are ready to go you can deploy a service to Dev and that takes like I don't know a few minutes yeah so filling a form in UI merging some PO requests and then you basically have the in version deployed in the development environment if I understood correctly yep great I think we could move on to the community aspect so what kind of internal codling Community do we have I believe we already mentioned that there are more than 150 monthly uh

contributors in our codling projects and I think you mentioned that there are more than 200 people in the internal cotlin knowled slack channel uh anything else how do we share knowledge among our cotlin BBS and how's the community like so yeah back to the slack channel uh folks usually just ask questions there and get answers we have pretty active community in that regard uh apart from that people share tips and their uh battles they won um in addition to that we also have engineering competence meeting that happens weekly where we share some knowledge sort of

engineering company companywide and I know that many groups also have their internal knowledge sharing meetings um and we also have our engineering blog where we share Cent stuff from time to time and yeah and overall we have a pretty active Community uh we try to give back uh in terms of contributing to open source projects uh for example I maintain a few open source projects of my own and I've have contributed to micrometer and um AWS CK recently and another member of our engineering team has recently created a repository v u idency plugin for cater

so yeah we're trying to give back as much as we can yep indeed in some cases we are able to open source the internal code but I believe in most cases it's too VA specific to to to make it kind of Open Source Apple okay thank you for the insights I think we could end this with uh the usual so do you have any tips for organizations which are scaling their cotlin backend development at the moment or perhaps these tips might be applicable for any any language um I would say the tips I have are

probably pretty pretty generic to be honest so uh probably aiming for one stack would be a good idea so that you don't get a spread of knowledge so that it can be more focused and yeah ensuring the best practices are followed probably the easiest way for that would be to have some shared libraries or something like an sdta and um yeah minimize uh minimize toil for developers so that they can focus on solving business problems instead of I know sitting in the meeting or battling with the dependency update or losing the right library but in

dependency updates yeah yeah great thank you I I believe those tips were indeed very generic and applicable for basically any language thank you for this discussion I enjoyed a lot hopefully listeners and enjoyed as well and see you next time bye-bye thank you bye