[Music] one of the big success factors here at Spotify is our agile engineering culture culture tends to be invisible we don't notice it because it's there all the time kind of like the air we breathe but if everyone understands the culture we're more likely to be able to keep it and even strengthen it as we grow so that's the purpose of this video when our first music player was launched in 2008 we were pretty much a scrum company scrum is a well-established agile development approach and it gave us a nice team-based culture however a few
years later we had grown into a bunch of teams and found that some of the standard scrum practices were actually getting in the way so we decided to make all this optional rules are good start but then break them when needed we decided that agile matters more than scrum and agile principles matter more than any specific practices so we renamed the scrum master role to agile coach because we wanted servant leaders more than process masters we also started using the term squad instead of scrum team and our key driving force became autonomy so what is
an autonomous squad a squad is a small cross-functional self-organizing team usually less than eight people they sit together and they have end-to-end responsibility for the stuff they build design commit deploy maintenance operations the whole thing each squad has a long-term mission such as make Spotify the best place to discover music or internal stuff like infrastructure for a be testing autonomy basically means that the squad decides what to build how to build it and how to work together while doing it there are of course some boundaries to this such as the squad mission the overall product
strategy for whatever area they are working on and short term goals that are renegotiated every quarter our office is optimized for collaboration here's a typical squad area the squad members work closely together here with adjustable desks and easy access to each other screens they gather over here in the lounge for things like planning sessions and retrospectives and back there is a huddle room for smaller meetings or just to get some quiet time almost all walls are whiteboards so why is autonomy so important well because it's motivating and motivated people build better stuff also autonomy makes
us fast by letting decisions happen locally in the squad instead of via bunch of managers and committees and stuff it helps us minimize handoffs in waiting so we can scale without getting bogged down with dependencies and coordination although each squad has its own mission they need to be aligned with product strategy company priorities and other squads basically be a good citizen in the Spotify ecosystem Spotify is overall mission is more important than any individual squad so the key principle is really be autonomous but don't sub optimize it's kind of like a jazz band although each
musician is autonomous and plays his own instrument they listen to each other and focus on the whole song together that's how great music is created so our goal is loosely coupled but tightly aligned squads we're not all there yet but we experiment a lot with different ways of getting closer in fact that applies to most things in this video this culture description is really a mix of what we are today and what we are trying to become in the future alignment and autonomy may seem like different ends of a scale as in more autonomy equals
less alignment however we think of it more like two different dimensions down here is low alignment and low autonomy a micro management culture no high level purpose just shut up and follow orders up here is high alignment but still low autonomy so leaders are good at communicating what problem needs to be solved but they are also telling people how to solve it high alignment and high autonomy means leaders focus on what problem to solve but let the teams figure out how to solve it what about down here then low alignment and high autonomy means teams
do whatever they want and basically all run in different directions leaders are helpless and our product becomes a Frankenstein we're trying hard to be up here aligned autonomy and we keep experimenting with different ways of doing that so alignment enables autonomy the stronger alignment we have the more autonomy we can afford to grant that means the leaders job is to communicate what problem needs to be solved and why and the squad's collaborate with each other to find the best solution one consequence of autonomy is that we have very little standardization when people ask things like
which code editor do you use or how do you plan the answer is mostly depends on which squad some do scrum sprints others do Kanban some estimate stories and measure velocity others don't it's really up to each squad instead of formal standards we have a strong culture of cross-pollination when enough squads use a specific practice or tool such as get that becomes the path of least resistance and other squads tend to pick the same tool squads start supporting that tool and helping each other and it becomes like a de facto standard this informal approach gives
us a healthy balance between consistency and flexibility our architecture is based on over a hundred separate systems coded and deployed independently there's plenty of interaction but each system focuses on one specific need such as playlist management search or monitoring we try to keep them small and decoupled with clear interfaces and protocols technically each system is owned by one squad in fact most quasi owns several but we have an internal open-source model and our culture is more about sharing than owning suppose squad one here needs something done in system B and squad two knows that code
best they'll typically ask squad two to do it however if squad two doesn't have time or they have other priorities then squad one doesn't necessarily need to wait we hate waiting instead they are welcome to go ahead and edit the code themselves and then ask squad two to review the changes so anyone can edit any code but we have a culture of peer code review this improves quality and more importantly spreads knowledge over time we've evolved design guidelines code standards and other things to reduce engineering friction but only when badly needed so on a scale
from authoritative to liberal we're definitely more on the liberal side now none of this would work if it wasn't for the people we have a really strong culture of mutual respect I keep hearing comments like my colleagues are awesome people often give credit to each other for great work and seldom take credit for themselves considering how much talent we have here there is surprisingly little ego one big aha for new hires is that autonomy is kind of scary at first you and your squad mates are expected to find your own solution no one will tell
you what to do but it turns out if you ask for help you get lots of it and fast there's genuine respect for the fact that we're all in this boat together and need to help each other succeed we focus a lot on motivation here's an example an actual email from the head of people operations hi everyone our employee satisfaction survey says 91% enjoy working here and 4% don't now that may seem like a pretty high satisfaction rate especially considering our growth pane from 2006 to 2013 we've doubled every year and now have over 1200
people but then he continues this is of course not satisfactory and we want to fix it if you're one of those unhappy 4% please contact us we're here for your sake and nothing else so good enough isn't good enough half a year later things had improved and satisfaction rate was up to 94% this strong focus on motivation has helped us build up a pretty good reputation as a workplace but we still have plenty of problems to deal with so yeah we need to keep improving okay so we have over 50 squads spread across four cities
some kind of structure is needed currently squads are grouped into tribes a tribe is a lightweight matrix each person is a member of a squad as well as a chapter the squad is the primary dimension focusing on product delivery and quality while the chapter is a competency area such as quality assistance agile coaching or web development as squad member my chapter lead is my formal line manager a servant leader focusing on coaching and mentoring me as engineer so I can switch squads without getting a new manager it's a pretty picture huh except that it's not
really true in reality the lines aren't nice and straight and things keep changing here's a real-life example from one moment in time for one tribe and of course it's all different by now and that's okay the most valuable communication happens in informal and unpredictable ways to support this we also have guilds a guild is a lightweight community of interest where people across the whole company gather and share knowledge within a specific area for example leadership web development or continuous delivery anyone can join or leave a guild at any time guilds typically have a mailing list
biannual on conferences and other informal communication methods most organizational charts are an illusion so our main focuses community rather than hierarchical structures we've found that a strong enough community can get away with an informal volatile structure if you always need to know exactly who is making decisions you're in the wrong place one thing that matters a lot for autonomy is how easily can we get our stuff into production if releasing is hard we'll be tempted to release seldom to avoid the pain that means each release is bigger and therefore even harder it's a vicious cycle
but if releasing is easy we can release often that means each release is smaller and therefore easier to stay in this loop and avoid that one we encourage small frequent releases and invest heavily in test automation and continuous delivery infrastructure release should be routine not drama sometimes we make big investments to make releasing easier for example the original Spotify desktop client was a single monolithic application in the early days with just a handful of developers that was fine but as we grew this became a huge problem dozens of squads had to synchronize with each other
for each release and it could take months to get a stable version instead of creating lots of process and rules and stuff to manage this we changed the architecture to enable decoupled releases using chromium embedded framework the client is now basically a web browser in disguise each section is like a frame on a website and squads can release their own stuff directly as part of this architectural change we started seeing each client platform as a client app and evolved three different flavors of squads client app squads feature squads and infrastructure squads a feature squad focuses
on one feature area such as search this squad will bill ship and maintain search related features on all platforms a client app squad focuses on making release easy on one specific client platform such as desktop iOS or Android infrastructure squads focus on making other squads more effective they provide tools and routines for things like continuous delivery a be testing monitoring and operations regardless of the current structure we always strive for a self-service model kind of like a buffet the restaurant staff don't serve you directly they enable you to serve yourself so we avoid handoffs like
the plague for example an operation squad or client app squad does not put code into production for people instead their job is to make it easy for feature squads to put their own code into production despite the self-service model we sometimes need a bit of sync between squads been doing releases we manage this using release trains and feature toggles each client app has a release train that departs on a regular schedule typically every week or every three weeks depending on which client just like in the physical world if trains depart frequently and reliably you don't
need much upfront planning just show up and take the next train suppose these three squads are building stuff and when the next release train arrives features a B and C are done while D is still in progress the release train will include all four features but the unfinished one is hidden using a feature toggle it may sound weird to release unfinished features and hide them but it's nice because it exposes integration problems early and minimizes the need for code branches unmerged code hides problems and is a form of technical debt feature toggles let us dynamically
show and hide stuff in tests as well as production in addition to hiding unfinished work we use this to a/b tests and gradually roll out finished features all in all our release process is better than it used to be but we still see plenty of improvement areas so we'll keep experimenting this may seem like a scary model letting each squad put their own stuff into production without any form of centralized control and we do screw up sometimes but we've learned that Trust is more important than control why would we hire someone who we don't trust
agile at scale requires trust at scale and that means no politics it also means no fear fear doesn't just kill trust it kills innovation because if failure gets punished people won't dare try new things so let's talk about failure our founder Daniel put it nicely we aim to make mistakes faster than anyone else yeah I know sounds a bit crazy but here's the idea to build something really cool we will inevitably make mistakes along the way right but each failure is also a learning so when we do fail we want it to happen fast so
we can learn fast and therefore improve fast it's a strategy for long-term success it's like with kids you can keep a toddler in the crib and she'll be safe but she won't learn much and won't be very happy if you instead let her run around and explore the world she'll fail and fall sometimes but she'll be happier and developed faster and the wounds well they usually heal so Spotify is a fail friendly environment we're more interested in fast failure recovery than failure avoidance our internal blog has articles like celebrate failure and stories like how we
shot ourselves in the foot some squads even have a fail wall where people show off their latest failures and learnings failing without learning is well just failing so when something goes wrong we usually follow up with a post-mortem this is never about whose fault was it it's about what happened what did we learn what will we change post-mortems are actually part of our incident management workflow so an incident ticket isn't closed when the problem is solved it's closed when we've captured the learnings to avoid the same problem in the future fix the process not just
the product in addition all squads do retrospectives every few weeks to talk about what's working well and what to improve next all-in-all Spotify has a strong culture of continuous improvement driven from below and supported from above however failure must be non-lethal or we don't live to fail again so we promote the concept of limited blast radius the architecture is quite decoupled so if a squad makes a mistake it will usually only impact a small part of the system and not bring everything down and since the squad has end-to-end responsibility for their stuff without handoffs Nikki
usually fix the problem fast also most new features are rolled out gradually starting with just a tiny percent of all users and closely monitored once the feature proves to be stable we gradually roll it out to the rest of the world so if something goes wrong it normally only effects a small part of the system for a small number of users for a short period of time this limited blast radius gives squads courage to do lots of small experiments and learn really fast instead of wasting time trying to predict and control all risk in advanced
mario andretti puts it nicely if everything is under control you're going to slow alright let's talk about product development our product development approach is based on Lean Startup principles and is summarized by the mantra think it build it ship it tweak it the biggest risk is always building the wrong thing so before deciding to build a new product or major feature we try to inform ourselves with research do people actually want this does it solve a real problem for them then we define a narrative kind of like a press release or an elevator pitch showing
off the benefits for example radio you can save or follow your favorite artists we also define hypotheses how will this feature impact user behavior and our core metrics will they share more music will they log in more often and we build various prototypes and have people try them out to get a sense of what the feature might feel like and how people react once we feel confident this thing is worth building we go ahead and build an MVP Minimum Viable Product just enough to fulfill the narrative but far from feature complete you might call it
the minimum lovable product the next stage of learning happens once we put something into production so we want to get there as quickly as possible we release the MVP to just a few percent of all users and use techniques like a bee testing to measure the impact and test our hypotheses the squad monitors the data and continues tweaking and redeploying until they see the desired impact then they gradually roll out to the rest of the world while taking the time needed to sort out practical stuff like operational issues and scaling by the time the product
or feature is fully rolled out we already know it's a success because if it isn't we don't roll it out impact is always more important than velocity so a feature isn't really considered done until it has achieved the desired impact note that like most things in this video this is how we try to work but our actual track record of course varies now with all this experimentation going on how do we actually plan how do we know it's going to be released by which date well the short answer is we mostly don't we care more
about innovation than predictability and 100% predictability means 0% innovation on a scale we'd probably be somewhere around here of course sometimes we do need to make delivery commitments like for partner integrations or marketing events and that sometimes involves standard agile planning techniques like velocity and burn up charts but if we have to promise a date we generally defer that commitment until the feature is already proven in close to ready by minimizing the need for predictability squads can focus on delivering value instead of being a slave to someone's arbitrary plan one product owner said I think
of my squad as a group of volunteers that are here to work on something they are super passionate about so where do ideas come from an amazing new product always starts with a person and a spark of inspiration but it will only become real if people are allowed to play around and try things out so we encourage everyone to spend about 10 percent of their time doing hack days or hack weeks that's when people get to experiment and build whatever they want like this dial a song product just pick it up and dial the number
of the song you want to listen to is it useful does it matter the point is if we try enough ideas we're bound to strike gold from time to time and quite often the knowledge gained is worth more than the actual hack itself plus it's fun as part of this we do a spotify wide hack week twice per year hundreds of people hacking away for a whole week the mantra is make cool things real build whatever you want with whoever you want in whatever way you want and then we have a big demo and party
on Friday we're always surprised by how much cool stuff can be built in just a week with this kind of creative freedom whether it's a helicopter made of lollipop sticks or a whole new way of discovering music turns out that innovation isn't really that hard people are natural innovators so just get out of their way and let them try things out in general our culture is very experiment friendly for example should we use tool a or tool B don't know let's try both and compare or do we really need sprint planning meetings don't know let's
skip a few and see if we missed them or should we show 5 or 10 top songs on the artist page don't know let's test both and measure the impact even the Spotify wide hack week started as an experiment and now it's part of their culture so instead of arguing an issue to death we try to talk about things like what sizes and what did we learn and what will we try next this gives us more data driven decisions and less opinion driven ego driven or authority driven decisions although we are happy to experiment than
different ways of doing things our culture is very waste repellent people will quickly stop doing anything that doesn't add value if it works keep it otherwise dump it for example some things that work for us so far are retrospectives daily stand-ups Google Docs get and guild on conferences and some things that don't work for us our time reports handoffs separate test teams or test phases and task estimates we mostly just don't do these things we're also strongly allergic to useless meetings and anything remotely near corporate bees one common source of waste is what we call
big projects basically anything that requires a lot of squads to work tightly coordinated for many months big project means big risk so we are organized to minimize the need and instead try hard to break projects into a series of smaller efforts sometimes however there is a good reason to do a big project and the potential benefits outweigh the risks and in those cases we've found some practices to be essential visualize progress using various combinations of physical and electronic boards do a daily sync meeting where all squads involved meet up to resolve dependencies do a demo
every week or two where all the pieces come together so we can evaluate the integrated product together with stakeholders these practices reduce risk and wastes because of the improved collaboration a short feedback loop we've also found that a project needs a small tight leadership group to keep an eye on the big picture typically we have a tech lead a product lead and sometimes a design lead working tightly together on the whole we're still experimenting a lot with how to do big projects and we're not so good at it yet one thing we keep wrestling with
is growth pane as we grow we risk falling into chaos but if we overcompensate and add too much structure and process we risk getting stuck in bureaucracy instead and that's even worse so the key question is really what is the minimum viable bureaucracy the least amount of structure and process we can get away with to avoid total chaos both sides cause waste but in different ways so the waste repellent culture and agile mindset helps us stay balanced the key thing about reducing waste is to visualize it and talk about it often so in addition to
retrospectives and post-mortems many squads and tribes have big visible improvement boards that show things like what's blocking us and what are we doing about it we also like to talk about definition of awesome for example awesome for this squad means things like really finishing stuff easily ramping up new team members and no recurring tasks or bugs and our definition of awesome architecture includes I can build test and ship my feature within a week I use data to learn from it and my improved version is live in week 2 keep in mind though awesome is a
direction not a place so it doesn't even have to be realistic but if we can agree on what awesome would look like it helps focus our improvement efforts and track progress here's an example of an improvement tracking board inspired by a lean technique called Toyota kata top left shows what is the current situation in this case the squad was having quality problems bottom left shows definition of awesome in a perfect world we'd have no quality problems at all top right is a realistic target condition if we were one step closer to awesome what would that
look like and finally the bottom right shows the next three concrete actions that will move us towards the target condition as these get done the squad fills it up with new actions boards like this live on the wall in the squad room and are typically followed up at the next retrospective all right I realized that maybe this video makes it seem like everything at Spotify is just great well truth is we have plenty of problems to deal with and I could give you a long list of pain points but I won't because it would go
out of date quickly we grow fast and change fast and quite often a seemingly brilliant solution today will cause a nasty new problem tomorrow just because we've grown and everything is different however most problems are short-lived because people actually do something about it this company is pretty good at changing the architecture process organization or whatever is needed to solve a problem and that's really the key point healthy culture heals broken process since culture is so important we put a lot of effort into strengthening it this video is just one small example no one actually owns
culture but we do have quite a lot of people focusing on it groups such as people operations and about 30 or so agile coaches spread across all squads and we do boot camps where new hires form a temporary squad they get to solve a real problem while also learning about our tech stack and processes and learning to work together as a team all in one week it's an intense but fun way to really get the culture they often manage to put code into production in that time which is impressive but again failing is perfectly okay
as long as they learn mainly though culture spreads through storytelling whether it happens on the blog at a post mortem a demo or at lunch as long as we keep sharing our successes and failures and learnings with each other I think the culture will stay healthy at the end of the day culture in any organization is really just the sum of everyone's attitudes and actions you are the culture so model the behavior you want to see that's it we're done I hope you enjoyed this story thanks for listening [Music]
