Hey you're back great now you've probably forgotten all about part one so let's do a quick recap our culture is based on agile principles all engineering happens in squads and we try to keep them loosely coupled and tightly aligned we like cross-pollination and have an internal open source model for code squads do small and frequent releases which is enabled by decoupling our self-service model minimizes the need for handoffs and we use release trains and feature toggles to get stuff into production early and often and since culture is all about the people we focus on motivation
community and trust rather than structure and control that was part one and now I'd like to talk about failure our founder Daniel put it nicely we aim to make mistakes faster than anyone else yeah I know it sounds a bit crazy but here's the idea to build something really cool we will inevitably make mistakes along the way right but each failure is also learning so when we do fail we want it to happen fast so we can learn fast and therefore improve fast it's a strategy for long-term success it's like with kids you can keep
a toddler in the crib and she'll be safe but she won't learn much and won't be very happy if you instead let her run around and explore the world she'll fail and fall sometimes but she'll be happier and develop faster and the wounds well they usually heal so Spotify is a fail friendly environment we're more interested in fast failure recovery than failure avoidance our internal blog has articles like celebrate failure and stories like how we shot ourselves in the foot some squads even have a fail wall where people show off their latest failures and learnings
failing without learning is well just failing so when something goes wrong we usually follow up with a post-mortem this is never about whose fault was it it's about what happened what did we learn what will we change post-mortems are actually part of our incident management workflow so an incident ticket isn't closed when the problem is solved it's closed when we've captured the learnings too the same problem in the future fix the process not just the product in addition all squads do retrospectives every few weeks to talk about what's working well and what to improve next
all-in-all Spotify has a strong culture of continuous improvement driven from below and supporters from above however failure must be non-lethal or we don't live to fail again so we promote the concept of limited blast radius the architecture is quite decoupled so if a squad makes a mistake it will usually only impact a small part of the system and not bring everything down and since the squad has end-to-end responsibility for their stuff without handoffs they can usually fix the problem fast also most new features are rolled out gradually starting with just a tiny percent of all
users and closely monitored once the feature proves to be stable we gradually roll it out to the rest of the world so if something goes wrong it normally only affects a small part of the system for a small number of users for a short period of time this limited blast radius gives squads courage to do lots of small experiments and learn really fast instead of wasting time trying to predict and control all risk in advanced mario andretti puts it nicely if everything is under control you're going too slow alright let's talk about product development our
product development approach is based on Lean Startup principles and it's summarized by the mantra think it build it ship it tweak it the biggest risk is always building the wrong thing so before deciding to build a new product or major feature we try to inform ourselves with research do people actually want this does it solve a real problem for them then we define a narrative kind of like a press release or an elevator pitch showing off the benefits for example radio you can save or follow your favorite artists we also define hypotheses how will this
feature impact user behavior and our core metrics will they share more music will they log in more often and we build various prototypes and have people try them out to get a sense of what the feature might feel like and how people react once we feel confident this thing is worth building we go ahead and build an MVP Minimum Viable Product just enough to fulfill the narrative but far from feature complete you might call it the minimum lovable product the next age of learning happens once we put something into production so we want to get
there as quickly as possible we released the MVP to just a few percent of all users and use techniques like a be testing to measure the impact and test their hypotheses the squad monitors the data and continues tweaking and redeploying until they see the desired impact then they gradually roll out to the rest of the world while taking the time needed to sort out practical stuff like operational issues and scaling by the time the product or feature is fully rolled out we already know it's a success because if it isn't we don't roll it out
impact is always more important than velocity so a feature isn't really considered done until it is achieved the desired impact note that like most things in this video this is how we try to work but our actual track record of course varies now with all this experimentation going on how do we actually plan how do we know it's going to be released by which date well the short answer is we mostly don't we care more about innovation than predictability and 100% predictability means 0% innovation on a scale we'd probably be somewhere around here of course
sometimes we do need to make delivery commitments like for partner integrations or marketing events and that sometimes involves standard agile planning techniques like velocity and burn up charts but if we have to promise a date we generally defer that commitment until the feature is already proven and close to ready by minimizing the need for predictability squads can focus on delivering value instead of being a slave to someone's arbitrary plan one product owner said I think of my squad as a group of volunteers that are here to work on something they are super passionate about so
where do ideas come from an amazing new product always starts with a person and a spark of inspiration but it will only become real if people are allowed to play around and try things out so we encourage everyone to spend about 10% of their time doing hack days or hack weeks that's when people get to experiment and build whatever they want like this dial a song product just pick it up and dial the number of the song you want to listen to is it useful does it matter the point is if we try enough ideas
we're bound to strike goals from time to time and quite often the knowledge gained is worth more than actual hack itself but it's fun it's part of this we do a spotify wide hack week twice per year hundreds of people hacking away for a whole week the mantra is make cool things real build whatever you want with whoever you want in whatever way you want and then we have a big demo and party on Friday we're always surprised by how much cool stuff can be built in just a week with this kind of creative freedom
whether it's a helicopter made of lollipop sticks or a whole new way of discovering music turns out that innovation isn't really that hard people are natural innovators so just get out of their way and let them try things out in general our culture is very experiment friendly for example should we use tool a or tool B don't know let's try both and compare or do we really need sprint planning meetings don't know let's skip a few and see if we miss them or should we show 5 or 10 top songs on the artist page don't
know let's test both and measure the impact even a Spotify wide hack week started as an experiment and now it's part of the culture so instead of arguing an issue to death we try to talk about things like what side processes and what did we learn what will we try next this gives us more data-driven decisions and less opinion driven ego driven or authority driven decisions although we are happy to experiment and try different ways of doing things our culture is very waste repellent people will quickly stop doing anything that doesn't add value if it
works keep it otherwise dump it for example some things that work for us so far are retrospectives daily stand-ups Google Docs git and guild on conferences and some things that don't work for us our time reports handoffs separate test teams or test phases and task estimates we mostly just don't do these things we're also strongly allergic to useless meetings and anything remotely near corporate bs one common source of waste is what we call big projects basically anything that requires a lot of squads to work tightly coordinated for many months big project means big risk so
we are organized to minimize the need and instead try hard to break projects into a series of smaller efforts sometimes however there is a good reason to do a big project and the potential benefits outweigh the risks and in those cases we found some practices to be essential visualize progress using various combinations of physical and electronic boards do a daily sync meeting where all squads evolves meet up to resolve dependencies do a demo every week or two where all the pieces come together so we can evaluate the integrated product together with stakeholders these practices reduce
risk and waste because of the improved collaboration a short feedback loop we've also found that a project needs a small type leadership group to keep an eye on the big picture typically we have a tech leader product lead and sometimes a design lead working tightly together on the whole we're still experimenting a lot with how to do big projects and we're not so good at it yet one thing we keep wrestling with is growth pane as we grow we risk falling into chaos but if we overcompensate and add too much structure and process we risk
getting stuck in bureaucracy instead and that's even worse so the key question is really what is the minimum viable bureaucracy the least amount of structure and process we can get away with to avoid total chaos both sides cause waste but in different ways so the waste repellent culture and agile mindset helps us stay balanced the key thing about reducing waste is to visualize it and talk about it often so in addition to retrospective than post-mortems many squads and tribes have big visible improvement boards that show things like what's blocking us and what are we doing
about it we also like to talk about definition of awesome for example awesome for this squad means things like really finishing stuff easily ramping up new team members and no recurring tasks or bugs and our definition of awesome architecture includes I can build test and ship my feature within a week I use data to learn from it and my improved version is live in week two keep in mind though awesome is a direction not a place so it doesn't even have to be realistic but if we can agree on what awesome would look like it
helps focus our improvement efforts and track progress here's an example of an improvement tracking board inspired by a lean technique called Toyota kata top left shows what is the current situation in this case the squad was having quality problems bottom left shows definition of awesome in a perfect world we'd have no quality problems at all top right is a realistic target condition if we were one step closer to awesome what would that look like and finally the bottom right shows the next three concrete actions that will move us towards the target condition as these get
done the squad fills it up with new actions boards like this live on the wall in the squad room and are typically followed up at the next retrospective all right I realize that maybe this video makes it seem like everything at Spotify is just great well truth is we have plenty of problems to deal with and I could give you a long list of pain points but I won't because it would go out of date quickly we grow fast and change fast and quite often a seemingly brilliant solution today will cause a nasty new problem
tomorrow just because we've grown and everything is different however most problems are short-lived because people actually do something about it this company is pretty good at changing the architecture process organization or whatever is needed to solve a problem and that's really the key point healthy culture heals broken process since culture is so important we put a lot of effort into strengthening it this video is just one small example no one actually owns culture but we do have quite a lot of people focusing on it groups such as people operations and about 30 or so agile
coaches spread across all squads and we do boot camps where new hires form a temporary squad they get to solve a real problem while also learning about our tech stack of processes and learning to work together as a team all in one week it's an intense but fun way to really get the culture they often manage to put code into production in that time which is impressive but again failing is perfectly okay as long as they learn mainly though culture spreads through storytelling whether it happens on the blog at a post mortem a demo or
at lunch as long as we keep sharing our successes and failures and learnings with each other I think the culture will stay healthy at the end of the day culture in any organization is really just the sum of everyone's attitudes and actions you are the culture so model the behavior you want to see that's it we're done I hope you enjoyed this story thanks for listening you
