Startup Technology - Technical Founder Advice

I would like to introduce Jared Frieden my partner and his esteemed panel who he will introduce to talk about technology thank you thank you Jeff okay well I am super lucky to have a very esteemed group of guests with me here today the everyone on this panel is is a tactical founder of a really successful company everyone here has built like a really cool company in a really amazing technology organization we've changed The name of the event for today it was originally called a CTO advice and on the advice of Lillian thank you so much

we have changed it to tactical founder advice which i think is a better description and I'd also like to thank everyone on the startup school forum who wrote in with questions for the panel we posted a few weeks ago and solicited questions we got over 150 responses so thank you everyone who wrote in with those questions we're gonna do our best To cover as many of them as we can and then at the end we'll open it up to the in-person audience or some questions as well okay so let's get started could we start by

having everyone introduce themselves and tell us about your company and about your technology like what's your tech stack what's some of the interesting technology that you build how does your technology organization look like today oh you monster hello everyone my name is Ralph Judy I'm CTO and co-founder of plain grid clean grid were 350 people based out of Mission San Francisco we write beautiful easy-to-use software for the 17 trillion dollar construction industry so what that looks like to you the analogy often uses we're like github for construction construction has blueprints blueprints change rapidly version controls

extremely important if you have changes that means there's issues that are Happening issues need to be tracked and then we build collaboration tools the top of that too as well as a lot of other tools for the construction industry that go into some deep jargon our stack we're you know based on AWS we used to be based on a variety of other things we had to move everything into AWS over time on our back end mostly Python on the back end that we've got some go and other things and then one of our Challenges is

we actually write native for every platform so we've got iOS with objective-c Android all Java and Kotlin the web reacts and then windows we have a full Windows app as well which has done that hi everyone I'm Calvin French Owen and CTO and co-founder of segment segment is a single API to collect organize and adapt all of your customer data into hundreds of downstream tools you might be using whether that's an analytics tool like Google Analytics or Mixpanel maybe a customer success tool like gain site maybe an email tool like customer i/o segments helps you

collect data from where it lives and get it where it needs to be in terms of the company overall we're a little over 300 people right now we have our headquarters in San Francisco but we also have offices in Vancouver New York and Dublin and the engineering product and design team which is kind of how we build product here at segment is a Little over 80 or 90 right now in terms of our tech stack we're also built entirely atop of AWS in terms of how we run our infrastructure we run on top of ECS

Amazon's container service and pretty much all of our different services our containerized today we're running about 300 different micro services which are kind of piping together various kafka topics and reading and writing transforming this data getting it to where it needs to go And our back-end is primarily written in go good morning everyone so my name is diana who i was the founder and CTO for SEO reality and we're building the backend for augmented reality and i say i was because my company just got acquired by Niantic the makers of pokemon go so what we

were building an Escher and now we actually continue doing it in Niantic is building this back-end technology for augmented reality to enable developers to build a Our experiences as easy as possible so we handle all the complexity with the computer vision rhythms with all the interaction with the backend all the hard parts of getting things to render so that developers could build easy a our apps in minutes so that's what we do and we provide a lot of advanced they are features like multiplayer persistence and cross-platform so that is easy for developers are just Developing

and let's say on unity and it just works so in terms of tech stack has been a bit of an adventure back at Asher we were we had a service that was hosted in AWS but that didn't matter too much because we had everything in docker containers now a Niantic is I think is heavy user of Google Cloud so move everything to Google Cloud but we have a lot of the bulk of the code is actually for us native means more C++ literally because we is a lot more efficient for Us to write the code

and a lot of the algorithms once and then cross compile it across all the architectures for Android or iOS and even the back end so that we run some of the CVL rails we could prototype it in the phone and then we could easily move them into the server because our service actually also written in C++ hi everyone I'm Lillian I am CEO oh not CTO of second measure so second measure we're about 50 people we got started back in 2015 and we're based In San Mateo what we do is we analyze credit card data

so basically we take billions of credit card transactions and try to build a daily view into the public and private company performance we analyze these billions of transactions automatically every day and just them clean them enrich normalize them and then service that in a front-end application for our clients which include VC firms hedge funds and big brands like blue apron or Spotify to Check trends do any sort of competitive intelligence or customer intelligence so we actually don't have a CTO so that's a fun fact about us and that's why I'm here on the panel and

myself and my co-founder actually both technicals that's been really pretty fortunate we don't have to deal with as many of the challenges around starting with non-technical founders so our I mentioned that our team is about 50 people primarily technical so our Technical organization is about 30 people and that's actually split evenly between data scientists and engineers and that's probably something that makes us unique is that our core product is actually the data itself so our data scientists and engineers have to collaborate super closely on a day to day basis and probably one of the things

that has technically been interesting is a lot of our users want to explore and dive deep into our data and building a Front-end that allows that flexibility of exploration while still creating a good user experience basically requires us to figure out how to rapidly query data and run really complex queries on our back-end yes about tech stock we're also primarily on AWS so for our pipeline we leverage services like lamda and spark and then for our front-end we react based and leveraged column columnar data stores on the backend to service queries So I think redshift

awesome guys that was super cool it's so cool the diversity of different kinds of technology organizations and products that are here today okay so most of the company is in start-up school are really early so I want to take everyone here back to their v1 the very first version before you had actually shipped anything tell us the story of how you built your v1 who built it how did you built it how long did it take you to build it and What things went wrong in the process I guess I'll start this is a great

story I've actually got my CEO and co-founder over there watching me right now and um so that's company hi Tracey yeah hey Tracey how's it going so that's that's where we started Tracy told me about her idea of putting blueprints on an iPad this was in 2011 right at the launch of the iPad so it was very early for the technology and I said yeah no problem I could put Blueprints on my PI it's like a PDF that's no problem at all so I was attempting to impress her really quickly by putting blueprints on an

iPad and it turns out there was actually some hard core challenges with putting these giant images on a really low computer availability on the iPad there's no graphics chip at the time so basically these are 20,000 pixel images and they would just crash or overrun the video memory and they're in PDF which is A kind of painful format to deal with so the first prototype sucks um is really slow and that actually told me that there's something real meaty here there's something that's actually a challenge and it took me about a month to write the

second prototype based off some my background actually I used to work at Pixar so I had some graphics experience there didn't use any proprietary stuff but use some off-the-shelf computer graphics Knowledge to write the first blueprint viewer that ever ran on mobile so that was our first prototype back then and I think the best thing I remember back on it is um the only thing we really focused on the first prototype was what was technically impossible there was no way you could load blueprints onto an iPad at the time and that was our prototype which

meant that all the data got there by side loading which meant there was no web interface and the Little crappy web interface we had had a delete button right next to a publish button that was not a pixel apart and there was no confirmation on the delete button so we actually as a team we managed everything on the back end we kind of like behind the curtains loaded all the documents for our first 30 or 40 customers but they all they saw was the the pot the prototype that was fast and did what they thought

it would they never saw the man behind the curtain so To speak so that's all the story of our first prototype cool um I'll share a little bit of a story of our v1 and then maybe our v1 next I think for those of you has seen the startup school talk where my co-founder Peter talks about finding product market fit he goes into this story and way more detail than I could today but I can share a little bit of our journey so when we first started we were in oh I see 2011 class which

sounds like a Dinosaur now in terms of YC years yeah but at the time we were actually building something completely different from segments today we were building this classroom lecture tool which would help professors get feedback about what their students were thinking in the middle of class we were students at the time we figured hey like this is seems like a great ideas professors will sometimes go on for five minutes and they'll lose the entire class and the Class gets confused and just waste everyone's time let's solve that problem we built it out over the

course of the summer and we deployed it back in Boston in the fall and the whole thing just kind of crashed and burned like students would go to Facebook YouTube Wikipedia like in retrospect all the things that you would assume college students would do if they have a laptop in front of them in a lecture but we didn't quite see it that way So we started looking around for a new idea because we've just raised a seed round and we say okay what are the problems that we have with this college lecture tool and one

of them was that we couldn't answer questions about how people were using our tool we can tell you how a college students at Harvard we're using the tool differently than students at MIT or we couldn't tell you how anthropology classes used it differently than biology classes and so We switched again to something which also we don't do today but effectively it was a competitor to Mixpanel or amplitude is this tool that would allow you to cluster your users by different rules and break them up into segments which is where the name came from and from

there you could then reach out to them or figure out what you're most engaged users were doing and so we built out that system for about 15 months the whole time we were doing revs on the Backend infrastructure we were building it out and we're trying to get users and no one wanted our tool and so we arrived fifteen months later we actually moved back to the Bay Area after living in Boston for a year we go and talk to PG and we tell them about kind of our whole saga that's been running for the

past year and a half and you listen to our story and he listens to everything that we've done and when we're finished he just sort of Pauses I think it was maybe 400 feet out there walking around the roundabout he says wow so you just burned half a million dollars and you're still at square one they're just like yes he said well on the plus side you still have some runway left so try again like want something new and at the time we had this internal tool that we had been using as kind of a

growth hack to get ourselves more users and the whole idea was that you would use the same API to Send us data as you would send to Mixpanel KISSmetrics Google Analytics all these other tools and people actually liked that single API and they were contributing to this open source library and github and starring it and using it and so my co-founder Ian said hey why don't we turn this into a product why don't we launch this and see what happens and my co-founder Peter said oh that's the worst idea I've ever heard it Will never

work he's CEO now but he's seen the light and so I he said okay I've got to figure out a way that we can test this idea incredibly quickly so we cleaned up the library and we launched it on Hacker News and to our surprise it actually got about a thousand stars that first day I'm github and so our real v1 was that first week taking that library and we basically just had a landing page up which said hey if you're interested in a hosted version of this leave your Email and about 500 people left

their email and we built out that v1 over about a week where just everyone was in Mad hackathon mode and building out the product which we launched a week later and so today I think there's actually no parts of that product which still live on today but it was enough to get us kind of the seed of product market fit and then start getting us a user base so one week is the short one week one week everybody hear that one week ok that's Kind of amazing one week sir story is definitely more than one

week so me and my co-founder have been thinking of building different things and one of the technology trends that we really saw was he it comes from a robotics background with a lot of experience with slam slam is this album called simultaneous localization and mapping and it's one of the core algorithms that run today in AR which helps tell where your camera Position is while you also build this map of the world so what happened is there's this category of algorithms that have been used mostly for robotics now AR had been kind of tried and

many times but never really being picked up there's AR with markers and that never really took off and then we thought about why do we bring these algorithms and run them on the phones and we got to the point where we thought we could do it because the computation at that point With phones for example for iPhone 7 has the same competition actually as a MacBook Pro from 2013 so that's you think about that that's kind of amazing in terms of all the trends with Moore's law and some of the ones with power efficiency and

compute so we decided to go do it so we were at that time I was actually still working and I was leading a data science team back then and say okay I'll do it because I was thinking of doing some transition and I was Working part time with another engineer and this one of the engineers that that we kind of got excited to work with us so we struggle a lot to try to get a version working the first was very duct tape and it wasn't very encouraging it was only working at five frames per

second which is terrible because if you want a good AR experience the whole thing is that you need it to render nicely and be at least 30 frames per second so that was very discouraging in A sense and then because it was all kind of put together this and then we started kind of really removing all the research code out and then it started working and seeing a lot of promise and it was running then at at least 30 frames per second and note that when we were working on this was at least about a

year and a half before air kid had launched and then we were excited about this so we were both very more from a technology side so we don't know product Market fit way to go with this so we had to find the market so the exercise we did is actually we went in an interview a bunch of people that we thought would use it and we found a really good for it with gaming so we decided to focus on that and that was sir YC application and the thing that really took us and this took

off and this is actually I guess a good story too is the first day of YC Apple announced air kit Which is basically what we had built and that was sort of a moment of truth for us it's sort of this really moment where you are looking at yourself and it is really go flight-or-fight so we decided why not let's just take on Apple so we decided to just double down and we had this idea that the start wasn't just on the device which we had all our demos and Technology was just working on the

phone we didn't have anything on the backend yet so back in YC we got it Working with a back-end and also with Android and basically accelerating the roadmap that we thought it was about a year shrink it and got it done in three months and that was the thing that once we put it out there and let people sign up if they were interested we go got about a thousand developers signed up in about a week so they find okay I think we found something and then after that we got a lot of interest and

among those Niantic was one and the story is we got Acquired later so how long was it from the time that you started working on the like original slam algorithms until you had it like a working product in users hands so we were not really working full time it was probably about a year of part time and even by the time was part time wasn't really working so when full time I took another six months of just me and another engineer to do it and then during the summer YC we hired two more Engineers and

that's when we were able to really accelerate and get it working well cool so for us our v1 probably took assuming full-time two or three months to build and I actually would say that most of that time was just figuring out what our product was going to be we knew that the that we were working on is gonna be really interesting initially we were focused on investors basically trying to get them an information edge on sort of Any investment decisions that they're making whether that be hedge funds focus on public markets or VCS focus on

private markets we ran through a few ideas we're like hey do these investors want predictions we they just want to know how like this company public company's quarterly sales are gonna hit and the short answer is yes but but no that doesn't really seem like a really sort of sustainable business model so then we shift it a little bit like okay Maybe these investors really really want to go deep and they just want to cut data any way they want so we put something together really quickly put that in front of some of our friends

your investors and found out very quickly through some white user testing that that was way too overwhelming our users wanted some guidance so what ultimately landed on is we needed to build our own self-service platform where we can apply our opinions around How our customers should be viewing this data and getting quick access to insights so once we figure that part out I was actually pretty quick so my co-founder focused on building the analytics product piece so the the front end application and I'd say you know and together we kind of focus together we built

sort of the data pipeline and sort of the transformations on the data but I think some important choices that we made early on is and maybe some that you Guys are facing right now is what technology do you want to use like what do you want to build this in ultimately we decided to build our first application in groovy using the Grails framework which is very not shiny and not that many people are doing that but that was actually the code that we had the most experience in so my co-founder Mike had built large production

scale system servicing hundreds of thousands of concurrent users and pretty much Within a week had the bones of the application and kind of like some of these other stories here like none of us it sounds like are running rv1 anymore so it's more important in the early days to be able to iterate quickly and usually that comes from using the technology that you know the best building things in a modular way such that once you actually find that product market fit and know what your audience needs then you can Focus on that area and actually

select tech technologies that are best fit for that problem awesome okay so I'm gonna go next to the very top voted question from the startup school forum which is about the trade-off between engineering best practices like good test coverage and security and scalability and he done and see versus writing code as quickly as possible and shipping something so can everyone talk about how you made that Trade-off for your v1 and then how it's evolved since then and sort of like the timetable of its evolution as your company grew and those things became presumably more important

and just to mix it up let's try flipping the order and we'll start with Lilly in this time sure so speed is paramount nobody's gonna pay you for having excellent test coverage so in the early days the it's it's you definitely want sort of the bit of like What you need right so it was that's the risk to your business around sort of security and not having things working and obviously you want your whatever you build to run every day you don't want to be fixing it breaking all the time and not actually building but

really it is a balance and I do believe that initially it's more important to have that speed of development and incorporate constantly testing and incorporating the learnings into your product than it is To have like the most robust most scalable product great you're trying to find something that people will pay you money for ourselves a real problem and that takes time and a lot of iteration that has definitely evolved since we got started you know once you find product market fit a lot of these problems change you know your system gets more complex your team

grows you have a lot more people contributing code a lot more ways for your systems to Break a lot more users who are relying on your product so for us the way that manifested itself is yeah we started writing unit tests to verify each engineer would be verifying that you know their code works then we introduced CIN CD to make sure that that could work with the whole system our director of engineering who leads our engineering organization started developing and formalizing our best practices around like code reviews testing etc in Defining those processes and then

finally building some more controls directly into our system to make it harder to break things and specifically like what was the rough timeline for those things like are we talking about like two months after launch two years after launch so I'd say most of that stuff I was just talking about happened probably the last year to a year and a half so and that's how many years after about a year and a Half after launch and the main reasons for that again were the growth and the complexity of our systems and you know our techno

are technical staff tripled in the last year so just that alone introduces a lot more need for process in our case it was always about trying to get to MVP that was reasonably walking wet well and all that was definitely duct tape code there is nothing that I'd be proud of it's just Frankie's I had put together and barely Working as much as possible so we were there even when we kind of private launch in a sense when we that the private beta and got customers who kind of sign up and we were actually had

a beta a private beta that we had some number of game developers working with us and that was also the duck tech code the only time we started actually having all the best practices is actually once we joined I antic because now we have to Integrate into the games which pokomoko has hundreds of millions of users so we're bearing ourselves to get a lot of the we pretty much we wrote the whole codebase none of what we had before is there it's all rewritten through pair of do that that expect that number of users cool

I think for us at segments there are kind of three distinct phases of our development I'd say the first one was when we were in full hackathon mode right where we definitely didn't have Any tests no CI we're I'd say there was an 80% chance that we were just going to throw it out two weeks later because it wasn't going to stick and then we were going to move on to the next thing I think because we had been burned so much by building out all of this infrastructure and investing a lot of these ways

to provision your infrastructure and write tests and spin up different new services over the past year and a half and it really I gotten Us nowhere we had no users it has created this like pretty aligning homing instinct for us where we just said no matter what the only thing that matters is getting users at this point in the game so that's where where we started and I think that lasted us probably for about the first nine or ten months where we were just focused on rapid iteration getting more users making sure that we didn't

run out of money and then the whole thing Wouldn't be been mattered I think from there the the first phase that then shifted was when we brought on our first engineer TJ Wright for those of you who know are involved with the node community TJ is one of the best engineers I think I've ever worked with I think 90% of node shops run on some part of his code and he basically came in two segments and he looked around at well this mess that we created the product that we Created any said well there's no

way for me to work in this like I have a horrible time onboarding I think it took him a week to get his entire laptop set up and so we kind of moved from this point where the entire development team shares the same tube of toothpaste to now starting to have more and more engineers involved with the project and when that happened it really stepped up our game in terms of testing in CI And just reproducibility when it came to running the builds and running the stack because suddenly we had to expand outside of just

the four of us and then I think so maybe that period lasted another two or three years probably in the past two three or four years from now or from this point in time we've gone through another shift where we've started thinking a lot more about end-to-end testing security kind of best practices around handling all of our Customers data and the real reason for that is that now we actually have a pretty significant amount both like revenue and customers and reputation to lose from day one we had no users so it didn't matter if the

whole thing went down for a day or hours or whatever was just the calculus didn't make sense there but today we have thousands of customers who are relying on segments to publish their data to not lose it to treat it and handle it securely and so For us the investment makes way more sense and and that that last period like roughly what scale did you have to reach before you started thinking seriously about those things yeah it's a good question I think when we started having enterprise customers who are paying north of six figures per

year that's when it started to shift and we started thinking oh we really have to take care and how we're handling this data because these customers are really depending on This to power their business so I think my story's a little different than the other panelists this might be interesting I would say all three of those different facets I think I heard security I heard scalability and I heard engineering best practices we really have to treat them independently at plain grid klinger's used to build all in most most hospitals most jails most heavy civil roadwork

were used on all The tech campuses all of the different government buildings that go up so obviously security from day one I mean that's that's key to us so we had no choice but to always be super conscious of security and scalability for that matter because our customers the only way we're useful is if we're used to build the project like we're the replacement for blueprints which means in construction if we're not working properly like no one builds and not Building for a dam construction can seriously impact the bottom line of these businesses so for

us the first two security and scalability we're always key scalability is a challenging one because it's I don't know I'm not sure if there's a way to do it without it being kind of reactionary you either over engineer everything and then you maybe never get a product to market or you eventually have to deal with scaling issues and for us we tried to architect It in such a way that we would last through the foreseeable future the foreseeable future would come and some customer would find the first kind of flaw in the scalability and we're

used I mean our to give you an idea of the scale we've probably about we've customers with projects that are like 500,000 blueprints with a ton of changes and maybe over 5 million annotations on one project and we're work we work offline as well so this is all Downloaded onto a cache on the device we can actually our physical size on an iPad can take up like a hundred gigabytes for certain projects so we have like all kinds of scaling issues we've had to approach and we always try to balance it between engineering like a

year ahead of time but not maybe three years ahead of time which is a trade-off so you know find that trade-off yourself I remember when we were in - I see we had all these stories of companies that Had built the most ironclad architecture and just never had a product to market so that was on our minds while we were building that for engineering best practices are v1 some of that code still exists in the product today so some of the code I wrote for v1 still there I would say it's probably our measure of

technical debt as an engineering organization so that's humbling but at the same time like I've seen some developers and you know we're fully uh You know we do everything cool now see ICD we run integration tests we've got a UI testing team so this is feedback I'm giving from the past but you know if you're a experienced developer it's pretty hard to write spaghetti Kraft code you know it's actually challenging most experienced developers and experienced some people are just really good from the get-go it's normally something through time and writing a lot Of production products

that you just generally learned the tricks of not writing spaghetti code pretty quickly and you know some of that code runs without heavy testing the other thing to mention is to properly scale test our product it doesn't I mean it would take days during the integration test to download and upload all the data so we really have a lot of functional testing to help us double-check but the thing I've noticed with UI and unit tests is We're big fan of unit tests for a big fan of tester and development I've definitely seen developers write the

most spaghetti test frameworks and the most spaghetti tests where they're just testing their mocks and they're doing this in the middle of a production environment you know like a production need for release and they spent so much time writing these unit tests that actually were not testing anything rather than getting the product out the Door so engineering best practices are great there's a trade-off between writing a lot of tests and writing good code from the get-go so speaking of test-driven development a lot of the questions were about the various engineering methodologies agile development lean startup

test-driven development extreme programming how do you guys think about those do you adhere to any particular methodology in your company I would describe us as agile ish So we do not fully adopt any of those methodologies basically we find what works well for our team and aren't super dogmatic about you know is this agile or is this not you know we run sprints we have daily stand-ups so we have some elements incorporated in our development but I think kind of with anything you know a lot of these problems you're going to be solving a lot

of problems for organization and you have to kind of find what works best for you and so I Think it makes sense to take bits and pieces that work for you and adapt it to your organization and then I guess it's just the one other thing I'd mentioned on this is as you grow a big a big challenge is just constantly evolving how you work right so it's very different to work on a small team before where everyone kind of knows everything and everyone everything is then everyone's heads versus a team of 30 or 50

or for some of us you know hundreds that is very different so you kind of constantly need to be assessing is this style of working or this methodology is this right for this stage of the company yeah I think the point of evolving the process is something we've been doing a lot so back when we were that's chair we were only with four engineers and me building the product so back then it was very messy we're just trying to move as fast as possible really so zero Documentation everything was kind of whiteboards and everyone kind

of could handle everything in their heads and and build it out and in terms of we didn't really do Sprint's because they think the some engineers are even quite liked it as much and there were just too many big tasks and we just trusted different individuals to go and tackle kind of one area and then we would integrate it but now things are very different as we hired I think team is triple ready in The past eight months for the AR platform product when you have a bigger team you definitely need a lot more process

to be able to be efficient and communicate because you don't want to duplicate and not everyone knows everything and the system grows in a lot of complexity so we went from zero documentation to a lot more when from we used to have a little bit CI but now is very much of a CI that actually builds to all the architectural flavors to Linux mac OS android x86 Android arm iOS etc and actually runs all the tests in all of them and actually catches a lot of bugs and we tests we have coverage and all those

things but we also have to train and get the team to be ok with having more process so now we do not quite like a sprint but we do weekly planning for the week and then we reconvene and I think the teams are kind of breaking break it down into sub teams that work and people kind of float in And out of their fan focus on areas we split in three main focuses one is sort of the back end folder work with the back end I lower the work on the client to make it cross-platform

the unity API is and the other the big one is bringing a lot of the computer vision algorithms to production so the funny thing is that everyone in the team has ended up being and trained up to be a computer vision out engineer because that's sort of the core we have CV algorithms on the server Let's see the algorithms in the client and the core algorithms that run so there's been this how it kind of shaped up to be but we don't really have a per se a process where we do right now Niantic that's

okay our so now we have all cares as well per quarter so now we're starting to plan longer longer and have ideas of before you still have like a rough idea where things would go but now this this and this and this yeah for us we don't have any set process that Teams have to follow at segments kind of individual engineering teams sort of like what Spotify does are able to self organize they're able to run however they want they want to do Sprint's at school they want to use JIRA they can do that if

they want to use asana whatever it is they're allowed to run however they'd like to run the one thing that we've introduced over the past two years similarly the okay our model for those of you are familiar and this was a model That was developed at Google it's a idea of O's or objectives and then key results so you have your one objective where you're trying to go and then key results that are supposed to be objective measures of how you get there such if you do every K R then it adds up to the

full objective and so those are something that we do on a company-wide basis every single topic team in the company gets together and puts together their ok ours They say one week before each quarter and then for that three-month period they're just executing on that plan and some teams grade those on a weekly basis and check in and say how am i doing against these goals some teams grade them on a monthly basis it kind of doesn't matter but at the end of the quarter every team is saying hey here's how well I did against

my stated goals separately for the teams that I work with and Particular we end up running kind of just a weekly meeting where at the beginning of the week we end up planning out what we want to get done and then we have kind of a daily stand up I honestly don't care too much about the content for those meetings so long as we're always discussing the most important problems we're pretty big believers that the tools and process should be there to serve us not the other way around my experience at plain grid echoes all

The other panelists it's agile ish self-organizing every team can choose the tools that they want maybe some things that'd be helpful to you from the beginning some things we've always found that's been very huge useful in every engineering team are again the daily stand-ups you've got to communicate on a daily basis as engineers sometimes it can be a little difficult to want to communicate every day and not just program when you wake up but that 15 Minutes and you can do it remotely as well over slack or something has always been really helpful in just

kind of unblocking people time boxing I mean I think that a lot of these management practices all roll up into some abstract philosophies and sprints or timebox methods where you're just not going to work on something infinitely that's very helpful to kind of keep people's realities in check because often estimates can be off and you know Sometimes what we thought was going to take us a week can take us a month myself included so time boxing is a good way to keep keep categories of that I'd also say some other light weight management techniques you

can probably employ right now and your team is probably one to ones weekly one to ones just make sure if you're talking to people in a group and you're having stand-ups make sure you have some time to actually connect with people Individually and get to learn more about their wants their needs their emotions and their careers that's great guys okay so now I want to change topics to the right way to work with non-technical co-founders and I think the topic that came up most commonly in the startup school forum was the one that ralph alluded

to which is how to deal with the deadlines and timeframes particularly with non-technical co-founders and I think particularly in the in the early Days so does any have thoughts on that any any orders get I'll volunteer for this since my technical non-technical co-founder is staring at me over there um a few a few ideas here if you've got someone that's really non-technical and I'm not saying my co-founders really non-technical I'm sure she can use your computer but you know if it's really non-technical this is an amazing testing opportunity for you I mean I remember I

would write this Software and I'd be so happy about it and I'd hand it to her and as soon as she touched it it would break I mean I don't know what she shook the iPad she rotated it three times but she had a way to break the stuff I wrote and that was amazing for testing in the beginning so that's one way to engage your non technical co-founder you eventually have to learn how to start pouting your deadlines that's really hard to do I never got very good at this I'm always Optimistic even to

this day I'm like oh yeah that's gonna take me a week so I never really got good at this I'm just aware that I'm optimistic about it and then I'm always kind of off by week and she's aware of that as well so eventually I've talked to other engineering leaders they keep two books they keep like a separate set of books I talked to their their technical co-founders non-technical co-founders about say okay here's the deadlines here And then you have another set of books that's actually when you think you're gonna get the job done I

think one thing that can be really helpful is having a process some very lightweight process of like here's where we're like testing and then here's release because often some people might not realize that iOS has a release cycle that's not controlled by the developers it's controlled by Apple so you got to pad that into your plan you have to plan a little bit so I found That a little bit of project management goes a long way when you're dealing with a non-technical co-founder and even better if they're good at project management that's a great place

to employ them as well so all of my co-founders are technical so we didn't really run into this problem in the early days kind of everyone's like oh yeah I know what's happening here if it slips we know exactly why it wasn't as big of an issue I found more Recently with deadlines kind of the trick that I've started using is in one-on-ones actually asking each person what they think will happen over the next three or four weeks and when I find asking one to one is that one people don't anchor against each other's answers

so you get a very unbiased view and it like forces each person to think about the your term future what's going to happen and second each person is much more wary Of the parts that they didn't work on and so you get kind of a sense of like where there might be trouble spots and typically the end results is sort of like a wisdom of the crowds where it's kind of the average of all three or four answers and so I've started using that as a way to get a better sense of deadlines where each

person has their prediction or mental model of what's going to happen and then you kind of average them all out to where it should Be um well I guess in my case my co-founder didn't work in engineering or build products or shaped products mostly just in research so we've been doing mostly a PhD so in that sense it's kind a little bit you know technical in a sense that not having experience of building and shipping products it's mostly just kind of the algorithm side so there was a bit of that work to kind understand why

certain things take long and why they need to be built a certain Way but at some point it was more he just letting me handle all of that and he got engaged mostly on the external communications business trying to find partners and all that so that's what happened sometimes that could be the split where as the technical founder you end up owning the product and a lot of development and having clear communications and at least try to set expectations that with deadlines that's the hard part is always I think this is Always a trade-off between

engineering and product and business right the trade-off on when to get things and be able to communicate that clearly so my co-founder is technical so I don't have much to offer here not much to add either on the deadlines and time estimates to do your best guess do some padding I'd say one thing kind of on the non-technical front though that I do think is worth mentioning is so you know today a lot of you are probably building Your products and in writing code especially if you're one of the technical co-founders at least in my

experience that changes so I pretty much stopped contributing to our code base about a year into the company and it's for us kind of as leaders well I can't speak for all of us but at least for me there's a pretty big shift that happens once you start gaining traction where it's less about building a product for a market and Finding that fit and it shifts more into turn like building an organization and a company and a system like this system of humans who are going to build something long-lasting and great without a lot of

your involvement so I guess what I'm trying to say is you know at some point you may end up doing a lot more non-technical things so get ready for that yeah that's great ok so a lot of the companies in start-up school are at the phase where They are trying to figure out the right way to structure their early engineering team if they want to hire people locally if they want to hire people remotely if they want to hire only employees who work full-time or if they want to hand off pieces of the product to

contractors or or to to third-party development firms can you guys talk about how you would think about that if you were starting a company now advice that you give them I think This is going to be the last question then we're going to open it up to the audience I can start so we did not start with contractors we started by building like hiring full-time employees I think the main guidance I would give here is you are building a product in a company and you're developing these core competencies I'd say for the most part like

if you end up contracting that out you're actually not just contracting out through that technology expertise but You're learning a lot in these early days of what's going to work for your clients wouldn't like what's the what the product actually is and so moving that sort of having that external to your company I think is a really big loss in the early days just in terms of all that knowledge I could see an argument if you're just like getting off the ground and you're doing contracting kind of as a way to evaluate potential new hires

I could see that Definitely being a path but again that sort of with the intention of building your team on outsourcing front I think it's a similar thing of you really need to look at like what is your core competency that's what you're investing in what's your IP you don't want to outsource that at some point if you do you're pretty much just a marketing company and so I'd say if you are if different business models might have different requirements so for some of You it may make sense to outsource a large part of what

you're building but for us we have experimented a little bit with outsourcing I'd say our approach has really been to make sure those projects are very well defined and not on the critical path so that we can experiment it is a different type of project to manage sort of outsource Talent and that way you can sort of learn see if that works for your for your business and if it does great If it doesn't it's it's fine was there also a question about remote local versus remote employees I think that depends on what type of

culture you want to build so for us we was really important for us to have our team together so we hired a local team and then sort of in the last year so we have actually started tapping into remote engineers there's definitely like you know a little bit of a culture shift with that we're still predominantly Local but there are a lot of advantages including being able to tap into other talent pools and it has actually turned out to be a really good forcing function for documentation right explaining your code explaining your decisions which is

good for a number of reasons including just like as you grow communication is one of the harder problems you have to solve so I think for us because so much of the core that technology is what our company is we didn't quite outsource any Of that we did contract people as more of an interview process so we would work with and contact someone for a month before we gave them an offer so that's what we did but that we did outsource more things that we really thought they were not essential to her company for example

building 3d models some of the design some of the actually writing technical documentation at some of it website things that were we could do it if we had the time I wanted to We knew we could do it stuff like I could build a website but our time was better spent on that core text so those sort of things we did outsource and in terms of remote versus local because everything was so complex with what we build we chose the culture to be more all on-site and to this day we still are building the team

locally but I think you could handle a lot of stories of companies that do remote but you have to kind of start remote first cool yeah I Can share a little bit from our story so in the very early days we basically only hired full-time folks we did an experiment with contractors kind of the only case where we'd outsource is when eight of us could take some piece of infrastructure that we're building or like some new product feature and we could build on top of that I think that was still probably the right move especially

in the early days it feels like a start-up is really fragile and You want a bunch of people who were just all pointed in the same direction and really kind of in it for the long haul on the local versus remote piece we actually started hiring pretty much only remote people we're really involved with a lot of different open-source communities on github particularly there were some node in the very early days some JavaScript ones component etc and so these were people who we had been working with and Collaborating with for years at this point who

then when we were ready to start hiring we just said hey which you like to work more closely together I think it came with its own set of trade-offs on the plus side we had access to these people who were insanely talented not in the San Francisco Bay Area but who were used to working with us already and who we very clear like built an insane amount of value for segments I think the trickier part for us to get right was around the communication piece and all kind of being pointed in the same direction from

a product perspective so while we grew remote from about the first 10 or 15 hires we then only grew locally through about the next 70 and then only recently started opening remote backup and I think now it's because we actually have enough bandwidth to create a really good remote culture where we're putting an emphasis On that documentation that communication piece and we're actually really able to hire folks who have been working remotely for years as well some really good advice given on this panel so far I think the trade offs are anything for you to

keep in mind the most I was thinking back towards contractors and I'll tell you a quick story of a contractor our first two hires were actually as I was thinking about it contractors one of them helped us Organize our bills and our like kind of office stuff and the other one helped us develop some technology someone I worked with when I used to work at Johns Hopkins and even though he was a contractor at first he actually you know we needed money to pay him because he was my friend so we paid him in equity

and he joined us as our first engineer eventually and helped us build some really core technology for our company and really Was a great addition to playing grid so I think that even though I wouldn't suggest hiring a contractor immediately we just did what was right for us at our company size and what we needed right then was like heli I need this kind of right some stuff for me and he'll take equity as payment wonderful because I don't got anything else to pay man so I mean I think just be open hearted with what

you think is needed for your business right then no there's a lot of Trade-offs with contractors the other thing to note is this was my friend we've had contractors have hired as well there's this ignorant feeling I can't even now I feel it when you hire a contractor that's like oh great I don't want to manage this person and that is so not true management of contractors is significant time for you as a founder that is not a free hire it sometimes is worse actually it's more management time for a Contractor than a full-time employee

so keep that in mind as well as the remote thing this is also an interesting topic when we started our company we tried to aim to have people in San Francisco but yet I realize actually as Kofi we split off for like months at a time we'd be like oh you know I'm gonna go back to Chicago for a month and then you know one of my co-founders to work out of Chicago for a month or two and that was totally cool as well we've gone back And forth on this over our eight years over

and over again between allowing remote not allowing remote allowing remote when it's a friend or a really good you know best in class I see that can really develop a lot of things now we're completely open as a company it doesn't matter we hire managers remote we hire local you know designers products everyone doesn't really matter we're just looking for the best talent we can get so I think the best advice You've gotten already is just do what's right for you at the size of your company and know that there's trade-offs with all these decisions

but there's no easy answer for any of these questions great okay I think we've got time for one or two questions from the audience questions questions one over there you okay I'm just going to repeat the question so everyone can can hear it um if you were going to start another company now what would you do to Accelerate the development process any takers yeah sure I mean I think we spent too much time building technology in my on my side to try to really get that product market fit as as soon as possible and it's

a challenge in terms of the areas that you pick because if you're in a kind of frontier tech space with like AR VR or this summer biotech those lead time could be really long and think about what kind of company you want to Build I'd say get in front of users all the time find your user base it's really it's actually a lot harder to build a product in this vacuum where you're just like in your head maybe your ID ating with your co-founder and you're like what if we do this what if we do

this these all sound great no this idea sounds bad but it's like way easier to just put it in front of somebody and have them say this is great or this sucks that feedback loop is invaluable Yeah echo the sentiment on users 100% get out in front of them start showing things to them start make sure your digging deep to make sure that you're really solving their problem I think if I were to start again today actually I'd build two or three versions which I just intended to throw away maybe this is biased on probably

prior experience but I think the fact that you can get reps and a very safe manner where then when you're really ready to Start you can just kind of go nuts building out the v1 I think is really powerful you know I love mobile development I love Swift I love Kotlin but I don't think I would choose to have to write for different platforms if I was going to write again I probably would have pick one of the cross-platform development libraries I don't know which is best they all you'd have to tell me if anyone

has like after this come your Strong opinions at which one of this is best they all seem to have trade-offs probably the best advice I could give though as I'd stopped coding a little bit earlier and I would start hiring a little bit earlier because I think there's a lot of me that wanted to passionately write those features and if I instead been passionately hiring other developers we would have you know we would have built faster I feel interesting okay one more question over There and your driving features given by the users what would be

the communication style get them in front of the users bringing them to the meetings get them over the phone well then give the questions you know kind of set it up a little bit where you're trying to bait the users into giving the feedback that you're looking for or you won't get the feedback you're looking for and maybe there's some adjustment there too so get them in front of users as much as you Can that's the most powerful tool you have my opinion okay thank you all so much this was great [Applause] you you