http://muratbuffalo.blogspot.com/2021/12/learning-technical-subject.html Skip to main content Search This Blog [ ] [Search] Metadata On distributed systems broadly defined and other curiosities. The opinions on this site are my own. Learning a technical subject * Get link * Facebook * Twitter * Pinterest * Email * Other Apps - December 18, 2021 I love learning. I wanted to write about how I learn, so I can analyze if there is a method to this madness. I will first talk about what my learning process looks like in abstract terms, and then I'll give an analogy to make things more concrete and visual. Learning is a messy process for me I know some very clear thinkers. They are very organized and methodical. I am not like that. These tidy thinkers seem to learn a new subject quickly (and effortlessly) by studying the rules of the subject and then deriving everything about that subject from that set of rules. They speak in precise statements and have clear and hard-set opinions about the subject. They seem to thrive most in theoretical subjects. In my observation those tidy learners are in the minority. Maybe the tidy thinkers are able to pull this feat off because they come from a neighboring domain/subject and map the context there to this subject quickly. But, again from my experience, it doesn't feel like that. It seems like they are wired that way. I am not wired that way. I am a messy learner. I yam what I yam. I need the context I am not able to learn by studying rules/principles and deriving everything from them. That shit only works for the tidy thinkers. Context is my crack, the same way rules/principles are the crack for the tidy thinkers. I need to know why, why not, surrounding, as much context as possible. The more context I know, the better I become able to locate something almost spatially, and the more I can make sense of it. Even reading the history and motivation for the subject can give my understanding a big boost. This is inefficient and slow. This is more of an inductive/intuitive method than a deductive one. I learn by making (mistakes) I need to wrestle with the subject myself. I need to grapple with it and roll on the floor. This comes in the form of asking questions, making guesses, trying things, and seeing them broken, and amending my understanding to make new hypotheses. This is not efficient learning, but that is not the goal most of the time. So I am not sure if efficient deductive learning is better than a gradual and more intuitive/inductive learning. In fact, not being able to learn all at once, and not being able to fit too many things in your head could be an advantage. It is like the Haruki Murakami quote about how non-gifted writers have to dig wells and get good at it. Similarly if you are unable to fit too many things at once in your head, you learn to decompose complexity to fit in your head and spatially/intuitively compartmentalize things in your brain. That can even make you a better and deeper learner than a rule-based learner. It turns out I had written about a similar thing 10 years ago: Falling short of superpowers (genius/giftedness), we invent processes /systems to succeed, and the good thing about these processes are they are replicable (unlike being genius/gifted). This is why I argue that explaining thought-processes are very important. The novelist Haruki Murakami comes to mind here. On his book "What I Talk About When I Talk About Running", he says roughly the following. Gifted writers write without effort; everywhere they touch in the ground the water pours. Other writers have to strive (he gives himself as an example); they have to learn to dig wells to get to the water. But when the water dries (inspiration leaves) for the gifted writer (which happens sooner or later), he becomes stuck and clueless because he has not trained for this. On the other hand, under the same situation, the other type of writer knows how to keep going and succeed. This is the reason we should teach our students/audience how to dig wells, and not to rely on occasional strike of insights for success. Re-reading that post, I think is fair to call Bourbaki-style mathematicians the tidy thinkers as the clothes fit: "[The Bourbaki project] changed the style of mathematics for the next fifty years, imposing a logical coherence that did not exist before, and moving the emphasis from concrete examples to abstract generalities. In the Bourbaki scheme of things, mathematics is the abstract structure included in the Bourbaki textbooks. What is not in the textbooks is not mathematics. Concrete examples, since they do not appear in the textbooks, are not mathematics. The Bourbaki program was the extreme expression of the Cartesian style. It narrowed the scope of mathematics by excluding the beautiful flowers that Baconian travelers might collect by the wayside." I need to revisit things to make sense of them and connect them to other things If you just understand how something works, you are missing out big time! You are missing out why it works. After understanding how something works, it is important to understand how this particular approach/design avoids mistakes and make things work well. For this you need to change things slightly to see why the variations would not work. This is what I observe when teaching students distributed algorithms. By just understanding how the algorithm works, they think they understand the subject. No, they are just a consumer/end-user of the algorithm, and they only have a superficial understanding of it. This is why I give TLA+ modeling assignments to get the students play with the algorithm, to see what breaks it, and understand better what makes it work. Only after all these grappling, you start to understand why that thing works. But you are still missing out. Is there a better way to restructure this, is there a different design to solve the problem better? This is where I step back at the meta plane, and start doing analysis and synthesis work. This is my way of getting closure on the subject and packing and compartmentalizing it for long-term internalized storage in my brain. The way I do this is by trying to teach it, and writing blog posts about it, and finally writing papers about it. This process of getting closure also involves a lot of learning by way of mistakes. It turns out, I had also written about this as well. In that down hour after teaching, what do I do? I have regret-flashbacks about some parts of the lecture I delivered. I have remorse about how I botched some parts of the lecture and how I couldn't answer a student's question better. The reason I botch up a part of the lecture is often because I haven't internalized that part yet. The reason I wasn't able to make a smooth transition from one part to another part is often because I didn't have a better understanding/insight about how those parts fit. It is likely I was missing some context about it. Or maybe more likely that I knew the context, but I didn't internalize it well enough. There are many different levels of knowing, and the more reliable/resilient/ dependable level of knowing is knowing by experience. Teaching is a way to have hands-on training in your research area. You learn through your regrets about how you couldn't explain it better. The indication that you have internalized your learning sufficiently is that you can teach it well. At this point you not only know why the way things are, but you also have a good sense of what other ways would fail to work and what other ways will work better. This comes from experience and making the mistakes. Negative results are as important as positive results. When you level up, you also notice what mistakes the current approach has, and you are able to suggest better designs/solutions. "What I cannot create, I do not understand." --Feynman Here is another gem from an earlier blog post: You truly master a [subject only when you master yourself and bend yourself with that [subject]'s thinking mode. Mastering the [subject] and yourself through the [subject], takes many weeks, and sometimes years. When you pay your dues and go deep, only then, you transcend and level up. Exploring a city analogy Suppose you are beamed up to a foreign city in another galaxy and you have to learn your way around. Why is there not a map? Oh, there are actually many different maps and books written by the locals. But these groups speak all different languages. You cannot understand the ledgers in these maps. No one uses standard lingo, there are a lot of technical (often inconsistent /incompatible) jargon in the maps/guides because this is a complex/ complicated city terrain. Moreover, the maps/guides are incomplete due to uncharted territory, not enough details, not enough instructions. In your quest to master the city, you go through different roles: spectator, participant, citizen, and architect. Initially you are a spectator. You follow your roommate while he walks around between home and work. You look at everything with confusion and awe. Nothing makes sense, and everything looks foreign. Then you become a participant. You learn to walk between home, park, and the cornerstone by using landmarks you identified. You get lost a couple times. Good, those are learning opportunities. Then you become a practitioner/citizen of the city. You learn your way around, and you start working. You are now contributing to the city, even starting to do some community building maybe. At this level, you start to notice larger blocks of the city. You start to develop a higher-level view of the city. Now those maps finally started making sense. New suburbs/towns of the city now open up to you. You start discovering highways between the suburbs. (Oh, that obscure dashed line in the map is this fast highway!) But you also go low-level. You start noticing ornaments on building tops, the different architectural styles, even different fonts in shop signs, and which eras they were built and how they interplay with each other. You start to recognize the 3 groups of people and become conversant in their languages as well. This is when things become intuitive as you internalized things thanks to all that context you gathered. Now, you know the culture, and start to enjoy the hip places and the city as a whole. You start to contribute to the city more and more. You now develop a taste for the city. And slowly you will realize, for certain places, doing things another way would actually work better. That means you are becoming an architect. You start building things. Maybe you first start-up a corner-convenience store, then you build a park, a canal, and ultimately a new neighborhood in the city. When you become an architect, you publish your own map. You also use different ledgers, you have to. The city is sophisticated/ complicated, and you need the jargon/symbols to describe things. Hopefully, you have not forgotten about the lessons you learned as a starter, and you are emphatetic, so your legends are easier to follow. But when another stranger arrives in the city, as helpful as your map and guide is, they need to learn their way around themselves inevitably. Of course, you will give them a hand and provide enough scaffolding to learn safely. Having/cultivating the following attitudes help you navigate this process better: * curiosity: being adventurous, asking a lot of questions, and not being content with your own corner helps the most. * relentlessness: not giving up, and putting in the hard physical and emotional work. Your success in this endeavor depends on emotional management for the most part. * being a hands-on maker: not just being a tourist or spectator. You should get involved, you should walk the streets and participate in the bazaar. * being social, learning to communicate well: for this you should cultivate good written and verbal communication skills. To level up to an architect, you will need to do some community organizing. * analyzing and drawing lessons: this is how you make sense of things and learn meta-lessons. * leveraging previous experience: If you have visited/explored many cities before, you can transfer some of your experiences (at least the emotional and attitude management), and learning this new city becomes easier. misc research-advice * Get link * Facebook * Twitter * Pinterest * Email * Other Apps Comments [blo] As i see it said... Such an insightful post and brilliant analogy. Thank you for writing this December 24, 2021 at 12:02 AM [icon_delet] Post a Comment Popular posts from this blog Foundational distributed systems papers - February 27, 2021 I talked about the importance of reading foundational papers last week. To followup, here is my compilation of foundational papers in the distributed systems area. (I focused on the core distributed systems area, and did not cover networking, security, distributed ledgers, verification work etc. I even left out distributed transactions, I hope to cover them at a later date.) I classified the papers by subject, and listed them in chronological order. I also listed expository papers and blog posts at the end of each section. Time and State in Distributed Systems Time, Clocks, and the Ordering of Events in a Distributed System. Leslie Lamport, Commn. of the ACM, 1978. Distributed Snapshots: Determining Global States of a Distributed System. K. Mani Chandy Leslie Lamport, ACM Transactions on Computer Systems, 1985. Virtual Time and Global States of Distributed Systems. Mattern, F. 1988. Expository papers and blog posts There is No Now . Justin Sheehy, ACM Queue 2015 Why Logical Clock Read more Graviton2 and Graviton3 - December 04, 2021 Image What do modern cloud workloads look like? And what does that have to do with new chip designs? I found these gems in Peter DeSantis's ReInvent20 and ReInvent21 talks. These talks are very informative and educational. Me likey! The speakers at ReInvent are not just introducing new products/services, but they are also explaining the thought processes behind them. To come up with this summary, I edited the YouTube video transcripts slightly (mostly shortening it). The presentation narratives have been really well planned, so this makes a good read I think. Graviton2 This part is from the ReInvent2020 talk from Peter DeSantis. Graviton2 is the best performing general purpose processor in our cloud by a wide margin. It also offers significantly lower cost. And it's also the most power efficient processor we've ever deployed. Our plan was to build a processor that was optimized for AWS and modern cloud workloads. But, what do modern cloud workloads look like? Let's start by Read more Your attitude determines your success - March 13, 2021 This may sound like a cliche your dad used to tell, but after many years of going through new areas, ventures, and careers, I find this to be the most underrated career advice. This is the number one advice I would like my kids to internalize as they grow up. This is the most important idea I would like every one undertaking a new venture to know. If you think you are not good enough, it becomes a self-fulfilling prophecy. If you think you are not enjoying something, you start to hate it. I gave examples of this several times before. Let's suffice with this one : In graduate school, I had read "Hackers: Heroes of the Computer Revolution" from Steven Levy and enjoyed it a lot. (I still keep the dog eared paper copy with affection.) So, I should have read Steven Levy's Crypto book a long time ago. But for some reason, I didn't...even though I was aware of the book. I guess that was due to a stupid quirk of mine; I had some aversion to the security/cryptography res Read more Progress beats perfect - August 06, 2021 Image This is a favorite saying of mine. I use it to motivate myself when I feel disheartened about how much I have to learn and improve. If I do a little every day or every week, I will get there. If I get one percent better each day for one year, I'll end up thirty-seven times better by the end of the year. $1.01^{365}=37.78$ Years ago I had read this idea in one of John Ousterhouts life lessons, and it stuck with me. "A little bit of slope makes up for a lot of y-intercept" Recently I noticed another advantage of progress over perfect. The emotional advantage. Progress is better because it makes you feel better as you see improvement. You are getting there, you are making ... progress. Progress is growth mindset . You have an opportunity ahead of you. Perfect feels bad.. It puts you on defense. You have to defend the perfect, you have to keep the appearances. You can only go downwards from perfect, or maintain status quo. Progress gives you momentum. As long as you manag Read more Cores that don't count - June 06, 2021 This paper is from Google and appeared at HotOS 2021 . There is also a very nice 10 minute video presentation for it. So Google found fail-silent Corruption Execution Errors (CEEs) at CPU/cores. This is interesting because we thought tested CPUs do not have logic errors, and if they had an error it would be a fail-stop or at least fail-noisy hardware errors triggering machine checks. Previously we had known about fail-silent storage and network errors due to bit flips, but the CEEs are new because they are computation errors. While it is easy to detect data corruption due to bit flips, it is hard to detect CEEs because they are rare and require expensive methods to detect/correct in real-time. What are the causes of CEEs? This is mostly due to ever-smaller feature sizes that push closer to the limits of CMOS scaling, coupled with ever-increasing complexity in architectural design. Together, these create new challenges for the verification methods that chip makers use to detect diverse Read more Learning about distributed systems: where to start? - June 10, 2020 This is definitely not a "learn distributed systems in 21 days" post. I recommend a principled, from the foundations-up, studying of distributed systems, which will take a good three months in the first pass, and many more months to build competence after that. If you are practical and coding oriented you may not like my advice much. You may object saying, "Shouldn't I learn distributed systems with coding and hands on? Why can I not get started by deploying a Hadoop cluster, or studying the Raft code." I think that is the wrong way to go about learning distributed systems, because seeing similar code and programming language constructs will make you think this is familiar territory, and will give you a false sense of security. But, nothing can be further from the truth. Distributed systems need radically different software than centralized systems do. --A. Tannenbaum This quotation is literally the first sentence in my distributed systems syllabus. Inst Read more Silent data corruptions at scale - June 12, 2021 Image This paper from Facebook (Arxiv Feb 2021) is referred in the Google fail-silent Corruption Execution Errors (CEEs) paper as the most related work. Both papers discuss the same phenomenon, and say that we need to update our belief about quality-tested CPUs not having logic errors, and that if they had an error it would be a fail-stop or at least fail-noisy hardware errors triggering machine checks. This paper provides an account of how Facebook have observed CEEs over several years. After running a wide range of silent error test scenarios across 100K machines, they found that 100s of CPUs are identified as having these errors, showing that CEEs are a systemic issue across generations. This paper, as the Google paper, does not name specific vendor or chipset types. Also the ~1/1000 ratio reported here matches the ~1/1000 mercurial core ratio that the Google paper reports. The paper claims that silent data corruptions can occur due to device characteristics and are repeatable at scale Read more Paper review. Sharding the Shards: Managing Datastore Locality at Scale with Akkio - February 14, 2019 Image This paper by Facebook, which appeared in OSDI'18, describes the data locality management service, Akkio. Akkio has been in production use at Facebook since 2014. It manages over 100PB of data, and processes over 10 million data accesses per second. Why do we need to manage locality? Replicating all data to all datacenters is difficult to justify economically (due to the extra storage and WAN networking costs) when acceptable durability and request serving latency could be achieved with 3 replicas. It looks like Facebook had been doing full replication (at least for ViewState and AccessState applications discussed in the evaluation) to all the 6 datacenters back-in-the-day, but as the operation and the number of datacenters grew, this became untenable. So, let's find suitable home-bases for data, instead of fully replicating it to all datacenters. But the problem is access locality is not static. What was a good location/ configuration for the data ceases to become suita Read more Sundial: Fault-tolerant Clock Synchronization for Datacenters - March 21, 2021 Image This paper appeared recently in OSDI 2020 . This paper is about clock synchronization in the data center. I presented this paper for our distributed systems zoom meeting group . I took a wider view of the problem by explaining time synchronization challenges and fundamental techniques to achieve precise time synchronization. I will take the same path in this post as well. It is a bit circuitous road, but it gives a scenic pleasurable journey. So let's get going. The benefits of better time synchronization For any distributed system, timestamping and ordering of events is a very important thing. Processes in a distributed system run concurrently without knowing what the other processes are doing at the moment. Processes learn about each other's states only by sending and receiving messages and this information by definition come from the past state of the nodes. The process needs to compose the coherent view of the system from these messages and all the while the system is movi Read more Powered by Blogger Theme images by Michael Elkan Murat Demirbas My photo Murat I am a principal applied scientist at AWS S3 Automated Reasoning Group. On leave as a computer science and engineering professor at SUNY Buffalo. I work on distributed systems, distributed consensus, and cloud computing. You can follow me on Twitter. Visit profile Recent Posts * 2022 4 + February 1 + January 3 * 2021 47 + December 5 o Best of metadata in 2021 o Learning a technical subject o A read-only transaction anomaly under snapshot iso... o Humans of Computer Systems: Ted o Graviton2 and Graviton3 + November 3 + October 6 + September 1 + August 4 + July 2 + June 12 + May 1 + April 1 + March 4 + February 4 + January 4 * 2020 76 + December 3 + November 7 + October 4 + September 1 + August 3 + July 6 + June 11 + May 9 + April 8 + March 8 + February 7 + January 9 * 2019 65 + December 10 + November 14 + October 6 + September 13 + July 3 + June 3 + May 4 + April 6 + March 2 + February 1 + January 3 * 2018 71 + December 4 + November 7 + October 2 + September 2 + August 8 + July 2 + June 4 + May 9 + April 6 + March 9 + February 5 + January 13 * 2017 77 + December 15 + November 15 + October 5 + September 8 + August 10 + July 3 + June 3 + May 3 + April 4 + February 4 + January 7 * 2016 42 + December 7 + November 9 + October 3 + September 1 + July 4 + June 5 + May 1 + April 4 + March 2 + February 2 + January 4 * 2015 34 + December 3 + November 2 + October 3 + September 2 + August 3 + June 1 + May 1 + April 6 + March 6 + February 4 + January 3 * 2014 29 + November 4 + October 4 + September 6 + August 2 + July 2 + June 3 + March 3 + February 4 + January 1 * 2013 25 + December 1 + November 2 + August 2 + July 4 + June 2 + May 5 + April 8 + January 1 * 2012 18 + December 1 + November 7 + October 1 + September 2 + August 1 + May 2 + March 1 + February 1 + January 2 * 2011 38 + December 3 + September 5 + June 1 + May 5 + April 5 + March 5 + February 9 + January 5 * 2010 31 + December 6 + November 9 + October 9 + September 7 * 2007 1 + August 1 Show more Show less Topics auditability5 automated reasoning5 Azure9 bestof5 big-data20 Blockchain39 book-review49 chaos2 cloud computing2 consistency23 Cosmos DB10 CosmosDB11 databases2 dataflow7 distributed consensus41 distributed transactions5 facebook14 failures16 fault-tolerance35 formal methods10 graph-processing1 humans10 indexing3 links2 mad-questions42 misc101 mlbegin7 mldl25 mobile2 my advice15 my-paper 10 newsql1 paper-review133 paxos43 presenting4 programming5 reading-group22 research-advice45 research-question43 Rust2 scheduling3 seminar9 serverless1 smartphones2 sonification1 stabilization5 stream-processing10 teaching30 tensorflow11 time8 time synchronization1 tla38 trip-report20 wpaxos5 writing26 Show more Show less Pageviews