harisrid Tech News

Spanning across many domains – data, systems, algorithms, and personal

recent posts

about

Graduated from the University of Texas, Austin’ 2018 Bachelors in computer science and bachelors in mathematics. Currently living in beautiful sunny San Francisco, CA, where the skies are clear and the grass is greener! Outside of work, I like to hike, explore new restaurants, go boxing, spend quality time with friends, and travel the world!
  • PERSONAL – My Leetcode Journey: How Persistency Transform Skills

    PERSONAL – My Leetcode Journey: How Persistency Transform Skills

    Persistency. Persistency matters. – a wizened engineering manager.

    Hi all 🙂 !!!! Just like you, I’m painfully average.

    I briefly want to talk about my Leetcode journey, and how I drastically improved and got better at the skill ( to the point of having solved 1000+ LC MEDIUMS and an almost 100 out HARDS, which BTW, is an ungodly number to solve ). To be forthcoming and transparent, I don’t think of myself as the world’s best natural problem-solver – I’m not a USACO/AIME/Putnam medalist, I never did competitive programming ( outside of an intro to competitive programming class ), I got a B- in my undergraduate algorithms, and there was a time where dynamic programming made zero sense to me.

    Yep! Turns out, I’m probably closer to downright painfully average. I used to struggle immensely with the puzzle-solving skills and the algorithmic-thinking needed to cut through the HARDEST of problems. But somehow, over time, things started to “click”, and I’ve gotten better.

    Figure : Dated on 03/28/2025 – number of problems solved

    Days Past – the Practice Routines

    My practice routine is simple. On a daily basis, following a block schedule, for 30-minutes a day in the morning ( e.g. 7:00 a.m. – 7:30 a.m. or 8:30 a.m. – 9:00 a.m. ), I practiced a single problem. Somedays, I had more luck where I solved the problem in fewer than 20 minutes ; on those days, I solutioned a second problem 🙂 .

    The practice routine also helped because I avoided the major pitfall that most other folks run into – CRAMMING. I know there’s people out there who can practice and code algorithmic problems for 2-3 hours straight – some an ungodly continuous 8-12 hours. Yes, there’s times that I can enter “the flow state”, but trust me, I am not a person who practices LC problems for more than even a consecutive two hours on end ( save for those occasional problems where I really had to invest my mental faculties and I obsessed over them; there was one where I once spent two 2-hour sessions on ). Nope, I’m a persistence hunter – a long-distance marathon runner – and where I excel at is showing up and being consistent. Because getting started is the hardest part.

    Figure 0 : Example visual and practice of a short leetcode problem.

    Back in the day, I used to practice by solving two problem categories – EASY and MEDIUM. I started off by solving the EASY problem, and then moved over to solving the MEDIUM problem. Solving the EASY problems helped tremendously – it got me into the groove and the momentum of problem-solving. It gave me motivation, because at least I could tell myself “Hey, even if I failed today’s MEDIUM problem, I solved today’s EASY problem”! I preferred the days where I solutioned a problem, versus solutioning nothing – it was more motivating. ( well not only that, but there’s only so many EASY problems online, so at some mathematical point, once they’re decently exhausted, there’s no choice but to “level up” and move to the MEDIUMS. You’ll get forced to level up because you will likely hit a “saturation point” ).

    As for MEDIUMS problems, I conjured up a “mental roadmap” in my head for how to tackle them. I focused on practicing problems by clusters or by categories. Some weeks, I focused on solving five SLL ( singly-linked list ) problems. Another week, I shifted to solving trees. The third week, recursion. I rotated across categories, which I think helped because problem-solving techniques within categories translated well across each other; solving one SLL problem helped me solve another SLL problem.

    “Giving Up” – But The Smart Way

    Another street-fighting tactic was learning “when to give up”. And I don’t mean give up by “all is night and admit defeat, All-Mighty Warrior“, but rather, give up as in “Hey, go circle back to this problem a few days or weeks later, you’ll crack it!”. I wish I could explain it better, but I also don’t think I can. There’s something subconscious taking place when we’re awake and when we’re asleep. I mean, humans think about problems at random times – when we’re showering, eating lunch, driving, hiking, talking to friends, or at the movies. I had moments where I spent 30 minutes on a MEDIUM and “gave up”, but then a couple days later, “kaboom” – I saw what to do. This also happened faster when I went and solved other problems in the meantime – I think solving those problems gave me additional problem-solving experience or different frames to view a former un-solvable problem.

    So instead of having a day where I would go “Oh my god I can’t solve a single MEDIUM today, I give up”, I cognitively reframed and strategized differently. I said to myself “Hey, let’s put those problems in the backburner and tackle something else”. And that tackling – of the something else – helped.

    Figure 1 : What does my multi-colored problem-solving look like.

    Cluster-Based Practice

    I also recognized that clustering practice based on the data structures and algorithms helped me. For example, one intuitive progress was “singly-linked lists” -> “trees” -> “graphs”. Which makes sense, because a SLL is a 1-D tree in the hiding and a tree is a graph in the hiding ( with n vertices and n-1 edges ). I noticed another intuitive progression with “recursion” => “top-down memorization ( TDM )” => “bottom-up dynamic programming ( BUDP ) “. In this scenario, I recognized that solving BUDP immediately would be HARD, but writing up the naive recursive solution and then converting that to a TDM-based approach made more sense! I could still pass those DP problems within reasonable time-limits without going down a far more difficult path.

    Pen-and-Paper To The Rescue

    And the last time, which I’m surprised, is to use pen-and-paper. I have no explanations of this either, but it helps me to diagram and think about problems using a pen. There’s some people out there who can think in their heads and type out directly ( I’ve gotten to this point, but only because of sheer practice ). It also helps to use multiple colors – not just one – to flesh out patterns or highlight different parts of problem ( I should share an image of what this looks naturally looks like ) . I personally don’t understand people who use a single black color market to solve problems – I like to see the kaleidoscope : blue, green, red, black, purple, brown, and whatever else.

    Figure 2 : The pens I use to solve problems” ( turns out a kid’s multi-colored pen does the job )

    My Takeaways

    I’m now at a point where I’ve become far to experienced – a maestro, I should say – at this oddly specific niche skill. And I need to do something with it – personally, I think it is pointless to keep all the knowledge and the tactics that I’ve learnt to myself. It would entail a slow transition towards teaching and writing, but, I think I can make some use out of this stuff or do something with it 🙂 !

    Wish me luck to this !!!

  • TOTW/4 – Effective Planning ( and Pre-Planning ) – Writing up your Features, your Stories, and your Tasks.

    TOTW/4 – Effective Planning ( and Pre-Planning ) – Writing up your Features, your Stories, and your Tasks.

    There’s a lot of work involved in pre-planning – a wise manager.

    An Intro

    Alright, let’s talk about something that we don’t really teach you in undergrad days, but it’s an industry expectation – feature planning. For some folks, it’s a harder skill to learn, and for others, it’s easier. And it’s not just feature planning – it’s all the other surrounding skills – estimating points, figuring out dependencies, and breaking down ambiguous, ill-defined work into smaller, concrete, more manageable chunks.

    So how do we do this? In fact, I’m still getting better at this skill, because feature complexity increases in complexity with level complexity – the complexity of senior-staff engineer’s features typically eclipses that of a mid-level engineer’s.

    But what makes feature planning so difficult?

    • Ambiguity – features get harder to develop.
    • You need to meet people – no one takes features all by themselves. You need to set up meetings and communicate to multiple organization personas
    • Timelines – multi-quarter features are harder to break down ( versus two-week deliverables ).

    Tell me the skills I need!

    • Task Decomposition – breaks down a large task into smaller tasks. Breaking down makes estimating timelines easier – a three day task is easier to estimate than a three-week task.
    • Task initiation – can start on a task.
    • Points estimation – estimates complexity with mostly accurate precision.
    • Establishing Priorities – can context switch and work on other features if they’re more pressing
    • Work Delegation – can create multiple stories and tasks and delegate them off to more junior engineers; place trust that other engineers can execute with autonomy.

    Can you share an example User Story with a Tasks Breakdown?

    Rock star senior engineer Archit works on a feature and needs to set up an ETL pipeline from an OLTP Database ( e.g. DDB ) to an OLAP Database ( RocksetDB ) for an up-and-coming reporting & analytics requirement. It’s a multi-week long effort ( at least one-quarter ), and Archit is on the design stage.

    Archit needs to write up a story about being on the design stage, but is struggling because “design” , in itself, is an ambiguous term. User story title “Design an ETL Pipeline from DDB to RocksetDB.”. Ok, well, it’s concise, but not granular enough.

    What are some tasks that we can introduce in the design stage?

    • [1] Gather business requirements – functional and non-functional
    • [2] Identify access requirements, caveats, dependencies, and major blockers.
    • [3] Brainstorm, get initial ideas and proposals, and identify best candidates
    • [4] Identify technologies and mechanisms to make design changes happen
    • [5] Write up a first-draft version of the FRD 1
    • [6] Write up final-draft version of the FRD

    The tasks don’t have to be followed in a rigid, linear order, but the ordering specified above breaks down the steps.

    Each task helps sets the agenda for meetings. E.g. for task #1, a 30 minute – 60 minute XFN ( cross-functional ) meeting can be coordinated with engineers across teams. Task #2 instructs the need for future meeting with stakeholders or L&M 2to adjust deadlines according to identified blockers. The tasks also introduce structure & scaffolding – if the design stage extends longer, we can create phase-based stories and migrate the tasks around – phase#1 can group tasks (1-3) and phase #2 can group tasks (4-6).

    Give me a second example of a user story!

    Alright, so Archit wraps up the (1) design stage and (2) the development stage of his works. His next tasks are to work on (3) the testing stage and (4) the deployment stage. Both make for solid standalone stories. How can we set them up?

    His user story title – [Testing] Write up tests for the latest feature – is short and concise : good for both technical audiences and non-technical audiences.

    But what are the tasks? Can we track tasks to view completion? What are good tasks to introduce in the testing stage?

    • [1] Write unit tests for the feature and get unit tests to pass in local env
    • [2] Trigger a feature build – have unit tests pass in CI/CD pipelines.
    • [3] Write integration tests that pass in non-PROD env.
    • [4] Write integration tests that pass in PROD env.
    • [5] Write end-to-end tests that pass in non-PROD env.
    • [6] Write end-to-end tests that pass in PROD env.

    The benefits of this breakdown are :

    • Dependency identification – we can create an abstract “DAG” ( Direct Acyclic Graph ) in our head and identify dependency sequence across tasks. In this case, we can see that task [1] has to be done before task [2] and task [2] has to be done before tasks [3-5].
    • Task Isolation – granularity helps us isolating and break down tasks. Ok, can we break down a deliverable by different conditions – by environment, by testing type, and by involvement of build-pipelines?
    • Facilitate trackability – a testing story can take a sprint ( or two ), and tasks can take their corresponding number of days. This helps your product owners and leadership & management get a better view of what’s going on.

    Wait but what if I don’t actually need to write integration or end-to-end tests? Well, that’s fine. We can remove them. The boilerplate structure above helps us understand an initial scope ( to do effective estimation ), but we can simplify further.

    Street-Fighting Tips

    I learned these tips from others, such as direct managers or PMS, so credit is due to them as well! They are not all originals.

    • Phase-based development – if a feature takes multiple sprints or quarters, can we introduce phases ( e.g. phase #1, phase #2, phase #3 )?
    • Scope down – it’s tempting to do everything “under the sun”, but that’s unrealistic. In a given Sprint ( or other timeline ), expect to primarily focus on 1-2 major deliverables ( or goals ) at a time.
    • SDLC Tagging – make it clear. Are we in design, development, testing, or deployment? If we’re given a feature we make user stories, can we add tags to the stories ? It’ll help others not only track your work, but categorize your stories in the future ( e.g. list all design stories or development stories ).
    • Track dependencies – does F3 have a dependency on F1 or on F2? This instructs on deadline changes.

    Terms and Definitions

    1. FRD – Functional Requirements Document. Where developers establish the scope and gather functional requirements and non-functional requirements from a top-down level view. ↩︎
    2. L&M – Leadership & Management. ↩︎

  • BEHAVIORAL – Tell me your personal story.

    BEHAVIORAL – Tell me your personal story.

    What made you a software engineer? How did you end up in your profession?

    Hmmm that’s an excellent question, and it’s something that’s been asked of me a couple of times by friends or by behavioral interviewing. I usually give a short answer, but I think it’s deserved to narrate and tell y’all a good story.

    But where do we even begin !!!

    Well, we first need to unwind the hands of the metaphorical clock and go back years ago. Harkening back to days past and looking into what I spent my time on as a kid.

    I’m visually transported to first grade in Mrs. Knapp’s class, where I’m introduced to a website – CoolMath4Kids ( https://www.coolmath4kids.com/ ) . I was enamored with this! My god, the joy of problem-solving and learning addition and subtraction with single-digit cases ( 9+8 = 17 ) and double-digit cases ( 17+11 = 28 ) . There were three young boys who spent time on the website, and somehow, I was the “ace of my class” ( of 30-kids or so ) :-P. I won a blue ribbon for this, ( as my parents like to praise around ) ! This is the beginning, where a young boy gets introduced to the fascinating world of numbers, patterns, and relationships.

    Mathematical thinking and puzzle solving remains a fixed constant ( pun intended 🙂 ! ). It’s fourth grade and I’m learning long division. It’s a summer in fifth grade – back in balmy Bombay, India when the monsoon rains are about to pour down – where he lists out conversions between Fahrenheit and Celsius for numbers from 0 – 100 for Celsius ( formula being F = C * (9 / 5) + 32 ) It’s also the morning bus rides of sixth grade, where I’m devouring my older brother’s pre-algebra textbook notes, learning concepts such as graphical inequalities, the volume of 3-dimension shapes ( A sphere’s volume = 4/3*pi*r^cubed ), and linear equations ( x + 3 = 5, solve for x ) . I place out of pre-algebra and skip right on to algebra.

    He spends his time solving puzzles and games for fun. He sometimes sits next to his dad during middle school orchestra practice, where he solves Sudoku problems, spending time filling in the single digits 1-9 in the 9x9x9 square boxes.

    He’s also an active participant in his school’s Math24 Competitions. He spends a lot of time practicing and memorizing different combinations for those card decks . He’s spends that time in the backseat of his parent’s car and when he’s at home and doesn’t have much else to do.

    Every year – sixth grade, seventh grade, and eight grade – he takes the lead and dominates the competition. He claims his rightful thrown as the first-place winner at both his school level and at his regional county level ( there’s no state level or national level – or maybe there was, but I was blissfully unaware 😛 ). WTG – Way To Go !

    The Math24 Competitions Deck : A Three-Point System Game that helps kids build their math skills.

    Now it’s my first-year of high school, and I get introduced to the world of computer science in a summer school class : Intro to Computer Science. I prep some of the assignments ahead of time, and somehow, things just click and magically fall into place. You can say that the thinking of loops, conditional logic, and variables really fitted well into my abstract reasoning skills. To me, it didn’t require a gargantuan effort to go from x = 3; y = 9; ans = x + y ; print(ans) to recognize the similarities to pre-algebra systems of equations ( the answer is 12 ). From intro to CS, comes other classes like AP Computer Science and a few special offerings from my school.

    But what about the other majors? There’s biology and chemistry – which is cool. Hey, science! But there’s not much mathematical thinking there, so it’s out.

    Ok, what about English, History. There’s writing and reading. Sure, I love to read and write – in fact, I still miss the feeling of devouring books and reading novels and old literature as a kid. But they are not my “forte skills”. I can give you my SAT scores as evidence ( e.g. my SAT Math being higher than my SAT reading or my SAT writing ), but let me set the record straight : I personally think the SAT, the ACT’s, and most forms of standardized testing are atrocious proxies for intellectual capability. Maybe it’s better that I can just tell you that “I know myself.“

    What about finance or economics? Ok, it’s interesting, but there’s not as much mathematical or computational thinking there.

    By process of eliminate, the choices become more crystal-clear and better known. Alright, young Hari needs to major in computer science and mathematics and pursue something along those disciplines.

    So he heads off the college and takes classes like Discrete Mathematics or Data Structures and Algorithms. He also takes classes like Introduction to Numerical Methods and Physical Simulation – here, he works through challenging modeling and simulation problems to model the movement of entities over time. He also does internships at world-renowned companies for three summers : Intel and Amazon.

    And somehow, he ends up spending more and more time nested deep ( pun intended again, IYKYK ) in programming and software engineering.

  • PERSONAL – Lemme talk about what’s it’s like being a senior software engineer

    PERSONAL – Lemme talk about what’s it’s like being a senior software engineer

    Tell me what’s your favorite part about working as a senior software engineer?

    Tell me what’s your favorite part about working as a senior software engineer?

    That’s a really good question.

    I derive tremendous satisfaction from collaboration and working with more senior talent than me. Even with my years of professional working experience, I still take pride in being a mentee and working with strongly experienced individuals.

    I admire the calm, composed personalities and demeanors of the older folks on my team – their experience level, their communication skills, and their 1:1 or group coaching skills shows up. When I go into a meeting with a person with years of experience, I can feel their presence – their aura – “light up a room” ( what some might label “executive prescence“) .

    Even during times of a crisis or a pressing deadline, they demonstrate optimism and steadfastness; they navigate organizational blockades with relative ease, they see how to scope down complex products to simpler deliverables, and they still meet end customer expectations. Basically, you can say that “they know what they are doing”.

    Can you identify what makes parts of your job challenging ( for you )? Give me an example of one or two things that makes your job hard for you? What aspects of a job do you find harder than others?

    That’s an insightful question. Lately, my biggest challenge has been figuring out future directions and scoping out a list of features.

    When I was a junior engineer, figuring out what to do was relatively straightforward – a more senior engineer created the tickets, assigned me work, and instructed me how to code up the feature ( with hints at helpful parts of a codebase or code structures to look at ).

    But now, I’m more senior : my responsibility is less about writing code. Now I’m tasking out and figuring out future direction and delegating work to others. Harder to do, for a plenitude of reasons : customer requirements frequently shift, the future is an unknown, and hindsight is 20/20.

    Sometimes, what I task out isn’t “always correct”. Tasks get scoped out to different time periods or planning sessions.

    I’ve had experiences where I’ve written up a design document for a potential multi-quarter organizational effort, only to realize that we as an organization didn’t need to do that work ( or we had to do that work, but in a completely different manner ).

    I’ve also had the experience where I did make the right stories, but the work has to be done three months or six months later, when dependencies have been resolved and feature releases are productionized, stable, and ready.

    Was my work wasted? Absolutely not. As a senior staff engineer put it, most of what we do always makes for good learning experiences. The level of cognitive thinking and deliberate thought put into such endeavors never leaves – it stays.

    What else makes being a senior software engineer challenging?

    Hmm … let me itemize them. There’s challenges that both myself ( and others I’ve talked to ) have identified :

    • Delegating and Trusting others – the more senior you get, the less time you get to spend on low-level code details. Sure you may review code and provide your input on some technical suggestions, but that’s about it. Outside of this very myopic scope, you have less visibility into your codebases. You’ll have to trust and rely more on your junior engineers to mid-level engineers to make the right choices. Release your control.
    • You’re both a manager – and you’re not, a manager – there’s an implicitly-spoken expectation or rule that senior-plus individuals need to demonstrate some form of “leadership”, even if it’s “informal”. BUT, you’re also not a manager. You can’t exactly “boss” around other engineers ( as much as you’d like to ) or instruct engineers on what they should or they shouldn’t be doing, BUT, you can expect to see yourself take on a more advisory/mentorship-esque role where you’ll provide a more informal form of guidance.
    • Scope Creation = you can’t just expect your leadership & management to always create scope ( e.g. your project complexity level/organizational impact ). Onus rests on you ! And you’re not just creating entry-level engineering scope ( e.g. a one-week refactor ticket ). If you’re say, staff+, you need to help identify scope for a senior engineer ( e.g. can you identify long-term organizational quarter-long features or design task and get a senior engineer to collaborate and lead the efforts with others )?
    • Teaching and training skills – Teaching is part of the role, and you’ll have to teach skills across different areas – coding, design, some Linux commands, technical writing, and diagramming. Everyone comes in with different backgrounds, understandings, and nomenclatures, so identifying what to teach ( and how to tailor your approach ) is an endeavor. Not only that, but mentorship is a hard skill – learning skills is a lot easier than teaching skills. Some of my more difficult instructive moments have been less about coding, and more about working through tasks or designing systems – I had that conversation one day with a junior engineer where I had to teach concepts on automated testing, API endpoint versioning, and the metastable failure problem of distributed systems. These were enlightening moments, because I was backfilling on design thinking – skills that would serve him powerfully in his long-term future.
    • Develop or Deploy – Pick One – How do you build an application, within a deadline and a tightly constrained environment, that satisfies your end customers needs? But at the same time, you make sure to pre-empt a business crisis and build the scaffolding & structures needed – security, rate limiting, auto-scaling, and performance? It’s tempting to want to keep on building, but there’s a point where engineering talent has to band together and say “Hey, let’s sign-off on the feature release”. At some point, we have to be pragmatic and content with what we have.

    What part of your jobs do you derive less satisfaction much? If you could delegate a task off, what would you delegate?

    I struggle tremendously with working on tasks related to setting up a local development environment or debugging environment set ups. Like a lot of engineers, I like it when I can immediately “hit the road going” and start writing change lists or productionizing features. Setting up and configuring a development environment can take a few hours, spread across a couple of days. Nonetheless, it’s a useful skill. And I want to go with the belief that the more often someone sets up an environment, the less work they have to invest spinning up and setting up environments.

    What gave you the most surprise about your role?

    The biggest revelation ( to me, that is ) is that more you “level up”, the less you code, and the more you communicate and talk to other people.

    You’d think as a junior engineer that your senior engineers write more lines of code or work through more technical work.

    On the contrary, I find that senior engineers are less engaged in code writing. They’re more engaged in all the other tasks that support an organization – reviewing code, menotrship, designing systems, communicating with stakeholders, or technical documentation. You could say that they do everything else to “make the world spin”.

    This transition can be hard. Most of us came into our profession as passionate coders ; I’d imagine a lot of us secretly want to spend a six-hour day by ourselves on a weekend ironing out a mobile application or a game.

    But I’d also argue that this is a good thing. There’s a joke that the best code is no code at all ( see https://blog.codinghorror.com/the-best-code-is-no-code-at-all/ ), and there’s truth to that ; any code you write is error-prone and bug-prone in run-time. But there’s also truth that the more you code, the less you write, because you know how to take efficient paths and shortcuts and you also know how to express your coding thoughts more concisely.

    What piece of advice would you give to a junior or someone starting a new?

    Learn the value of listening more and speaking less.

    Yes I know you feel those feelings of anxiety of “If I don’t speak.” or “If I don’t say something for five minutes in a thirty-minute conversation, I’m no good”.

    No this isn’t true. Actually, I want you to sometimes imagine you’re a student in a college-lecture style setting, where you need to listen attentively to your instructor.

    Because even if you’re not speaking, you’re doing other things.

    You’re listening. You’re learning. You’re absorbing new information, and new context.

    Skills that serve strongly in future positions, like leadership or the C-suite, where it’s more crucial to listen and take in the inputs of the others in the room before making final decisions.

  • TOTW/5 – Avoid the Dreaded Death Spiral – with API Endpoint Separation and Versioning!!

    TOTW/5 – Avoid the Dreaded Death Spiral – with API Endpoint Separation and Versioning!!

    “The best leaders anticipate future scenarios.” – a senior staff engineer

    A Primer

    Alright, let me talk about one of the most exciting scenarios I ran into at work. I was amazed at the the foresight and the forwards-thinking my staff engineer demonstrated in a single 30-minute meeting. He recognized how to avoid the meta-stable failure problem and the consequential run-time crashes that occur with high traffic loads.

    During feature development, my junior engineer and I deliberated on how to incorporate a new business workflow into an existing API endpoint. The workflow entailed additional steps : local PostgresDB database write-IO operations , write-IO operations to a data storage tier held by another team, and complex business rule chains.

    These changes entailed performance degradations across the board : CPU, Mem, and Disk utilizations. They also entailed latency and processing delays, with synchronous, blocking I/O operations to store to both data layers.

    If a K8S pod had to manage both ( or egads, multiple ) API endpoints, than we could run into a death spiral esque issue.Let’s imagine we’re in the first state stable, where our traffic hasn’t hit high scale loads. Initially, the world seems nice.

    But traffic load is increasing – we’re entering into a vulnerable state, and we’re dangling on a precipice. This state can happen in a thundering-herd scenario, where in customer requests would inundate a a K8S pod ( or other compute resources ). The pod would naturally die out, and requests would be forced to route to other pods. But then those pods would also shut down.

    In a futile but bold attempt at system recovery, the cluster would attempt to spin up a new pod, but alas, there’s still to many requests – the new pods quickly die out. Now we entered Metastable failure state : a state whose recovery lies contingent on manual human intervention.

    But there’d be benefits. We’d have two API endpoints – one for the version before the new feature development, and one after. And instead of having one K8S pod dedicated to servicing both API endpoints, we could introduce two distinct K8S clusters – each with their own pods – for each API endpoint. Let’s call them Cluster 1 ( for /v1 ) and Cluster 2 ( for /v2 ).

    Resource Allocation for Cluster 1 ( /v1 ) can be a scalar multiplier less than resource allocation for cluster 2 ( /v2 ) , since /v2’s compute needs supersedes /v`1’s needs.

    Benefits of API Endpoint Separation ( and Versioning )

    • Pre-empt cascading failures and scalability issues ahead of time
    • Follows best-industry practices
    • Single Responsibility Pattern : can have API endpoints represent different compute processes.
    • Minimize resource footprint
    • Allocate optimized resource groups tailored to endpoints – e.g. specialized K8S pods and separate auto-scaling actions for /v1 versus /v2

    Cons of API Endpoint Seperation

    • Up-front investment time in creating separate code, mocks, tests, and deployment pipelines.
    • Extras configuration management

    Footnotes

    • Names, PII, and Sensitive Data Elements, and other corporate details have been redacted or modified to respect journalistic integrity and prevent the disclosure of highly-sensitive work information.
    • Shoutout to the rock star engineers at my company for working through this distributed systems topic!

  • PERSONAL – So tell me … the top 10 character traits that you’re looking in an engineer!

    PERSONAL – So tell me … the top 10 character traits that you’re looking in an engineer!

    Everyone … and I mean everyone … is looking for different things.

    This is a really good question! What am I looking for? In fact, what are most of us in our profession looking for?

    Are we looking for the next 10-x rockstar engineer? Are we looking for someone who knows how to do things immediately? Are we looking for a “star-performer” or a “go-getter”?

    Or are we looking for something else?

    I don’t have perfect answers, BUT, I do have close proxies for what most of us are looking for. Much of this is based on my own personal experiences. Let’s dive in !

    The Qualities List

    Understanding over memorizing

    1. I’m looking for someone who understands their material thoroughly. Like ELI5 ( Explain Like I am Five ) level or teach me using The Feynman Technique method. It’s to easy for someone to memorize a large set of facts ( e.g. Python is a dynamically-typed language or Let’s throw a cache because that’s what I’ve seen in other problems ). But that’s surface-level ; someone needs to be able to tell me more. Like Tell me that a dynamically-typed language is a language that lacks a strong type system, is usually interpreted, with benefits of being faster to write and fewer compiler errors, but drawbacks being a slower runtime and a higher probability of runtime errors.

    Asks good clarifying questions

    1. I love asking questions. In fact, I’d argue that asking clarifying questions is my forte skill! And we need people who can ask good questions. They can ask :
      (A) If we need to do work or don’t need to do work
      (B) If approaches or workarounds need to be thought of
      (C) If we can do something, that we didn’t think we could do
      (D) If we can make things better.

    Solid Collaboration Skills

    Some of my best work in the past has never been by myself. It’s been in lock step with other engineers. Pair programming and learning best ideas and best practices from others teaches a lot. And the higher up in level you go, the more you will have to collaborate with multiple different parties in order to deliver an organizational goal. Collaboration is expected.

    Leave your ego out the door

    1. everyone is the room is smart and is bright ( in their own unique ways ). In the real-world, brightness and intellectual capabilities can take you only so far – organizational problems require loads of people to work effectively. Yes you’ll work with people less bright or more bright than you ( across different stats ) – value each individual for what you can learn from them, and you’ll go far!

    Your Eagerness to Learn

    1. you can be the world’s next Einstein or a pile of bricks, but show me that you’re willing to learn, and I might backfill you more than you think.

    Be Blameless!

    1. everyone makes mistakes. It’s bound to happen. In fact, I’d be amazed if someone had a perfect track record and never did the following : break production, accidentally deletes a production database, loose out on millions of dollars of revenue on a major project, or fail to meet a deadline deliverable. Things happen – and that’s ok. Because that’s where some of the best learning comes from.

    Be able to say “I don’t know”

    There is no one single engineer or other person in the room who “knows it all”. There’s always something you don’t know. And it’s better to admit and say to someone else “I don’t know, but let’s take a look at that” or “I don’t know, but let’s find out!”. It demonstrates positive character traits : (A) intellectual humility, (b) curiosity, (c) interest, and (d) open-mindedness. Traits that are needed in a segment of the population dedicated to solving some of the world’s toughest challenges.

    Your G Factor

    When I interview, I place greater value on an engineer’s general cognitive ability than whether they pass or do not pass the interview ( but kudos for passing ! ) . I want to see how they reason about their problem ( see my other posts for what makes folks shine in interviews TO_BE_DONE ). I’m less concerned about syntax mistakes, compilation issues, or passing only 11/13 test cases. Those aspects of programming can be ironed out. But the General Cognitive Intelligence ( the g factor ) is what I’m seeing out.

    Passion

    No one thrives ( or for that matter, survives ) in our profession while hating this profession. It just won’t work in the long run. There has to be something about computer science that allured you in. Did you like theoretical computer science and algorithms? Did you like developing mobile apps? What about a video game you created? Did you like debugging and diving into OS kernels? Tell me what you like and bring it to the table, because I’m sure it’s highly valued.

    A good attitude

    It’s like passion, but you need to be able to come in and want to solve problems. You can’t just come in and say that you don’t want to solve problems or work on a feature. Otherwise, nothing would move forward and happen! And a good attitude is contagious : your co-workers and your managers pick up on it!

    Other ways to ask this question

    1. Tell me the five qualities that you’re looking for in an engineer?
    2. Tell me what made an engineer in the past stand out to you/
  • TOTW/6 – Overcoming “Analysis Paralysis” – strategies for decision-making when faced with multiple paths

    TOTW/6 – Overcoming “Analysis Paralysis” – strategies for decision-making when faced with multiple paths

    Less Thinking, More Doing!

    – Your mid-western American, middle-aged dad’s advice

    Tell Me Your Story

    Alrighty then, let’s talk about one of the harder skills that I struggled to learn to master : Analysis Paralysis. How to make decisions in an organization when faced with multiple paths.

    Back at Capital One, I led the delivery of a feature for handling failed events in an Apache Kafka event stream. For these failed events, I had to store them in logs. But many questions arose up on handling the logging?

    The Big Technical Questions :

    1. Persistency – do we have to store the logs? For how long? Is 90 days sufficient? 365 days?
    2. Log level – do I need to store only requestIds or additional information ( request ID, request body , responseId, response Body )
    3. Costs – I have three solutions, should I bias for the cheapest?
    4. Anticipating Future scenarios – what if request payload structures change? What if they transform from structured to unstructured?

    And all of a sudden, I’m now in charge of setting the lead and the agenda for design meetings – collaborating with my upper management and other seniors engineers – on what are the best choices to make.

    The Choices I had ; the cards I was dealt with

    NOTE : PLEASE FACT CHECK CORRECTNESS OF INFORMATION BELOW

    1. Amazon S3
      • Pros : Established solution across most places, can handle multiple data types ( unstructured, BLOBs, etc., ), offers hot, cold, and archival tiers with automated lifecycle transitions, version control, and quick access to metadata
      • Cons : overkill when we need low persistency ( 90 days only ) and store only structured, textual data with known limitations ( e.g. 1000 bytes of information )
    2. AWS Cloudwatch :
      • Pros : Already existing solution with cloud vendor.
      • Cons : Supports only textual data with short-term persistency of 90 days before archival .
    3. Splunk/New Relic
      • Pros : Commonly-used third-party Enterprise logging tool. Offers advanced grep-esque/search capabilities
      • Cons : Harder for folks not as familiar to work with.

    The Group Dynamics/Psychological Questions :

    1. Not trusting yourself – what if I made the wrong decisions? I come back 6 – 12 months later and I find out things are breaking. Am I moving fast enough?
    2. Others don’t know the answer ( or your context/problem ) better – you can’t always expect other engineers on your team to know the answer. Soliciting input and feedback is usually a good idea, but you don’t want to run into a situation where you think to yourself ( 4/5 of us agree and 1/5 of us doesn’t agree, what do we do ). Quorum/consensus with majority agreement gets us unstuck.

    In Conclusion

    It’s typical of very bright people to be able to see all the different paths that one can take to conjure up a solution. Which is good, but what we want to avoid is getting stuck. We want to get moving, even if we get something imperfect moving.

  • SYSTEMS – Automated Testing – wait so I can’t always just visually inspect my code using a UI?

    SYSTEMS – Automated Testing – wait so I can’t always just visually inspect my code using a UI?

    A primer!

    No matter which company you go to – Big Tech, Start-ups, Insurance Tech – you can expect to run into the issue of needing to hold discussions with your engineering team on how to best execute testing for your applications. In the world of a single developer working in their own machine, testing is relatively straightforward – if it works on my local machine and I feed inputs X and get out Y, everything works.

    Well yes, but that’s in the world of a single application on a stand-alone computer. Great for academia or student settings, BUT, not the way that industry goes. Turns out real-world testing is far, far more involved.

    The benefits of automated testing.

    What is the Automated Feature Testing? Why not manual testing?

    • Frequently, engineer talent tests their features in manual, one-off ways. It’s effective, but it’s not the fastest. Reasons
      • Single engineers need to be told to trigger scans using existing tools ( UI or API ) to verify if a feature in production works and “does what it does”.
    • Let’s introduce a continuous polling / execution mechanism for where we mock the feature tester
      • Assert on a cadence ( every 24 hours/hourly/weekly ) to verify the feature behaves as expected


    The Big Questions Engineering Talent Should Ask!

    • Do business requirements or technical complexities justify the need to introduce automated testing?
    • How do we automate effectively? With environment constraints, costs, and other factors?
    • Can a team still work effectively with one-off, visual/human-based end-to-end feature testing?

    What’s my End Goal?

    • Create business value – can communicate to any stakeholder/end user that a feature does what it does.
    • Get customer trust : tests communicate correctness, latency, and performance guarantees to end users.
      Simulate production environments – mirror the real-world, in a non-production environments, as safely as possible.
    • Worst case scenario preparation – can we think about the worst case scenarios before they even happen? Can a single test catch a black-swan event?
    • Audit log : Have a maintainable audit/log history with the status of whether a productionized feature works and meets SLAs or SLOs.

    Benefits

    1. Comprehensiveness : Capture more errors codes, paths, and details that a human can miss out on in one-off UI testing.
    2. Time savings : A short-term time investment of 10-20 hours over two weeks is needed, but the long run savings of time spent not dealing with debugging, root cause analysis, or triaging issues late at night pays of long-term dividends.
    3. Externalize dependencies : remove reliance on the following : Devs to execute ad-hoc tests and working UI interfaces ( it’s all programatic ).
    4. Yields confidence in correctness and path coverage – tests emulate production environment closely.
    5. Can introduce metrics or track frequency of errors/other failure types towards centralized logging.
    6. Follows best practices : in many companies, we lack client-side / UI applications to execute triggers
    7. Addresses automation gaps.
    8. Portability of automated test set ups to other use cases
    9. Ensures client-agnostic testing ( Insomnia, UI, Airflow )
    10. Addresses failure mode concerns ( e.g. distributed system problems )

    Frequently used Tech and Tools

    • Apache Airflow – open source job scheduling service. You can define Python-like code to make API calls to external microservices and execute a sequence of tasks with their corresponding dependencies.
    • Cron – scheduling service. Used in Linux-heavily environments.

    When to bias towards manual testing?

    • Release prioritization – meet urgent deadline deliverables.
    • Set up Challenges – endpoint calls drastically differ.

    What are Automated Health Checks?

      • Can hit a generic API endpoint or a targeted /health API endpoin
      • Receives the equivalent of a 200 OK HTTP response.
    • Pros
      • Quick execution
      • Standard industry best practice
    • Cons
      • No verification of async operation behavior
        • Say for an API call, we can get a 200 OK response, but no ability to verify a successful scan
      • No verification of application behavior
        • Upstream health checks ( at component with sequence number 1 ) does not verify downstream components work ( say at sequence number 6 )
    • Automated health checks is not the same as automated feature testing
      •  ( I’ll be using the term feature testing here ).
    • Is in more depth – verifies that an application does what it does
    • Emulates similar industry practices – see Netflix’s Chaos Engineering principles, or SRE practices

    Credit where Credit is Due

    Geico Tech Talent : credit tech talent – Vincent Chong, Ashok Subedi, and Nagendra Kollisetty – for refinement of ideas

  • SYSTEMS – Mastering System Design Interviews: Key Challenges

    SYSTEMS – Mastering System Design Interviews: Key Challenges

    Online resources

    • algoexpert.io – a solid peer-to-peer real-time practice platform
    • interviewing.io – a solid peer-to-peer real-time practice platform
    • pramp.com – good for junior/entry-level engineers, with a targeted focus on algorithmic-style questions ( and limited system design )
    • leetcode.com

    How I prepped System Design Interviewing?

    I’ve been asked this by a good number of peers, including mid-level ( L4 ) and senior ( L5 ) peers who work in name-renowned companies like Google and AirBnB. And I’ll definitely admit the truth ; preparing for a system design interview is harder than an algorithmic interview.

    So how did I prepare?

    1. Self-studying the books : I plowed through Alex Xu’s Volume 1 and Volume 2 System Design Interview Books
    2. Paid practice sessions : I immediately signed up for 20 mock interview practice sessions on interveiwing.io ( https://interviewing.io/ ) for system design preparation. Technically speaking, I think those slots were meant to be only one-hour long each, but many interviewers I practiced with were nice and oftentimes extended sessions to 75-minutes or 90-minutes in length. I got good feedback on where I could improve, what to communicate, and what to emphasize – pointers such as changing the order of sections, thinking about security mechanisms, etc.,
    3. Mock interview sessions : I set up 5-6 practice mock interview sessions with my friends, each about an hour ( or 90-minutes ) in their session length. I figured that the best way to learn a subject is to teach a subject; I already knew how to mock algorithmic interviews – mocking a design interview would be incredibly similar.

    But why are they significantly harder, Hari?

    Well, there’s a couple of reasons. Let’s dive into them!

    But why are System Design Interviews Harder to Prep?

    1. Resource Prep Scarcity– unlike algorithmic interviews, which have been around the block for a long time and where we have a long list of resources ( e.g. CTCI, Leetcode.com, AlgoExpert.Io ), we don’t have as many resources for System Design Preparation. Alex Xu’s one of the few authors who even took a stab at architecting books to help others learn system design skills.
    2. Interview Practice Scarcity – companies ask algorithmic questions more than design questions ( and whether that’s a good or a bad thing is up to question ). Let’s suppose a senior engineer has to do three on-sites for Big Tech Companies – they can expect upwards to fifteen hours of total interviewing : each one hour block 50 minutes of interviewing and a 10 minute cooldown break. Each company on-site will entail a single 1-hour phone screen and a 4-hour onsite. I expect the phone screen to be algorithmic and 2/4 sessions in the on-site to be algorithmic, netting us about 9 ( out of 15 hours ) of deliberate practice in the land of algorithms. Consequentially, developers get solid algorithmic practice, BUT, not as much design practice.
    3. Domain Knowledge Expertise – System Design skills aren’t writing APIs contracts or throwing a SQL database models. Harder design questions require some knowledge in networking protocols, distributed systems ( e.g. failure modes, leader election ) , and databases index data structures ( LSM trees or B-trees ) as well .
    4. Hard to prep for atypical, non-standard questions – There’s plenty of material on questions such as Designing Instagram or Designing WhatsApp, but questions asked are occasionally niche and akin to real-life case studies ( e.g. designing a large-scale data ingestion/streaming service for a social media platform ). You may need to deviate from traditional patterns or templates.
    5. A lack of on-the-job practice – not all software engineering jobs need system design skills, and even if they do, they might not need the skills frequently exercised. A discussion on quota bucket might involve a month long project, but other projects can be scoped to code-only features or other proprietary organizational goals.
    6. Mismatches between Academy and Industry – undergraduate classes in networking or distributed systems are useful ( from their theory-heavy emphasis ), but they don’t extend full comprehensive coverage of all system design topics ( e.g. API contracts design or tradeoffs discussions between SQL databases versus No-SQL databases ).

  • PERSONAL – Growing from Junior Engineer to Senior Engineer – the challenges and the takeaways

    PERSONAL – Growing from Junior Engineer to Senior Engineer – the challenges and the takeaways

    Speak less, and listen more.

    Being a senior engineer across companies has been both an amazing experience and an arduous experience. I’ve gotten more used to the requirements, roles, and responsibilities of the position, but I still notice gaps and places where I can upskill more. Alright, let’s dig and dive in!

    My Personal Story of Growth

    My growth into this role wasn’t easy ; there were definitely growing pains. I faced challenges with feelings of frustrations in adopting new day-to-day habits, understandings, and workflows. I had to learn how to change my habits and re-architect my calendars to accommodate the role’s needs . I’ve had to learn how to better leverage block scheduling to redesign a calendar to work with others. I had to learn to collaborate more with my Product Manager to create feature-trackable tickets and tasks, where they both communicated the most appropriate level of context across audiences – technical and non-technical – and needed to ship features.

    The hardest skill of them – “ACTIVELY LISTEN – AND SPEAK LESS”

    This skill matters, and in so many places too.

    I also had to learn how to engage in more active listening. Coming into the role of senior engineer, I imagined that I would be the one “calling the shots”. I imagined that I had to be the main talker in the discussions or conjuring up the full design schematics. That I’d have to behave like the sole person who says to their team “Here’s how we do everything “under the sun””.

    I was absolutely wrong!!! What was I thinking! Oh my god!!!

    Nowadays, I speak less – I focus more on soliciting input from others, across all organizational roles and levels. I’m amazed how much others can meaningfully contribute – staff engineers, senior staff engineers, managers, tech leads from other teams, and even junior engineers . It’s amazing to see systems and problems evolve when you enable your peers to drive the conversation and the “take the lead” in their expertise areas. The best example was one week where I personally saw three individuals ( names not shared for journalistic integrity 🙂 ) in my design meetings scope down a system and help me create three tickets – each focused for three different phases of work – to meet a multi-quarter deliverable .

    WOAH!!!

    As a senior engineer, it’s become my responsibility – in design discussions that can go across a week from half-an-hour to one-hour long – to “hold the space” and let the other participants ( typically more experienced in their domains ) drive the conversations, the questions, and the answers. I take on the role of an observer or a note taker; I combine the best of their inputs to make the set of decisions needed to evolve our team’s understandings of business problems.

    Meeting Role Expectations

    1. Tickets and task creation – coding features is easy, but solving the right business problems and translating those into code features is hard. A single person – such as a PM ( product manager ) or a direct manager – can not bear the sole responsibility to figure out what their organization needs to accomplish. It’s not pragmatic and it’d never scale!!
      Imagine a worst case scenario where on a 11-person team – with 1 PM and 10 engineers – that each engineer asks the PM to figure out what the team should be doing. Nope, it turns out that senior and senior-plus engineers need to collaborate with their product teams and management chain to figure out how to steer the ship. Collaboration with PMs, PPMs, and other parties on long-term, multi-quarter deliverables matters more.
    2. Doing work – to figure out, not do work – sometimes, your work will involve figuring out NOT TO do work. Wait what! But I just poured hours in design discussions, reading through code, and setting up meeting agendas. And I ended up finding out that … we didn’t even have to execute on a task. Wait, we don’t have to handle all data sources? We don’t have to build something that already exists :-O ! WHAT!!!
      Yep this happens. And it’s not a bad thing. Welcome to the R&D ( Research & Development ) side of engineering. It means you ( and your collaborators ) are being smart. Turns out there’s a lot of work that was done to figure out not to do work – those 16 hours you spent over two weeks helped you avoid multi-quarter efforts in the hiding :-O . Woah, your team gets to focus on more pressing deliverables instead. It also presents as an opportune time to make good write ups on your R&D.
    3. You’re no longer “just a coder” – being a junior engineer means getting to spend most of your time writing code or reading code. I’d argue that 80% of time as a junior engineer is spent deep in code bases : reading code or writing code. But the higher your level, the less time you’ll code – you’ll either naturally transition or get forced by meetings with others to more high impact tasks : designing systems and infrastructure, junior talent mentorship, code reviews, navigating ambiguity, and design documentation. Don’t go into the senior, staff, or principal level roles if you want to code 24/7.