harisrid Tech News

The Value of Writing – a Primer

Hi all!

Today, I’m going to delve into another topic – writing!!! Yep, writing!!!!

Yep, the subject you ( probably ) badly wanted to avoid back in high school and college. The dreaded dark days when you read classical literature like Moby Dick or All Quiet on the Western Front and write out long-form essays, reflections, or short stories on the books your teacher assigned.

Well, it turns out that writing is a crucial skill for engineers. In fact, I’d argue more important than coding ( especially with the advent of Chat-GPT and GenerativeAI tools that can quickly spit out code ). Many of us share this feedback with each other – the best engineers are solid technical writers. They’re not just good coders. In fact, they may not even be your team’s best coder. Someone more junior might have shipped out more LOC – lines of code – then them during their entire tenure!

But the best engineers sure know how to write. They’re good at convincing multiple audiences – engineers, product owners, leadership and management – that their ideas are “worth their salt”, that their system designs are solid, and that their user stories make sense. Their documentations serve multiple roles – mentorship, conveying impact, and tracking accomplishments.

More ever, seniority and leveling naturally endows writing. Your role responsibilities -coupled with the folks you collaborate with ( senior talent ) – will act as a “forcing function”. You’ll either get tasked out – or create the task yourself – to write up up design documents and technical specifications. Writing at the high level is ineluctable ( and inescapable ).

Lastly, it’s not a bad thing. In fact, I argue that writing is good. Think about it – the most famous people in your life and in your profession write. Barack Obama ( former US president ) has to read and write. Roger Penrose ( renowed astrophysicist ) wrote The Road to Reality. And Thomas Cormen wrote the bible of algorithms – CLRS. Emulating your profession’s titans is a good starting place!

But what do I write about? How do I get started?

But what if I run into writers block? What if I can’t find a topic to write about.
That’s a good question. Let’s tackle that too!

• Write about what you learned – did you create a side project in your free time? Contribute to a long-running OSI ( open source initiative ). Did you engage in a fascinating project back in your undergraduate days? Write about your learnings. What were your challenges? What were your biggest takeaways?
• Write about what you know – Finding topics is a hard subject, but start from what you know. Words flow out easily from a priori established knowledge bases. Look into starting with topics from your day-to-day work – this can help you in a few ways. Writing about your work, outside of work, not only helps you better understand your work, but enables you to deliver faster on your work deliverables ; it’s a positive feedback loop in the hiding!
• Write opinion pieces – do you have a technical opinion? Something you want to share? Do you want to talk about what you think makes for an effective design document? Or how to make good user stories? Perhaps even a dissenting opinion ( gaaaspp, turns out you can disagree with engineering best practices, because a best practice isn’t a best practice in all situations ).
• Write about engineering best practices – there’s a lot of good engineering practices out there, and every engineer knows a good habit or two. I’d be hard-struck to say that I’ve met an engineer who never taught me a better technique in coding, debugging, designing, or investigating issues.

How to Refine Your Technical Writing Skillz?

• Practice, Practice, Practice – the best way to get better at technical writing, is to practice writing. Practice makes perfect. Find opportunities to engage in technical writing. This can include outside your company.
• Solicit feedback – the best writers don’t operate in a vacuum; they solicit and ask for feedback from their peers. Peers can see things that you don’t see in yourself – the good and the bad. I’ve asked for feedback in the past, and I’ve gotten examples such as
  a. Be concise and clear – shorten sentences and write less verbosely.
  b. Think about your audience – write different types of documents, based on the audience? You can set topics for documents – brainstorms, condensed, and expanded versions.
  c. Focus on the big picture – shift your focus away from the low-level details towards the bigger picture.
  d. Write about today and tomorrow – write about problems from a timeline perspective – what does the current state of systems look like today, and what do we want those systems to resemble tomorrow?
  e. Incorporate visuals – focus on capturing reader attention to convey your points more effectively – leverage state transition diagrams, system diagram, and tables. Color-code and label the visuals too.
• Read ! Read widely! – are there examples of engineering writing you admire? Books whose styles resonated and “clicked” with you? Blogs that captured your interest? I strongly admired these blogs ¹for their specific traits :
  a. Communication styles – Gusto ( https://engineering.gusto.com/ ) and AirBnb ( https://medium.com/airbnb-engineering ).
  b. Communication Simplicity and openness – the mathematical blogs Better Explained ( https://betterexplained.com/ ) .
  c. In-depth expositions written out in Professor Scott Aarson’s blog ( https://scottaaronson.blog/ ) .
  

1. I still consult these blogs – and other online media – from time-to-time 🙂 ↩︎

( and please don’t let the fear-mongering of the world migrating to streaming get to you )

Building Automation – a Primer

Hi all,

I’ve been working hard on a ETL-esque project at Geico to set up the internal plumbing, the infrastructure, and the business workflows to automate and execute the following :
Step #1 ( SCAN ) : Scan a data source and populate a local table with records of each scan
Step #2 ( EXPORT ) : Trigger an export of the data to its destination

In today’s current state, we’re automatically scan data sources and populate our table of records, but, we manually trigger endpoints to export.

But tomorrow’s world. Well that needs to look different? The scan and the export needs to be automated. Not only because end customers need to see their results as soon as possible, but also because we can’t keep asking humans – a.k.a. the engineers on our team – to go into an application and trigger the endpoints to expose data. That’s so much work.

Now on the surface, the automation looks easy to do. Hey, I’ll write out a couple of shell scripts, call API endpoints in a loop, and we should be good to go, right? Not exactly. There’s a couple of caveats and unknowns making this work challenging to execute. Let’s delve deeper and dive in 🙂 !

The Unknowns to Navigate

(A) When did an export need to be triggered? Was it after a scan’s completion? On each record – r1,…,rn – within a scan? Or on batches of records ( e.g. batches of 10,20, or 50 records )?
(B) When did data exports need to show up? Did they need to show up within five minutes? 24 hours? Or other configurable windows?
(C) How would we handle failed events or failed processing? Did we have ways to avoid duplicate work? Or can we allow duplicate events, if the code triggered for events is deterministic ( e.g. the output is the same for a given input )?
(D) What can we leverage to keep the design simple?
– Can we incorporate intermediate tables, Kafka queues/event buses, or long-running background daemons?
– Can we make changes to application-layer code or data-layer code? Can we leverage invariants of database triggers to confirm the completion of events – upstream and downstream?
(E) What design components are owned by our team, versus owned by other teams?
– Components owned by our team are faster to configure, customize, and develop.

Scenario #1 : The Stream Mode

The true stream mode emulates a more event driven/CDC [ Change Data Capture ] idea, in which once a scanned record is written out to a database, some notion of an update having just occured needs to immediately trigger an export.

Before I started gathering requirements, I made the project more complex than it needed. I imagined in my hand that on each scan ( s_i ), I needed to trigger the export on the immediate storage of scan records ( r_1,r_2,…,r_n ). This means that at the smallest level of granularity, I needed to immediately export results to their intended destination. That meant introducing a notion of an event bus or CDC ( continuous data capture ) on each insert or update of a record into our local table.

Now while CDC or event-streaming is cool, it’s more complicated. The idea introduces additional steps and questions such as :
(A) Who should fire the event to inform that a record was stored? Whomst is the best authoritative source of truth? Should it be the producer ( the scanning application )? Or should we periodically poll the database?
(B) If we leverage events, how long should events persist?
(C) What do we do if a consuming application or daemon dies?
(D) Do we own Kafka queues or event-driven architectural components?

All of this introduces additional tasks and JIRA tickets. Like building out the Kafka queues or conducting a research & development story to find out if a local table has CDC capabilities. This delays the date of feature deliverability and adds long-term maintenance layers. More ever, it’s easy to get caught in the woods and introduce to many optimizations ( e.g. do we have to change the database storing our scanned results )? Optimization is good, BUT, over-optimizing a design is a “bad practice”.

What do we do instead?

… hmm, let’s try the batch mode.

Scenario #2 : The batch mode

The alternative solution involves batch operations. Instead of triggering an export on each record scanned, what if exports could be triggered on the completion of an entire scan? It turns out that a configurable time window for when the exports need to show up for our end users and a local database table enabled us to introduce the notion of time-bounded processing delay ( e.g. one hour or two hours ) before exporting results.

But how did we determine what to export? Did we conjure up criteria? Hmm – we have a local table of scan results with a flexible schema; we can introduce additional columns :
a. ScanExecutionFinished – Boolean flag ( TRUE = finished, FALSE = not finished )
b. ScanAlreadyExported – Boolean flag – ( TRUE = is exported, FALSE = not exported )
c. ScanExecutionStartTime – UNIX timestamp ( informs when a scan started )
d. ScanExecutionEndTime – UNIX timestamp ( informs when a scan ended )

We can leverage the four columns to enable a FIFO-esque automated processing of scans and exports ; exports should auto-trigger for scans with earlier execution times than those with later execution times. A cutoff time window ( e.g. a timestamp value ) can be used – let us denote this as T_prime

If all three conditions are met for a given scan :
a. ScanExecutionEndTime <= T_prime
b. ScanExecutionFinished=True
c. scanAlreadyExported = False
Trigger an export and set scanAlreadyExported = True
Else, do not trigger an export.

As for how to do the batch processing, that’s up to the engineering team. I won’t delve to deep here, but I’ll briefly touch upon the two big ones –
(A) Utilize Airflow DAGS to trigger endpoints
(B) Introduce periodic consumer-side polling of database tables.

My Biggest Takeaways

I learned a lot from this experience in designing systems – lemme share them!

1. Requirements Gathering – Focus on the gathering of the requirements! . Don’t just dive in. Set up meetings with other engineers and understand the overall ask at hand.
2. Build Simpler – Streaming is cool and we say that’s the future of the world. But if we’re operating in an unconstrained environment and need to ship faster, why not build simpler? Let’s bias to batch processing if we can!
3. What’s under your control – Identify your locus of control – interior and exterior. Ask what’s under your control and what’s not under your control? In our case, we had control over the following :
   (A) The time window of exports showing up
   (B) The handling of duplicate events
   (C) The components we could introduce and own ( e.g. local PostgreDB tables or Kafka queues )
4. Conjure criteria sets and conditions – Can we determine a rules engine – or set of criteria – to determine when to execute an operation? Let’s imagine we built from the perspective of a grossly-simplified data structures & algorithms problem?

“What did I just get myself into”?

Hi all,

I quickly want to write on how to model tracking datasets within Enterprise organizations. It’s a frequent ask of engineers – especially data engineers – to figure out effective schemas to model the multitude of variegated data abstractions float – managed and unmanaged – across companies.

But what makes this challenging? Isn’t it sufficient to build out a single SQL table that captures all pieces of relevant information ( e.g. the owners, the creators, the line of business, or the datasets themselves )?

Not exactly? There’s a lot of complexity and pre-empting of challenges that can rise with scalability, permissions, data evolvability, schema changes, or even new data sources getting introduced. There’s no one-size-fits-all solution. In today’s 2025, datasets come in a multitude of forms : Excel spreadsheets, Large BLOBs, or Kafka queues. There’s also the problem of scalability – the volume of data builds up the longer an enterprise organization exists and the more capabilities a company builds out. Then throw in product evolution – how do we build effective model today, that can withstand unexpected changes in product scopes and stakeholder requirements, coming up in the next 10 years? And what about testing? Can we even “battle-test” these models? What are the edge case scenarios where I need to think about pre-empting ( e.g. building indexes for fast lookups ahead of time or reducing table dimensionality to fit data into fewer stores of external memory ? Do I need to also build for a future inventory, if years down-the-road, someone needs to identify and catalog all data assets ( existing and removed )?

But back on focus? In the story, senior developer Natalie is given ambiguous customer requirements ; she needs to quickly conjure up a MVP – minimal viable product – for a feature that needs to be shipped and delivered to an end customer within a quarter-long scope. The feature requires customers to query the underlying datasets and to answer commonly-recurring questions, such as :

• When was the last time a dataset was created? When was it last modified?
• Who created a dataset? Who owns a dataset?
• What team or what organizational unit ( OU ) does a dataset belong to?
• How large are the datasets? What datasets consume the most external disk space?
• Can I introduce filters for efficient searching? Should it be a singular filter ( e.g. a categorical type ) or multiple filters ( e.g. tags ) ?
• If I’m executing searches, should they be exact searches ( e.g. match a tag for verbatim ) or fuzzy-based searches ( e.g. prefix matching )?

What would Natalie design and why? And more ever, can she justify and explain her rationales to her team members, if other senior engineers – like Emilio or Vinay – engage in collaborative back-and-forth Q&A?

What Database Solution would you use?

I’m using a SQL table for modeling since :
(A) The data is naturally relational.
(B) The schema is mostly fixed and easily accommodates schema-on-read – it won’t frequently adjust.
(C) The information is mostly structured – there’s minimal unstructured elements ( e.g. large BLOBs, images, or videos )
(D) I expect external personas – data analysts or business analysts – to execute ad-hoc queries with SQL or other relational languages of familarity.
(E) The data’s natural relationships also engenders natural hierarchy – it’s easy to build a “visual tree” of relations.
(F) SQL is easy-to-understand by most developers, and old solutions continue to persist effectively for modern-day settings. Shipping and iterating quickly usually commands precedence over modern solutions – even those that are “more performant” or “more efficient”.

Alright, let’s get to building – but wait, what am I building?

But let’s take a step back – before we dive into schemas and models, think about the actors. Who and what are we modeling? Are we modeling people? places? things? tangible products ( e.g. e-commerce products )? or logical abstractions ( e.g. a sense of ownership or a sense of creation )? And how are these abstractions related to each other? Does a person have only one sense of ownership, or, can they have multiple ( it’s typically one ).

They’re five primary entities – datasets, creators, owners, permissions, and organizational units ( OUs ) – with cardinality relationships :

Datasets : Creators – 1:1
Datasets : Owners – 1:1
Creators:Permissions – 1:1
Owners:Permissions – 1:1
Creators:OU – 1:1
Owners:OU – 1:1

Datasets – created to capture metadata about enterprise data abstractions ( e.g. last created/last modified at ). This is the “bread-and-butter” table.
Creators and owners – in any organization, somebody created a dataset and somebody owns the data sets. While most of the time, the creator = the owner, there’s many cases where the long-term owners are a distinct party. Let’s think of a situation where in downstream ML model developers ask upstream data engineers to create datasets for ML model testing, BUT, it’s the developers who maintain and own those datasets. And then there can be multiple owners and transfers taking place over the longevity of data. Thus the usefulness of SOC – separation of concerns.
Permissions – what are creators and owners allowed to do? Can owners read, write, or delete data ( unrestricted rwx-esque UNIX permissions )? Or in most cases, are owners allowed to only READ data?
OU – the organizational unit. Creators and owners are typically employees of a company ; they belong to a team in a larger organizational structure and they report to a direct manager. What if I need to know the teams or the orgs who own data? Introducing a layer of information – even if used in only a few query patterns – endows usefulness.

What would schemas resemble? Why are schema choices made?

That’s an excellent question!

My goals are the create tables that have the right number of columns – not to little, not to much. The goldilocks zone. If I start noticing a surplus of columns, I’ll introduce PK-FK ( primary-key foreign key ) relationships and data normalization ¹; this should effectively eliminate redundancy and helps us built out solutions with a more customizable, granular view of enterprise information.

Datasets : [ PK ( INT ), Dataset Name ( STRING ), Created_Time ( TIMESTAMP ), Updated_Time ( TIMESTAMP), Size ( in KB ) ( INT ), Description ( VARCHAR 200 ), Creator ( FK ) (INT ), Owner ( FK ) (INT )
Creators : [PK ( CreatorID), First Name ( VARCHAR ), Last Name ( VARCHAR ), Created_at ( TIMESTAMP), Permissions ( FK ) (INT), OU ( FK ) (INT ) ]
Owners : [PK(Owner ID ) (INT), First Name ( VARCHAR ), Last Name ( VARCHAR ), Created_at ( TIMESTAMP ), Permissions ( FK ) ( INT ), OU ( FK ) (INT ) ]
Permissions : [PK ( Owner ID ) ( INT ), First Name ( VARCHAR ), Last Name ( VARCHAR ), Created_At ( TIMESTAMP ), Permission ( FK ) ( INT ) ]
Organizational Units ( OU ) : [ PK ( ID ) (INT), Team Name ( VARCHAR ), Org Name ( VARCHAR ), Direct Manager ( INT ) ( FK ) ]

Optimizing targeted data types for columns.

Why use a TIMESTAMP in place of INT or VARCHAR(200) in place of strings?
a. Use VARCHAR(200) or VARCHAR(<insert_fixed_limit_size>) over STRING – this enforces constraints and safety on inputs programatically if there’s known bounds on input sizes ( e.g. the size of a standard tweet on twitter.com is 140 bytes in totality ).
b. Leverage TIMESTAMP over STRING when dealing with unix-stored time. Unix-stored time is interpretable across more formats than human-readable dates ( e.g. DD/MM/YYYY or MM/DD/YYYY ) . TIMESTAMPS also save on disk space and can quickly be ordered – monotonically increasing or decreasing.

Examples of what datasets look like ( I promise I’ll populate them later )

Schema 1 : Datasets

Schema 2 : Creators

Schema 3 : Owners

Schema 4 : Permissions

Schema 5 : Organizational Unit ( OU )

1. Normalization will incur performance degradations and costs with regards to complex queries involving JOIN(…) statements across multiple tables, BUT, for the purposes of building out a model that’s flexible and maintainability, I’m biasing towards the strategy. Normalization typically enables highly-scalable solutions ↩︎

Why does this work matter? How does it impact stakeholders or end users?

Hi all,

I’m writing this blog post since I’ve been working on a data engineering-esque feature extensively at work¹ , BUT, I think it’d be awesome to explain what I’m up to.

Most companies – from large tech to old-school enterprise organizations – hire for data engineering talent : talent that builds out the technical solutions for a company’s data vertical. These engineers focus on problems involved in data controls & standards, governance, and compliance. Data engineers are a “security layer” for companies in the hiding who shoulder an umbrella of corporate responsibilities. Their work involves identifying data sources, scanning sources, and then export their findings to their end users. Each step entails R&D-esque work or feature ticket work, and “the cream of the crop” of engineers ask really good clarifying questions that reduces the level of effort of tasks and enables a long-lasting solution in production with minimal issues .

Why identifying PII/SDEs Matters?

(a) Ensures that customer records are kept complaint and up-to-date with legal requirements – enforced across different levels of government ( federal, state, local )
(b) Pre-empts capital losses incurred from data breaches
(c) Engenders stronger customer base trust in corporate applications

Let’s dig deeper into their complexity and the types of questions good engineers should ask!

Identify the data sources

1. Are the sources with organizational scope? If so, are they internal or external?
2. How can I find out my sources?
   • Identify from configuration files ( e.g. .yaml files )
   • Look into inventories – made available on internal websites or APIs
3. Do I need to meet with other teams?
   • Can other teams identify the inventory of data sources?
   • Can we delegate maintaining the inventory to another team?

Scan the data sources

1. Do we need to scan data sources? Can we identify the path of least resistance?
   • The sources may already be encrypted
   • The sources may be unencrypted BUT they meet enterprise-requirements.
2. If we scan the data sources, what do we look for?
   • Comprehensive scanning – of each record, across all databases
     • Targeted subsets of tables and columns?
3. What data elements do we need to scan for?
   • Focused on conventional PII ²( e.g. passwords and SSNs )
   • Looking for custom SDEs ³( e.g. classification ML-model determined )?
4. How often do data sources need to be scanned?
   • Do I scan on a historical basis? Once every six months ( for all records )?
   • Do I execute continuous scans on a periodic basis ( e.g. hourly runs or nightly runs )?

Export and Deliver the Findings

1. How should findings be communicated?
   • Which party needs the finding? Are they internal or external auditors?
   • Is the easiest strategy – outputting to a CSV/Excel-esque dump – satisfactory?
   • Do we need a ( simple or complex ) UIs to enable the real-time update of identified elements?
2. Retention of findings?
   • Can we remove our findings after a time window ( e.g. a month or a year )?
   • Can we compress and archive findings?

What was my feature deliverable ?

Figuring out the data sources was mostly “done for me”, but I had to develop out the other two steps of the feature ( and it’s capability ) for a new data source ( most of the feature had been developed, but for pre-existing sources ):
a. Scan multiple data sources and databases for PII and SDEs
b. Export the findings to an Enterprise-specific layer for external personas to execute an audit.

What are the data sources you scanned?

My feature works across four major databases – the SQL flavor one being SQLServer and the NoSQL flavor ones being Snowflake, DB2, and PostgresDB. For each data source onboarded, I operated at the granularity of database.schema.table.column, following enterprise/team conventions. The scans ( so far ) have been historical versus continuous – once a source had been scanned, it didn’t have to be scanned again to facilitate export.

How to Feature Test Across Environments?

1. Local environments
2. Non-production environments
   • Sandbox envs
   • Other envs
3. Production environments

Testing types :
1. One-off tests
2. Comprehensive tests – all data sources

Assessing Feature Complexity

1. Metrics collection – build out dashboards, reports, or visualizations showing the volume of data under operations – scan and export. Capture metrics :
   • Across fuzzy/exact text search ( e.g. all databases or database.schemas with names starting with <insert_prefix_here>* )
   • Across logical granularities with tag-based filters : at database, database.schema, database.schema.table, database.schema.table.column granularities
   • Across time windows ( for continual scans ) – on a periodic basis fixed by hour / day / week / month.

1. and I’ll make sure to redact sensitive information a long the way 🙂 ↩︎
2. Personally Identifiable Information ↩︎
3. Sensitive Data Elements ↩︎

“Why is this so hard?”

Hi all,

I want to step back in the realm of enterprise architecture and talk about business workflows. Specifically, the challenges I noticed.

(1) Pre-empting micro-services from becoming an untangled cobweb

A business workflow is more complex than a few lines of code – it’s a singular function ( or multiple functions ), where within each function, we make API service calls. The workflows are conductors who orchestra a business ; the micro services – the musicians – execute a plenitude of supporting business operations : REST operations, data mutation, statistical analysis, or security measures. The intrinsic complexity of workflows and service calls naturally lends itself to a collection of micro services – I’ve once used ten micro services ( back at Capital One ) to process customer assets.

For a singular workflow, managing the undergirding microservices is relatively straightforward. But the number of the complexity of workflows expands in lock step with overarching product complexity. Suddenly, Ia developer gets violently thrusted into a land where each workflow requires a number of services – some similar to others, and some drastically different.

So the question is – how to do we avoid building out to many micro services? Can we make the architecture as clean as possible? Ok, there’s a few things to think about before we build them :

What should I ask ( before I write )

1. Do we need to create a micro service? The best micro service is no micro service.
2. Can we create micro services that respect SRP – single responsibility pattern?
3. Should we create micro services for singular standalone functions? Do we need to create a new module, in case we add further future functionality?
4. Is it better to create separate micro services – for each workflow – or a shared micro service with workflow-tailored functions or conditional logic?
5. Can I get rid of an existing micro service? Let’s remove what’s orphaned or unneeded.
6. Can I make an existing service better and more comprehensive?