Item — HN Reader

If you’re a data professional in a company where data quality is low or non-existent, then improving the quality of data in the organization is absolutely part of your job, most likely the most important one

It’s less cool and flashy than the real time online machine learning model you want to build, or the multi-level bayesian model to determine causality between revenue and an obscure event, but that data dictionary and table/db catalog really needs to be built before everything else

Then document the essential reports to the extent that a high schooler should be able to produce/maintain them

Then adjust the reports so that the metrics indicate what management actually needs to learn from them (if you don’t know what’s important for them and for the business, then finding out is part of the job, too)

All these things are extremely important, and it’s silly to suggest your work is meaningless just because it’s not fun or interesting. There are no tv series about civil engineers building sewage networks but public sanitation saves more lives than medicine. Data work works the same

6 comments

> then improving the quality of data in the organization is absolutely part of your job, most likely the most important one

You're absolutely right, but it can be hard to get management to bat for this kind of stuff. The biggest hurdle I've seen with getting "good" data is fixing the issues involves other teams prioritizing the work on their backlogs. Depending on the company and where the teams lie on the org chart, fixing things might require a multi-quarter directive from the CTO. Which isn't happening.

so then ... confidence intervals are going to be so use as to be useless. and every second week the data science team can present a 2 slide presentation, where the first slide says bad data = bad result, and the second says good data = good results, and then furiously flip between the two during the Q&A.

and everybody clapped!

This is already what I and the others I've spoken to do (who haven't given up). But, at the types of organisation I'm describing, it's near impossible.

You know that Airflow instance I mentioned? Three years later, that team still doesn't have anything to schedule Python workloads, despite extremely smart people trying to angle the politics of the damn thing to finally get traction for months on end. When you've got no leverage, you've got no leverage, and you don't want to spend years of your precious life arguing so that at some point, in a few years, you can do your job properly.

I absolutely agree with what you're saying, but I suppose I'm saying that, at many jobs, you're better off leaving and somehow locating the places where you have a chance at accomplishing what you've laid out. There's a sliding scale of incompetence, and some organisations are close enough to competence that you can actually move them the right way with some diligence and effort.

That isn't possible most of the time working for a larger company. Too often the problem with bad data starts not at the collection stage but the data is bunk to begin with.

Since the data is not under your control -- it belongs to the "real" engineering team, which has vastly greater status -- you will not be able to. You will complain and they will ignore you. You will offer to help and they will refuse. You will open a PR and they will reject it.

I’m dealing with the opposite experience right now. I work at a big tech company and lately we’ve been working closely with a data science team. I’m desperate for some guidance on what sort of data pipeline and practices we can set up for them and have a hard time getting back anything but “can you export some rows as a csv”. Which is actually easier said than done considering data compliance policies and what not. Maybe there’s a good reason but it gets a bit frustrating and I can’t imagine it’s not slowing them down as well.

You know it's a badly run team when they keep pulling you off your self-built, self-motivated ETL packages to breathe down your neck about incremental, barely-there releases.

Thanks for the space to vent a sec...

Too bad data scientists rarely have the skillset necessary to even begin doing that work.