▲ 13 points
Ask HN: What Are You Working On? (September 2022)
What are you working on? Any new ideas you're thinking about?
back
10 comments
Still working on growing my newsletter[0] which funnily enough is about ideas.
I’m also working on a proxy for the Mailgun API (as in take Mailgun requests and convert them to something else) —- finally getting the chance to do some more Rust which I haven’t in a while!
Not so much "work" per se, but something that I've been thinking about quite a lot recently is how ML practitioners test their systems.
For example, before launching a model to prod, I've commonly seen that teams have a suite of datasets/evals, and a model has to be at least as good as the existing model in production before replacing it.
But what about the training pipelines? You can have unit tests for all the individual components, but the analogy of "integration tests" would involve just training a full model, which might be very expensive for large neural networks.
One option would be to have a special tiny dataset and tiny model that can be trained fast on CPU. But this would be an imperfect test of the real system. I've also never seen this in practice, so maybe it wouldn't actually be useful.
Curious if anyone can chime in with their experiences?
I once froze a subset of the training data and used it in the test suite for the training pipeline. It was pretty handy as a quick test while refactoring. I don't think I got it integrated into our Jenkins pipeline so it wasn't used by the rest of the team and eventually got outdated as the training data changed.
We didn't do the "at least as good" thing though. In prior jobs I'd seen too many legitimate situations in which a metric declines even though there's no regression, such as bug fixes in metrics or occasional updates to the test data. Instead we committed the model evaluation to git and had to review and approve model updates.
I wish we'd done more testing for that pipeline, particularly the parts that fetched and preprocessed data. I think we had a couple bugs there over the years, or partial missing data.
Interesting, thanks for sharing.
> I'd seen too many legitimate situations in which a metric declines even though there's no regression, such as bug fixes in metrics or occasional updates to the test data.
I guess you could keep both the old and new versions of the test data or metrics in parallel for a few cycles of model update to get a feel for how the old/new settings differ
I've always found the OpenSSL tools painful for managing internal self-signed certificates. At work we make fairly heavy use of them, and are starting to make even heavier use. Our use is more than EasyRSA can provide. So I've been working on a new CA tool:
https://github.com/linsomniac/rgca
In a nod to OpenSSL config files, it can take almost all values: from the command line, from the environment, or from one or more config files. It also allows "pre" and "post" commands so you can run a script after generating the cert, for example for server certs I have a "post" script that will copy it into the appropriate location in the Ansible repo, encrypt the key file, and commit it all.
I still need to implement a "renew" which will take an existing cert, update the expiration date, but also allow adding/removing SANs, possibly other features. But I've been using it to generate all our certs recently and it's working great.
Hardware: I've got a cheap ($200) CNC mill I'm getting ready to use for various projects.
Software: MSTOICAL[1] is a port of a port of STOIC, a Forth like system that does types in the stack, and is weirder, but more logical, than Forth. It's written in C, a language I'm learning as I go.
Software: TreeHouse[2] is an exploration of the ideas show off by DION in this 2020 demo.[3] The direct manipulation of a program as an AST, with the source code being an OUTPUT, a view, of the program. I'm doing it in Pascal, because that's what I'm used to. It's amazing how cool it looks already, even though it's just a toy so far.
[1] https://github.com/mikewarot/mstoical
[2] https://github.com/mikewarot/TreeHouse
[3] https://media.handmade-seattle.com/dion-systems/ (watch the 1st video!)
My diy CNC mill says hi. Making anything cool?
Not yet... but I spent 5 years making gears on 1950's vintage machines. I'll be doing a lot of involute curve interpolation. 8)
I've been working on an event orchestration platform[1] with the initial use case being "surface your deadletter queue, and resolve problems" mostly to scratch an itch. In other words, what happens when you can't process an event and you want a human to investigate the why and perform some kind of action to resolve the issue.
[1] anketta.com
Finishing a book, trying to arrange a few workshops, and trying to put in some time to complete an API mocking service.