New blog post: "Why computational reproducibility matters"
https://blog.khinsen.net/posts/2025/06/20/computational-reproducibility.html
Discussion
Loading...
@khinsen Thanks!
I may say the reproducible subject is a headache of all DevOps when you start thinking how to deploy the code A from developer machine B to the cloud C with some CI running on blackbox 😱 docker is not an answer even with package lock files you still can't control the base image 100% unless your department has extra budget for that (rare case...).
I'm approaching module 1 of the MOOC slowly
while completing Astro update 2025/06 https://codeberg.org/guix/guix/pulls/706
I may say the reproducible subject is a headache of all DevOps when you start thinking how to deploy the code A from developer machine B to the cloud C with some CI running on blackbox 😱 docker is not an answer even with package lock files you still can't control the base image 100% unless your department has extra budget for that (rare case...).
I'm approaching module 1 of the MOOC slowly
while completing Astro update 2025/06 https://codeberg.org/guix/guix/pulls/706
@khinsen
If you can't identify why you're getting different results on different versions of Debian that means your understanding of your code or the problem is incomplete and you should resolve that before publishing.
@resuna In theory, I agree. But my experience tells me that this is an unrealistic expectation.
@khinsen
I think this should be classified with all the rat studies that haven't corrected for what rats are really capable of.
This guy actually did a study on what it took to keep rats from using external cues to run mazes and it involved freshly sterilized floors for each run to eliminate odor clues, building the maze on a sand table so vibrations didn't cue them in, and so on.
@khinsen bit-by-bit reproducibility in a way 'only' helps to make it easier to talk about the interesting problem of replicability imho.
@gedankenstuecke Exactly! That's the point I have been trying to make - for years, with little success so far.
@khinsen yeah, same here! It's been impossible to convince people that, given the current level of discussion, computational reproducibility is a distraction. But of course there's the "reproducibility cottage industry" and it's hard to make people understand as their salaries depend on this 😬
@gedankenstuecke That's my impression as well. As long as you stick to seemingly helpful bits of solution that never actually solve the problem, you keep that field of work open forever. If in addition you focus on marketing and teaching your ideas, you can even become popular for it.
@khinsen to be honest, that's how I've come to think about the larger #openscience field. The move to "professionalize" it has led to a similar cottage industry with plenty of "experts" that have salaries or income depending on being helpful but not too helpful.
And even worse, it makes it seem like it's something that "regular" researchers can't/shouldn't do as part of their jobs!
@gedankenstuecke I suspect that this is inevitable. As soon as some topic becomes visible enough, there will be people who position themselves first as its managers and then as its bureaucrats. If all goes well, they do a useful job, but sometimes, they end up suffocating the people who do the real work.
1 more replies (not shown)
@khinsen thanks, especially for pointing out that a lack of replicability is a lot more interesting and relevant than reproducibility.
Given the current state of computational reproducibility, I've come to find that problem very boring and irrelevant in a non-CS context. Even in the best case scenario of being bit-by-bit reproducible, it doesn't help learning anything new, but somehow that's the issue people are focusing on in many disciplines.
@khinsen I liked that piece, in particular the critics of "just use [conda/docker]".
What matters more for Alice and Bob "reproducing our results" or "understand how we are reproducing it" (and why we can't).
In the same idea is understanding why we get different numbers important for Alice/Bob or is it more important for the CS fields?
I want time to learn GUIX ...
A small Bonfire corner on the internet
This is a small personal instance of Bonfire in the Fediverse.
Automatic federation enabled