[HN Gopher] Snakemake - A framework for reproducible data analysis
___________________________________________________________________
Snakemake - A framework for reproducible data analysis
Author : gjvc
Score : 148 points
Date : 2023-07-15 11:35 UTC (11 hours ago)
(HTM) web link (snakemake.github.io)
(TXT) w3m dump (snakemake.github.io)
| spratzt wrote:
| How does this differ from something like Airflow ?
| samuell wrote:
| One way to contrast these in broad terms is that tools like
| Airflow are generally much more focussed on continually running
| well defined production workflows, while tools like Snakemake
| are mostly focused on creating a reproducible pipeline for one
| particular analysis, or making it easy to perform exploratory
| analysis, exploring the effect of different input data and
| parameters.
|
| One way this focus is reflected is how e.g. Snakemake are much
| more focused on naming files in a way that you can figure out
| what input data and parameters were used to produce them,
| making it easier to compare results from variations on the main
| analysis.
| chazeon wrote:
| There are a lot of differences.
|
| By design, Airflow needs a centralized server/daemon, whereas
| snakemake is just a command line tool, like make/cmake. This
| would become an issue in the HPC environment.
|
| In Airflow the workflow is assumed to be repetitively executed
| whereas in snakemake it is usually run once (like you will only
| compile the program when source files change).
|
| Airflow has a queuing system, whereas Snakemake is to be used
| in conjuncture with other task management systems.
|
| In Snakemake, shell script always feels like a second-order
| citizen, whereas with Snakemake, it has good support for shell
| scripting, which enables easy integration with tools made in
| other programming languages.
| Kalanos wrote:
| yaml
| samuell wrote:
| In the (unusually long) introduction of our paper on SciPipe, we
| did a pretty thorough overview and contrasting of the pros and
| cons of the top similar tools, including Snakemake, as we
| basically tried most of them out before realizing they didn't at
| the time solve our problems:
|
| https://academic.oup.com/gigascience/article/8/5/giz044/5480...
|
| (A little background, FWIW: We had extra tough requirements as we
| were implementing the whole machine learning progress in the
| pipeline, so needed to combine parameter sweeps for
| hyperparameter optimization with cross-validation, and none of
| the tools, at least at the time did meet the needs for dynamic
| scheduling together with fully modular components.
|
| Snakemake and similar tools in our experience are great for
| situations where you have a defined set of outputs you want to be
| able to easily reproduce (think, figures for an analysis), but
| can become harder to reason about when the workflow is highly
| dynamic and it might be hard to express in terms of file name
| patterns the desired output files.
|
| Nextflow subsequently has implemented the modular components part
| which we missed (and implemented in SciPipe), but we are still
| happy with SciPipe as it provides things like an audit log per
| output file, workflows copileable to binaries and great debugging
| (because of plain Go), all with a very small codebase without
| external Go dependencies.)
| remram wrote:
| FYI the link to the "installation page" in scipipe's repo's
| README is 404.
| samuell wrote:
| Ouch, thanks, fixed!
| dodslaser wrote:
| Shameless plug for a project I'm somewhat involved in: Hydra-
| genetics provides a growing set of well structured snakemake
| modules for bioinformatics (NGS) workflows.
|
| https://github.com/hydra-genetics/
| krastanov wrote:
| You might like mandala (https://github.com/amakelov/mandala) -
| it is not a build recipe tool, rather it is a tool that tracks
| the history of how your builds / computational graph has
| changed, and ties it to how the data looked like at each such
| step.
| samuell wrote:
| That's a cool approach, thanks for sharing!
| dwheeler wrote:
| For a very different approach, check out make-booster:
|
| https://github.com/david-a-wheeler/make-booster
|
| Make-booster provides utility routines intended to greatly
| simplify data processing (particularly a data pipeline) using GNU
| make. It includes some mechanisms specifically to help Python, as
| well as general-purpose mechanisms that can be useful in any
| system. In particular, it helps reliably reproduce results, and
| it automatically determines what needs to run and runs only that
| (producing a significant speedup in most cases). Released as open
| source software.
| Liam2010 wrote:
| [dead]
| derbOac wrote:
| Snakemake seems really interesting (in a good way). I was going
| to say I wasn't aware of it but then in watching the Intro video
| I realized I had seen it before.
|
| However, I find myself scratching my head about when I would use
| it, or what need it's fulfilling.
|
| It seems like most of what it brings to the table should be
| directly inferrable from the analysis and presentation code and
| comments. If you're working with multiple systems, isn't this
| what scripting languages were originally for?
|
| It seems like another layer of complexity on top of what's
| already there, which ... just adds complexity.
|
| Or is the idea to make a data analysis-specific scripting DSL?
| teekert wrote:
| _However, I find myself scratching my head about when I would
| use it, or what need it 's fulfilling._
|
| When you are in Next Generation Sequencing data analysis, it is
| basically THE tool. Snakemake comes with anything I ever
| needed, since it uses conda to install workflow dependencies,
| tools like BWA, fastqc, multiqc etc, are all present and fresh.
| I feel that Snakemake was made for NGS data analysis.
|
| Many Snakemake users, like me, are biologists that know some
| Python and bash, I have never used Make for example. The people
| pointing out weaknesses here have complex, CS related use cases
| it seems, for bio-informaticians in the NGS field it's a
| godsend.
| matthew_stone wrote:
| Yes, it's a DSL.
|
| Here's a simple scatter-gather example. Let's say you want to
| count the number of lines in each file for a list of samples,
| and report a table of counts collected from each sample. Define
| a rule to process each input file, and a rule to collect the
| results.
|
| I find this much less complex than an equivalent bash workflow.
| Additionally, these rules can be easily containerized, the
| workflow can be parallelized, and the workflow is robust to
| interruption and the addition of new samples. Snakemake manages
| checking for existing files and running rules as necessary to
| create missing files, logic that is much more finicky to
| implement by hand in bash. with
| open('data/samples.txt') as slist: SAMPLES =
| [l.strip() for l in slist.readlines()] rule
| all: input:
| "results/line_counts.txt" rule count_lines:
| input: "data/lines/{sample}.txt"
| output: "processed/count_lines/{sample}.txt"
| shell: """ cat {input} |
| wc -l | paste <(echo -e {wildcards.sample})
| - > {output} """ rule
| collect_counts: input:
| expand("processed/count_lines/{sample}.txt", sample=SAMPLES)
| output: "results/line_counts.txt"
| shell: """ cat <(echo -e
| "sample\tn_lines") {input} > {output} """
| flanked-evergl wrote:
| It's really nice, but I think it would have been better if it did
| not mix YAML and Python source and rather provided an API for
| writing workflows completely in Python. All that you get from the
| YAML + Python mix is that tools that work with either YAML or
| Python does not work well.
| Decabytes wrote:
| When I was doing my bioinformatics masters I learned how to make
| all my pipelines in Snakemake, so I never learned how to set up
| pipelines by chaining bash scripts. I was confused when I
| encountered pipelines that weren't written in snakemake or
| nextflow.
| jerrygenser wrote:
| I see it can be used to define and run workflows. But for
| reproducibility on top of the execution of operations on data,
| I'm wondering if there's any way to version the underlying data?
|
| Or would you typically use this -in addition- to a tool like dvc?
| I've used dvc a bit and while it's quite good for data
| versioning, I find the workflow aspect is clunky.
| troymc wrote:
| Pachyderm has data versioning / data version control built-in.
| I guess other tools do too.
| samuell wrote:
| The way a workflow like Snakemake can help here is generally by
| letting the filenames pretty much describe how each particular
| output was created, meaning data outputs can act as immutable
| in a sense.
|
| What I mean is that rather than create a new version of a file,
| if you run the same analysis with different sets of parameters,
| it should generate a new file with a different name rather than
| a new version of the old one. This also helps comparing
| differences between output from different parameters etc.
|
| That said, there are workflow platforms which support data
| versioning, such as Pachyderm (https://pachyderm.com), but it
| is a bit more heavyweight as it runs on top of Kubernetes.
| bafe wrote:
| The reliance on filenames to define (parametric) dependencies
| was among the reasons I later adopted nextflow. The model fit
| the type of computation dependencies better for my case. In
| the mean time snakemake grew and many DAG that were hard to
| describe back then are now expressed directly with snakemake
| primitives
| bafe wrote:
| There is indeed an old issue asking whether they could return a
| provenance graph using the prov ontology in JSON format
| https://github.com/snakemake/snakemake/issues/2077 I think it's
| a good task to work on if you like to contribute
| jlduan wrote:
| I am very fond of https://zenodo.org, especially for small
| datasets for scientific publications.
| samesense wrote:
| I use snakemake along with dvc. It detects changes in files,
| and reruns steps to produce downstream files as needed.
| teekert wrote:
| Snakemake is a beautiful project and evolves and improves so
| fast. Years ago I realized I needed to up my game from the usual
| bash based NGS data processing pipelines I was writing. Based on
| several recommendation I choose Snakemake. I have never regretted
| it, It worked perfectly on our PBS cluster then on our Slurm
| cluster. I made some steps to make it run on K8s, which it
| supports, and most recently, I'm still/again happy with my choice
| for Snakemake because it (together with Nextflow) seems to be the
| chosen framework for GA4GH's cloud work stream's "products" like
| WES and TES [0]. This seems to be the tech stack where Amazon
| Omics and Microsoft Genomics focus on [1]. It enables many cool
| things, like "data visiting": Just submit your Snakefile (the
| definition of a workflow, a DAG basically) to a WES API in the
| same data center where your data lives, and data analysis starts,
| near the data. Brilliant.
|
| I owe a lot to Snakemake and Johannes Koster, I hope some day I
| can repay him and his project.
|
| [0] https://www.ga4gh.org/work_stream/cloud/
|
| [1] https://github.com/Microsoft/tes-azure
| bsmith89 wrote:
| I too owe a lot of my PhD and postdoc productivity to
| Snakemake. It's my bioinformatics super-power, allowing me to
| run a complex analysis, including downloading containers
| (Singularity/Apptainer) and other dependencies (conda), with
| one command.
|
| Great for reproducibility. Great for development. Great for
| scaling analyses.
|
| Snakemake is vital infrastructure for my work.
| matthew_stone wrote:
| 100% agree, and it's wonderful to see Snakemake on the top of
| HN.
|
| Snakemake is an invaluable tool in bioinformatics analysis.
| It's a testament to Johannes' talent and dedication that, even
| with the relatively limited resources of an academic developer,
| Snakemake has remained broadly useful and popular.
|
| Super nice guy too, he's always been remarkably responsive and
| helpful. I saw him present on Snakemake back when he was a
| postdoc, and it really changed my approach to pipeline
| development.
| tetris11 wrote:
| Its fantastic but it doesn't scale laterelly particularly well,
| compared to just Make.
| ta988 wrote:
| What dimension are you referring to?
| tetris11 wrote:
| Large scale reproducibility was a problem a few years back
| for one. Conda and containers were a constant problem for
| us back then, especially if you had multiple NGS tools
| running in different environments. This has probably been
| solved by now, but we went with another workflow system
| bafe wrote:
| Nextflow seems to scale very well
| krastanov wrote:
| Snakemake is great, but it does feel like just a slightly more
| modern Make.
|
| I am pretty excited about research projects that tie the recipe
| and the computation closer together so that you do not preserve
| just the last recipe, but the whole history of exploratory
| computation and analysis.
|
| E.g. mandala (https://github.com/amakelov/mandala), a project of
| a colleague of mine which is basically semantic git for your
| computational graph and data at the same time.
| bafe wrote:
| I love snakemake. It almost saved my PhD, but then I found
| nextflow which suited my type of problems better. What's is
| slightly off-putting about snakemake is that the internal API
| isn't well documented, I wanted to contribute a new remote for
| SQL database and I had to figure out most of the method by
| comparing with other examples. Anyway my PR has been inactive
| since months, which surprised me since usually they tend to
| review and approve quickly
| gschoeni wrote:
| Super cool! Would love to see an integration with Oxen and their
| data version control https://github.com/Oxen-AI/oxen-release
| Protostome wrote:
| This is a fantastic project. It's crucial to note that Snakemake
| is an extension of Python, meaning you can directly incorporate
| Python code into your makefiles.
|
| In our team, we utilize Snakemake along with Singularity. The key
| operations, such as model training and inference that aren't
| straightforward shell commands, are compartmentalized using
| containers. Snakemake significantly simplifies the process of
| integrating these various modules.
___________________________________________________________________
(page generated 2023-07-15 23:01 UTC)