The Nextflow Podcast

In this episode, Phil is joined by Ben Sherman, Jon Manning and Rike Hanssen to take a fresh look at Nextflow plugins, two years on from the plugin walkthrough in episode 35. They cover what plugins are for: sharing reusable functions across pipelines, hooking into workflow events with observers (e.g., nf-slack notifications for nf-core full-size tests, or carbon footprint tracking with nf-co2footprint), and building custom executors and file systems. Ben explains what changed in Nextflow 25.10: the new nextflow plugin create command to scaffold a project, far less Gradle boilerplate, make install for local testing, and make release to publish to the plugin registry at registry.nextflow.io, with a one-time approval for the plugin name. Older Nextflow versions still read the legacy JSON index, which the registry keeps updated for plugins that support them. Rike demos nf-emoji, her first plugin, with themed emoji progress bars, special-day countdowns and optional terminal confetti, and talks about using AI to write tests once she knew her way around the code. Ben explains how custom config scopes let the Nextflow language server validate plugin settings. Jon introduces the new plugin development side quest at training.nextflow.io, covering scaffolding, custom functions, trace observers, testing and config scopes. The episode wraps up with how plugin name claims are reviewed, the expectation that community plugins are open source, and options for distributing private plugins internally.

00:00 Podcast Episode 58 - Plugins

00:09 Introduction

01:01 Welcome to our guests

01:48 An introduction to plugins

06:14 Common plugin use cases

08:25 What's new in Nextflow plugins since 2024

17:11 Demo: Rike's NF-emoji plugin

30:23 Global installs & custom config scopes

33:30 The new plugin training material

42:34 Publishing and approving plugins

47:50 Closing thoughts

In episode 57 (March 2026), Phil is joined by Edmund Miller and Ben Sherman to discuss how recent and upcoming Nextflow language changes can make pipeline chaining and meta pipelines easier.

The episode was split into two parts: this first part surveyed current solutions, listen to episode 56 to check that out.

In this episode we focus on defining workflow “contracts” through clearer inputs (typed params in Nextflow 25.10 and proposed record types, including streamlined sample sheet/record ingestion from CSV/JSON/YAML or queries) and clearer outputs (workflow outputs leaving preview in 25.10, adding an entry-workflow publish section and an output block to publish channels, route files, and serialize channel contents into JSON/CSV/YAML index files).

We discuss reducing direct params access inside modules, using records to pass validated param bundles, generating/leveraging nextflow_schema.json for external tooling like Seqera Platform to validate chains, exploring glue pipelines, command-line piping of Nextflow runs, and how lineage IDs could support more data-centric, automated workflows.

00:00 Welcome

01:18 Nextflow language

02:09 Inputs and outputs

04:14 Streamlining sample sheet parsing

07:47 Placement of params

12:03 Sharing workflows

14:09 Outputs

20:04 Workflow ouputs to chain pipelines

25:56 Publishing channels

32:03 Schema

34:37 Schema + pipeline chaining

37:13 Piping pipelines

41:20 Nextflow Lineage

45:56 Conclusion and wrap-up

Episode 56 of The Nextflow Podcast (March 2026) focuses on pipeline chaining and meta pipelines in Nextflow, with guests Ben Sherman and Edmund Miller.

The episode was split into two parts: this first part surveys current solutions, while part two will cover future Nextflow language changes.

The discussion defines meta pipelines as importing pipelines (e.g., nf-core/rnaseq) as subworkflows to form one DAG with parallelization and full resume, versus pipeline chaining using external orchestration to run Pipeline A then feed outputs to Pipeline B. They cover obstacles to meta pipelines in nf-core, including tooling, parameter/config clashes, and tight coupling of pipeline code and configuration. Current chaining approaches include bash/Makefiles, Python, Seqera Platform APIs, nf-cascade (running Nextflow inside Nextflow), wrapping Nextflow with Snakemake, and automation/orchestration tools like Node-RED, n8n, Dagster, and Temporal, including event-driven patterns on AWS.

00:00 Nextflow Podcast, Episode 56

00:08 Welcome

01:42 Introduction to meta pipelines and pipeline chaining

05:01 What makes importing pipelines difficult?

06:55 CLI tooling to import pipelines

09:13 Overlapping config scopes

10:53 Subworkflows or pipelines?

12:38 Pipeline chaining

13:36 nf-cascade

16:42 Nextflow in Snakemake

22:26 Automating Nextflow runs

24:10 Event-driven bioinformatics

26:32 Node-RED + Seqera

30:40 Node-RED flexibility

33:45 Glue code

35:31 Other automation frameworks

37:02 Bioinformatics pipelines vs. ETL workflows

38:56 Tangent: What makes Nextflow special

44:11 Dagster automation demo

47:29 Temporal automation demo

51:13 Wrap up

In episode 55 of the Nextflow podcast (February 2026), Phil Ewels is joined by Adam Talbot and Ken Brewer to discuss a new Seqera project: using Terraform to manage Seqera Platform configuration as infrastructure as code. They cover why reproducible, reliable infrastructure matters alongside portable Nextflow pipelines, and how Terraform helps reduce “click ops,” provide version control, prevent drift, and support change control via plan/apply workflows and pull requests.The conversation introduces Terraform concepts and highlights use cases such as deploying many workspaces at scale (e.g., universities), regulated clinical environments with approvals, integrating Seqera with existing enterprise Terraform workflows, benchmarking and iterating cloud infrastructure, and managing dev/staging/prod via variables.https://github.com/seqeralabs/terraform-provider-seqera

In this episode, Phil Ewels sits down with Lorenzo Fontana, an engineer at Seqera with deep expertise in Linux kernel internals, eBPF, and systems programming.

Full transcript and summary blog post here: https://seqera.io/podcasts/episode-54-fusion-snapshots/

Lorenzo is the co-author of the O'Reilly book "Linux Observability with eBPF" and a key developer behind Fusion and Fusion Snapshots.

We explore Lorenzo's fascinating journey from Linux security tools to bioinformatics infrastructure, and take a technical deep dive into how Fusion Snapshots actually work under the hood: including CRIU, incremental dumps, and how tasks can be frozen and migrated between cloud instances in under two minutes.🔗 Links mentioned in this episode:Fusion & Seqera:• Fusion Snapshots Blog Post: https://seqera.io/blog/fusion-snapshotting/• Try Fusion Snapshots with Seqera Compute: https://seqera.io/platform/Lorenzo's Background:• Linux Observability with BPF (O'Reilly): https://www.oreilly.com/library/view/linux-observability-with/9781492050193/• Falco Runtime Security: https://falco.org/Open Source Projects:• criu-static: https://github.com/seqeralabs/criu-static• staticreg: https://github.com/seqeralabs/staticreg• CRIU Project: https://criu.org/⏱️ Timestamps:00:00 Nextflow Podcast ep54: Lorenzo Fontana00:04 Welcome and intro01:33 Background in eBPF06:46 Becoming an expert in eBPF11:29 O'Reilly Book13:02 Falco and linux kernel bugs16:07 Move to Seqera20:42 Intro to Fusion26:18 Comparing Fusion to other solutions29:00 Fusion Snapshots31:15 How Fusion Snapshots moves tasks35:58 Contributions back to CRIU39:05 criu-static43:12 staticreg44:36 Fusion Snapshots broken down46:50 Fusion Snapshot spot reclamation50:18 Incremental dumps53:59 Fusion snapshot process timing57:25 Seqera full stack59:20 Call to action01:00:52 pigz01:03:41 How to try Fusion Snapshots01:04:42 AI usage in the future01:11:06 Wrap up📢 Connect with us:• Community forum: https://community.seqera.io• nf-core: https://nf-co.re• Seqera: https://seqera.io#Nextflow #Bioinformatics #CloudComputing #Seqera #FusionSnapshots #SpotInstances #eBPF #Linux

The Nextflow Podcast is back! After a brief hiatus, Phil Ewels returns with a comprehensive look at everything that happened in 2025 across the Nextflow ecosystem. Joining Phil for this year-in-review are Rob Syme (Scientific Support Lead), Marcel Ribeiro Dantas (Senior Developer Advocate), and Rob Newman (Product Manager Lead) from Seqera.In this episode, we cover:* Nextflow Language Evolution - The new syntax parser, VSCode extension with native language server support, nextflow lint and format commands, and the game-changing Lineage tracking feature for full data provenance. * Plugin Ecosystem - The new plugin registry at registry.nextflow.io, easier plugin development, and the complete AWS SDK v2 rewrite for dramatically improved S3 performance at scale.* Workflow Inputs and Outputs - How workflow outputs are replacing publishDir, typed parameters, and the foundations being laid for static types in Nextflow.* Seqera Platform Updates - The redesigned pipeline run details page, single-VM compute environments with 4-6x faster startup, Seqera Compute, Fusion Snapshots for spot instance resilience, Studios improvements, and Data Explorer S3 API support.* AI and Automation - Seqera AI developments, the MCP server for integrating with Claude/Cursor/Copilot, and Node-RED integration for pipeline chaining and automation.* Community Growth - nf-core tools releases, the syntax adoption roadmap, Nextflow Summits in Boston and Barcelona, the Ambassador program expanding to 40 countries, Training Weeks, and Seqera's Series B funding.Links mentioned in this episode:- Nextflow Plugin Registry: https://registry.nextflow.io- Node-RED Integration: https://seqera.io/blog/node-red/- Seqera MCP Server: https://seqera.io/blog/seqera-mcp/- nf-core Syntax Roadmap: https://nf-co.re/blog/2025/nextflow_syntax_nf-core_roadmap- Fusion Snapshots: https://seqera.io/blog/fusion-snapshotting/- Seqera Compute: https://seqera.io/blog/seqera-compute-public-preview/Subscribe to stay updated on all things Nextflow, and check out the full show notes at seqera.io/podcast00:00 Nextflow Podcast - Episode 53: 2025 Recap00:09 Welcome01:10 Introductions02:54 Nextflow Language Evolution03:44 Nextflow VSCode Extension06:47 Nextflow lint and format08:21 Nextflow Lineage12:33 Nextflow Plugin Registry17:29 AWS SDK v2 upgrade21:09 Workflow Inputs/Outputs and Static Types26:38 Seqera Platform: Pipeline Run Details29:57 Dynamic resource labels30:56 Pipeline versioning31:58 Pipeline Chaining with Node-RED35:07 Single VM Compute Environments39:14 Seqera Compute41:24 Fusion Snapshots44:32 Seqera AI49:03 Seqera MCP Server50:04 Data Explorer & s3 APIs51:24 Datasets52:54 Seqera Studios Updates56:56 Seqera company updates57:31 nf-core/tools updates58:41 Topics01:00:05 Syntax updates in nf-core01:00:55 Community growth01:03:43 Nextflow Summits and events01:05:00 Nextflow Ambassadors01:07:19 Training Weeks01:09:13 Wrap up

In Episode 52 of the Nextflow podcast, Phil Ewels is joined by Ziad Al Bkhetan and Steven Manos from the Australian BioCommons to explore how Nextflow and Seqera Platform are transforming bioinformatics research across Australia.

Discover how the Australian BioCommons emerged as a national infrastructure investment to support molecular life sciences researchers, building digital platforms and tools that serve diverse research communities - from genome assembly to proteomics. Learn about their journey from identifying Nextflow as the leading workflow management system in Australia to successfully deploying Seqera Platform for over 90 users across 18+ research institutions.

The conversation covers fascinating success stories including the collaborative development of the nf-core/proteinfold pipeline, supporting structural biology researchers with advanced protein folding tools, and the innovative Ozark community phylogenetic trees workflow. Steven and Ziad share insights on overcoming challenges like integrating web platforms with traditional HPC infrastructure at Australia's tier-one supercomputing facilities (NCI and Pawsey), and building hybrid compute environments that seamlessly span local HPC and cloud resources.

Key highlights include Australia's growing contribution to the global nf-core community, culminating in their first official participation in the March 2025 nf-core hackathon with a dedicated Sydney hub. The team also discusses practical advice for other countries looking to establish national Nextflow services, emphasizing the importance of building local expertise and community engagement.

For Australian researchers interested in accessing these services, visit biocommons.org.au to learn more about available platforms, training opportunities, and community resources.

00:00 Podcast Ep 52: BioCommons

00:06 Welcome

00:41 Ziad - introduction

01:36 Steven - introduction

03:40 Introduction to Australian BioCommons

08:13 Nextflow usage in Australia

09:05 BioCommons collaboration with Seqera

12:24 Compute infrastructures in Australia

15:05 Nextflow support & training

16:43 Highlights and lowlights

19:11 Protein fold

21:50 Ozard Community

23:39 Automation with Seqera Platform

25:52 Nextflow Communty in Australia

29:27 2025 nf-core hackathon

31:57 Future plans for Nextflow & Seqera Platform

33:04 Tips for building a National Nextflow platform

36:21 Next steps

37:51 Wrap up

Resources:

• Australian BioCommons: https://biocommons.org.au
• nf-core office hours: https://nf-co.re/blog/2025/apac-helpdesk-2025

In Episode 51 of the Nextflow podcast, Phil Ewels and Ben Sherman discuss Nextflow's upcoming strict syntax - a significant evolution bringing clearer error messages and a more consistent language framework.

They explore key changes including the separation of code from declarations, removal of for/while loops in favour of operators, and changes to type annotations. Learn how these improvements lay the groundwork for better developer experiences with features like static type checking and auto-formatting.

The episode covers migration timelines, practical steps to prepare, and how these changes will ultimately lead to more robust, maintainable pipeline code. Check out the Nextflow docs for the "Updating Nextflow Syntax" guide to start preparing today.

00:00 Podcast Ep51: Strict Syntax
00:07 Welcome and introduction
01:20 Language server background
04:18 Syntax sugar is just empty calories
06:50 Bespoke language syntax
08:20 To DSL3 or not 2 DSL
10:59 Nextflow versioning
15:13 Diving into specifics
16:08 Mixing statements with script declarations
19:19 Better inspect command
21:08 Custom Classes
23:32 For loops, while and switch
26:02 Standard library
26:56 Type annotations
28:49 addParams
32:26 Process script section
34:20 Moving into lib/
36:36 Configuration syntax
41:14 Relieving pressure from the config
43:38 Timelines
44:55 The gain for the pain
47:54 Next steps
48:43 Conclusion

In this milestone 50th episode of the Nextflow podcast, host Phil Ewels sits down with Krešimir Beštak, a PhD student and active contributor to the nf-core community, to explore an exciting frontier in bioinformatics: microscopy and spatial omics.

While Nextflow is traditionally associated with genomics workflows, Krešimir shares how it’s being used to power image analysis pipelines like MCMICRO, supporting complex research into cardiovascular disease and cancer diagnostics. Based at the University Hospital Heidelberg, Krešimir discusses his transition from master’s student to PhD researcher, the translational applications of spatial proteomics, and how Nextflow enables reproducible workflows far beyond its original scope.

00:00 Podcast Ep 50: Krešimir
00:09 Welcome and introductions
03:13 Introduction to Spatial Omics
04:26 Multiplexing markers
06:38 Metabolite microscopy
07:23 Myocardial multiomics
08:58 Microscopy data analysis
11:32 nf-core/mcmicro
13:05 2D vs 3D microscopy
13:38 Computational bottlenecks within analysis
15:43 Other nf-core imaging pipelines
18:36 Downstream analysis after molkart
19:53 Manual interventions
23:27 Minerva
25:39 Google Maps for cells
27:16 Microscopy community around Nextflow
30:34 How to get involved
31:38 Changes in Nextflow for microscopy
32:43 Nextflow Ambassador program
33:17 Conclusion

Join host Phil Ewels in Episode 49 of the Nextflow Podcast as he interviews Dr. Olga Botvinnik, founder and CEO of Seanome. Olga discusses her journey from studying at MIT to founding her non-profit research institute focused on marine genomics.

Discover her groundbreaking work in annotating unknown proteins and her efforts in collaborating with the Earth BioGenome Project.

Olga also shares her experience with the Nextflow and nf-core communities, and her vision for developing tools to aid research in understudied ocean organisms.

Join us as we chat about the vast potential of marine genomics and its applications.

00:00 Podcast Episode 49: Olga Botvinnik / Seanome

00:06 Welcome

01:18 Olga's background

07:30 Colonial tunicates

09:38 Starting Seanome with Arctic clams

11:07 Vision for Seanome

12:46 Earth BioGenome Project

13:46 Animal genomics not metagenomics

14:53 Building nf-core/proteinannotator

19:46 Software engineering in bioinformatics

22:21 nf-core early days

27:46 Bioinformatics Beyoncé

30:50 Seanome future steps

32:10 Final thoughts: Why the ocean

34:19 Goodbyes

In this episode of the Nextflow Podcast, we dive into highlights from the recent Nextflow Summit held in Barcelona in 2024. The summit featured an incredible range of talks, demonstrations, and discussions showcasing the latest advancements in workflows, data analysis, and community collaboration. With so much exciting content to explore, we’ve picked out some of our favourite moments to share with you. From impactful new features to inspiring keynote addresses and humorous presentations, this episode offers a curated selection of clips that capture the energy and innovation from the event. Join us as we relive the summit’s best bits and discuss their significance for the Nextflow community. 00:00 Start 00:10 Podcast introduction 00:47 Introductions: Group 1 01:56 Clip 1: Wave mirroring 04:32 Clip 2: Throwing cat food 06:00 Clip 3: crisprseq 08:03 Clip 4: nf-test 10:34 Clip 5: Workflow output definition 12:39 Clip 6: Open Science Software Foundation launch 15:08 Clip 7: FAIR code with nf-core 16:58 Clip 8: Teaching with Nextflow 18:56 Clip 9: Conda lock files 21:24 Clip 10: MultiQC in scripts 22:43 Clip 11: Small Nextflow Big Nextflow 25:17 Introduction: Team 2 26:11 Clip 12: Genomics England 29:47 Clip 13: Alphafold interactive reports 33:26 Clip 14: Data Studios 35:32 Clip 15: Covid and Ebola 37:35 Clip 16: Seqera AI 41:34 Clip 17: Nextflow VSCode DAGs 44:33 Clip 18: Fusion snapshots 48:38 Clip 19: Sustainable computing 52:38 Clip 20: nf-core/tools 56:37 Clip 21: Industry best practices with nf-core 58:58 Clip 22: Federated data queries 01:01:56 Clip 23: Nextflow Ambassadors 01:05:04 Conclusion

In episode 47 of The Nextflow Podcast, Phil Ewels and Ben Sherman discuss the launch of the new Nextflow language server, a significant upgrade providing advanced code intelligence features such as code completion and error hints for VS Code users. They detail the benefits of formalizing Nextflow as its own programming language, reducing reliance on Groovy, and improving error messages and code clarity. They also cover new documentation, plans for future features like type annotations, and encourage community feedback and adoption. Find the new docs here: https://www.nextflow.io/docs/latest/vscode.html 00:00 Podcast Ep 47 - Version 2 - VSCode Extension 00:09 Welcome and introductions 00:44 Podcast rebranding 01:29 Nextflow Summit announcements 02:48 Language server background 06:27 Nextflow as a programming language 09:58 Building a language server 14:12 VS Code interactions with the language server 15:22 VS Code demo 15:50 Errors + warnings tray 17:10 Errors: Level 0 20:00 Errors: Level 1 22:32 Errors: Level 2 23:02 To def or not to def 24:46 Variable references 25:34 Number of arguments 25:57 Hover hints on workflows and processes 26:19 Code navigation 26:52 Output hints 27:17 Code peeks 27:56 Getting help 28:29 Code formatting 30:25 Nextflow schema params 32:03 Future warnings 34:02 Preview DAG 34:54 Fixing new syntax errors 35:42 Config files 37:07 Updating the plugin 37:33 Roadmap: Type checking 38:55 Conclusion

In this episode of Channels, the Nextflow podcast, host Geraldine from Seqera and her colleague Ben Sherman dive into the differences and similarities between Nextflow and the Workflow Description Language (WDL). They discuss the origins and development of WDL at the Broad Institute, the challenges of working with different workflow languages, and how Nextflow's channel-based data flow model compares to WDL's approach. The conversation covers key features, type systems, and runtime specifications, highlighting the strengths and limitations of both languages. Ben hints at upcoming improvements and enhancements for Nextflow, aimed at making the language more robust and user-friendly. The episode offers valuable insights for bioinformaticians and developers navigating the complexities of workflow management.

In this episode of Channels, the Nextflow podcast, join us as we explore Nextflow and AI. Our special guest, Sasha from Seqera, shares his remarkable journey from the consumer tech industry to bioinformatics. Learn about the birth of TinyBio, its innovations in bioinformatics AI applications, and its recent acquisition by Seqera. We delve into AI concepts, the challenges faced, and the future of integrating AI with Nextflow. Tune in to discover how advancements in AI are reshaping the bioinformatics landscape. 00:00 Podcast 45: Tinybio : Nextflow x AI 00:09 Introduction 00:34 Sasha's background 03:17 How tinybio started 05:57 Tinybio co-founder: Vishal Patel 06:35 tinybio chat public launch 09:20 Failed feature launches 10:39 Running the code too 12:46 User trust 14:11 Accessing files via chat 15:08 Experiment idea generator 17:16 Pipeline generator 20:25 AI concepts: Foundational models 22:36 AI concepts: RAG 24:53 AI concepts: Langchain 26:37 AI concepts: Fine tuning 29:14 Challenges in bioinformatics AI 31:53 AI concepts: Q&A training 33:09 Motivation for joining Seqera 34:43 What happens to tinybio now? 35:36 Future plans 38:20 AI and open-source 40:50 Find more updates 41:37 Wrap up

In episode 44 of the Nextflow Channels podcast, Phil Ewels welcomes Alex Pelzer— a key figure in the Nextflow and nf-core community. Alex shares his journey from academia to the pharmaceutical industry, highlighting the critical role of regulatory frameworks in bioinformatics.

🌟 Key Highlights:

• The transition from working at QBiC to Boehringer Ingelheim.
• The importance of maintaining high-quality, reproducible pipelines in bioinformatics.
• Understanding regulatory requirements and their impact on clinical trials.
• How regulatory standards and open-source initiatives like @nf-core are shaping the future of bioinformatics.

Alex also discusses the formation of the @nf-core Regulatory Special Interest Group, which aims to bring together experts in the field to establish and promote best practices for ensuring pipeline quality and compliance with regulatory standards.

Looking forward to seeing many of you at the upcoming #NextflowSummit in Barcelona this October! We’ll be working with this #regulatory group during the nf-core hackathon.

00:00 Episode 44: Regulatory revolution 00:06 Introductions 00:33 Upcoming Nextflow Summit Announcement 01:34 Alex’s Background and Early Days with Nextflow 04:41 Formation and Growth of nf-core 06:18 Transition to Pharma and Regulatory Work 08:56 Regulatory Challenges and Practices in Bioinformatics 13:30 Tool Selection and Quality Assurance 15:23 Evaluating Software Trustworthiness 17:08 Overhead in Academia vs. Industry 18:11 Downstream analysis 20:57 Regulatory Special Interest Group 25:28 Special Interest Groups in nf-core 28:59 Future Directions and Community Involvement 30:15 Open source contributions in pharma 33:27 Conclusion and Upcoming Events

In this episode we explore the new features of Seqera's Data Studios and Data Explorer, with Phil Ewels, Rob Newman and Rob Syme from Seqera.

Discover how to use these tools for troubleshooting Nextflow pipelines, tertiary analysis and Nextflow development. We discuss the pain points that led to the creation of Data Studios and how it's designed to allow scientists to interactively and collaboratively work with data and complex workflows, without having to move large datasets around.

Rob Syme wows us with another fantastic practical demonstration, setting up and using Data Studios to write and test a Nextflow pipeline in VSCode running on the cloud in a Data Studio environment, including running the Nextflow CLI with task submission to AWS Batch.

We cover features like session persistence to save work states, and upcoming custom container support for your own specialized applications.

Learn how these tools can enhance your computational biology projects and make seamless cloud integration a reality.

00:00 Channels Podcast 43: Data Studios 00:26 Introductions 01:54 Data Studios 04:51 Move the compute to the data 06:13 Real-time collaboration 06:47 Data Explorer 09:41 Access to public data 10:45 Data Explorer demo 13:56 Data Studios setup 20:17 Session persistance 22:52 Data Studios RStudio demo 28:24 Nextflow development in Data Studios 36:17 Future development 37:01 Custom containers 40:01 Boston Summit demo 44:01 Lifetime management 47:14 Wrap up

A small but intrepid team of Seqerans recently attended the ISMB 2024 conference in Montreal, a great mix of computational biology and bioinformatics with a largely academic / research-focused audience. Join Geraldine Van der Auwera, Rob Syme and Florian Wuennemann for a lively discussion about the scientific themes of the conference, their experience running the booth, meeting with the ISMB and BOSC community, and resources for job seekers.

Join us in exploring the latest Nextflow release, 24.04. Phil Ewels and Ben Sherman dive deep into the new updates in Nextflow’s most recent version. We discuss advanced retry strategies, job arrays, resourceLimits, Singularity’s OCI mode, and the game-changing Workflow Output Definition.

We’re actively seeking feedback from the community on these new features. We would love for you to experiment with the new syntax and let us know your thoughts.

Please let us know what you think either via Nextflow GitHub issues or via the community forum.

In this episode we refer to a the recent blog post about the release, which you can find here: Nextflow 24.04 - Release highlights.

Intro - tweaks, fixes and new support in 24.04

Performance and Stability Improvements

Ben kicks off the discussion with key performance and stability improvements. The latest stable release, 24.04, includes numerous bug fixes and performance enhancements. There’s a significant focus on closing gaps and adding retry strategies for better stability, especially concerning cloud providers. Notably, if Nextflow’s API calls against cloud providers fail for any server-side reason, it will retry automatically, avoiding pipeline failures caused by temporary server issues.

Publishing Changes

Previously, if a file failed to publish, Nextflow would only issue a warning. It was possible for a pipeline to complete successfully without noticing missing output files. Now, by default, Nextflow will fail the pipeline if the publishing fails, though there’s an option to revert to the old behavior. Additionally, retry strategies for publishing have been introduced to ensure retry attempts if an issue arises.

Job Arrays

One highly requested feature is the introduction of job arrays. This allows users to submit many jobs as a single submission, alleviating strain on schedulers. The submission happens in a batch, and then the scheduler can process and plan effectively. Once a job is submitted as part of a job array, the jobs run independently. If any child job fails, it is resubmitted without affecting the rest of the array.

Singularity OCI mode and GA4GH TES

In the past, Singularity supported Docker images by converting them into SIF files, consuming storage and time. Now, both Singularity and Apptainer can run OCI images directly, saving valuable resources.

Additionally, the TES executor has seen significant improvements, now supporting TES 1.1, which brings broader compatibility and integration with existing workflows.

Major new syntax features

Topic Channels

A new feature called topic channels offers a more straightforward approach to collecting channel outputs across pipelines. Channels can emit data to a named topic, simplifying the collection and use of version information from various processes within Nextflow pipelines.

Eval Outputs

Eval outputs simplify the addition of shell commands to tasks. With eval outputs, necessary post-task commands can be defined neatly, avoiding repetitive code within process definitions.

Workflow Output Definition

The concept of workflow output definitions has been introduced. This new syntax streamlines the publication of files by defining publish targets within workflows. Instead of defining publication behaviors within process definitions, users can now manage them at the workflow level, ensuring better clarity and fewer repetitions in the code.

Join us to hear Phil Ewels chat with Maxime Garcia and Franziska Bonath about how nf-core started at the National Genomics Infrastructure (NGI) at SciLifeLab in Sweden.We chat about how the nf-core/sarek pipeline grew alongside and eventually joined nf-core,as well as the early days of nf-core bytesize talks and outreach organisation. Links: - SciLifeLab - National Genomics Infrastructure - Maxime's Nextflow Summit Boston 2023 presentation about Sarek - nf-core bytesize talks

Tune in to hear all about Pixelgen Technologies - creators of a novel technology for doing single-cell spatial proteomics.

We chat about how this fascinating technology works, what you can do with the data and why they chose to develop their analysis suite as an open-source nf-core pipeline.

Links:

• Software:

• nf-core/pixelator pipeline
• Pixelator Python package

• Science:

• Nature Methods publication: “Molecular pixelation: spatial proteomics of single cells by sequencing”

• Presentations:

• Nextflow Summit 2022 talk: “Pixelgen Technologies ❤︎ Nextflow”
• Nextflow Summit 2024 Boston talk: “Nextflow allows Pixelgen innovation engine to reach the market quicker”
• Nextflow Summit 2024 Barcelona

• Pixelgen:

• Pixelgen Technologies homepage
• @PixelgenTech on twitter / X and LinkedIn

• BioLizard:

• BioLizard homepage
• @BioLizard_nv on twitter / X and LinkedIn

Chapters:

• Start (00:00)
• Welcome (00:09)
• Introductions (00:50)
• Intro to Pixelgen Technologies (01:41)
• Lab methods (03:08)
• Nature Methods publication (06:15)
• Data visualisation (09:35)
• Working with single cell data (12:29)
• Cell interactions (15:06)
• Why Pixelgen chose open source (16:05)
• Pixelator Python package (20:10)
• Building within nf-core (21:40)
• Florian - Biolizard (23:01)
• Pipeline overview (25:19)
• Using nf-core from a commercial setting (26:50)
• Coming out of stealth (28:53)
• Extending Pixelator (32:24)
• Dealing with customers (34:58)
• Looking to the future (39:38)
• Interactive analysis (41:32)
• Advice for others (43:12)
• Pixelgen at the Nextflow Summit (45:16)