Reproducible pipelines

class: title-slide

<div>
<style type="text/css">.xaringan-extra-logo {
width: 110px;
height: 30px;
z-index: 0;
background-image: url(img/00/logo_uni_elixir.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
bottom:1em;left:1em;
}
</style>
<script>(function () {
  let tries = 0
  function addLogo () {
    if (typeof slideshow === 'undefined') {
      tries += 1
      if (tries < 10) {
        setTimeout(addLogo, 100)
      }
    } else {
      document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
        .forEach(function (slide) {
          const logo = document.createElement('a')
          logo.classList = 'xaringan-extra-logo'
          logo.href = 'https://rworkshop.uni.lu/'
          slide.appendChild(logo)
        })
    }
  }
  document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

# Reproducible pipelines

## using `targets` and `renv`

.center[<img src="https://docs.ropensci.org/targets/reference/figures/logo.png" width="80px"/>
<img src="https://raw.githubusercontent.com/rstudio/renv/master/man/figures/logo.svg" width="100px"/>]

### A. Ginolhac | rworkshop | 2021-09-10

---
class: middle, center, inverse

# targets

---

# targets and companion package tarchetypes

.flex.items-center[
  .w-10[
  ]
  .w-70.bg-washed-green.b--green.ba.bw2.br3.shadow-5.ph3.mt3.mr1[
  .large[.bbox[A workflow manager for R]]

.large[
- Saving you time and stress
- Understand how it is implemented in `targets`
    + Define your `targets`
    + Connect `targets` to create the **dependencies** <svg aria-hidden="true" role="img" viewBox="0 0 640 512" style="height:1em;width:1.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M384 320H256c-17.67 0-32 14.33-32 32v128c0 17.67 14.33 32 32 32h128c17.67 0 32-14.33 32-32V352c0-17.67-14.33-32-32-32zM192 32c0-17.67-14.33-32-32-32H32C14.33 0 0 14.33 0 32v128c0 17.67 14.33 32 32 32h95.72l73.16 128.04C211.98 300.98 232.4 288 256 288h.28L192 175.51V128h224V64H192V32zM608 0H480c-17.67 0-32 14.33-32 32v128c0 17.67 14.33 32 32 32h128c17.67 0 32-14.33 32-32V32c0-17.67-14.33-32-32-32z"/></svg>
    + Check **dependencies** with `visnetwork`
    + Embrace **dynamic** branching <svg aria-hidden="true" role="img" viewBox="0 0 384 512" style="height:1em;width:0.75em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M384 144c0-44.2-35.8-80-80-80s-80 35.8-80 80c0 36.4 24.3 67.1 57.5 76.8-.6 16.1-4.2 28.5-11 36.9-15.4 19.2-49.3 22.4-85.2 25.7-28.2 2.6-57.4 5.4-81.3 16.9v-144c32.5-10.2 56-40.5 56-76.3 0-44.2-35.8-80-80-80S0 35.8 0 80c0 35.8 23.5 66.1 56 76.3v199.3C23.5 365.9 0 396.2 0 432c0 44.2 35.8 80 80 80s80-35.8 80-80c0-34-21.2-63.1-51.2-74.6 3.1-5.2 7.8-9.8 14.9-13.4 16.2-8.2 40.4-10.4 66.1-12.8 42.2-3.9 90-8.4 118.2-43.4 14-17.4 21.1-39.8 21.6-67.9 31.6-10.8 54.4-40.7 54.4-75.9zM80 64c8.8 0 16 7.2 16 16s-7.2 16-16 16-16-7.2-16-16 7.2-16 16-16zm0 384c-8.8 0-16-7.2-16-16s7.2-16 16-16 16 7.2 16 16-7.2 16-16 16zm224-320c8.8 0 16 7.2 16 16s-7.2 16-16 16-16-7.2-16-16 7.2-16 16-16z"/></svg>
    + Run **only** what needs to be executed
    + Bundle **dependencies** in a Rmarkdown document with [`tar_render()`](https://wlandau.github.io/tarchetypes/reference/tar_render.html)
    + Increase reproducibility with the package manager [`renv`](https://rstudio.github.io/renv/articles/renv.html)
- Example with RNA-seq data from .bold[Wendkouni Nadège MINOUNGOU]
]]]

---

# Folder structure

.left-column[

```r
├── .git/
├── run.R
*├── _targets.R
├── _targets/
├── Repro.Rproj
├── R
*│   ├── functions.R
*│   └── utils.R
├── run.R*
├── renv/
├── renv.lock
└── report.Rmd
```

]

.right-column[
.large[
- With [`renv`](https://rstudio.github.io/renv/). Snapshot your package environment (and restore! 😌)
- `_targets.R` is the only mandatory file
- Use a `R` sub-folder for functions, gets closer to a <svg aria-hidden="true" role="img" viewBox="0 0 581 512" style="height:1em;width:1.13em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"/></svg> package
- `Rmarkdown` file allows to gather results in a report
- In a RStudio project
- Version tracked with <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M100.59 334.24c48.57 3.31 58.95 2.11 58.95 11.94 0 20-65.55 20.06-65.55 1.52.01-5.09 3.29-9.4 6.6-13.46zm27.95-116.64c-32.29 0-33.75 44.47-.75 44.47 32.51 0 31.71-44.47.75-44.47zM448 80v352a48 48 0 0 1-48 48H48a48 48 0 0 1-48-48V80a48 48 0 0 1 48-48h352a48 48 0 0 1 48 48zm-227 69.31c0 14.49 8.38 22.88 22.86 22.88 14.74 0 23.13-8.39 23.13-22.88S258.62 127 243.88 127c-14.48 0-22.88 7.84-22.88 22.31zM199.18 195h-49.55c-25-6.55-81.56-4.85-81.56 46.75 0 18.8 9.4 32 21.85 38.11C74.23 294.23 66.8 301 66.8 310.6c0 6.87 2.79 13.22 11.18 16.76-8.9 8.4-14 14.48-14 25.92C64 373.35 81.53 385 127.52 385c44.22 0 69.87-16.51 69.87-45.73 0-36.67-28.23-35.32-94.77-39.38l8.38-13.43c17 4.74 74.19 6.23 74.19-42.43 0-11.69-4.83-19.82-9.4-25.67l23.38-1.78zm84.34 109.84l-13-1.78c-3.82-.51-4.07-1-4.07-5.09V192.52h-52.6l-2.79 20.57c15.75 5.55 17 4.86 17 10.17V298c0 5.62-.31 4.58-17 6.87v20.06h72.42zM384 315l-6.87-22.37c-40.93 15.37-37.85-12.41-37.85-16.73v-60.72h37.85v-25.41h-35.82c-2.87 0-2 2.52-2-38.63h-24.18c-2.79 27.7-11.68 38.88-34 41.42v22.62c20.47 0 19.82-.85 19.82 2.54v66.57c0 28.72 11.43 40.91 41.67 40.91 14.45 0 30.45-4.83 41.38-10.2z"/></svg>
]
]

---

#Example with RNA-seq data across 3 cell lines

### PCA shows that differences between cells .bold.red[`>>`] biological effect (roman numbers)

### Solution: Split counts and metadata for each cell

.Large[Do we copy code 3 times?]

---

# Define targets = explicit dependencies

.pull-left[
### `_targets.R`, define 4 targets

Last `target` depends on the **3** upstreams

```r
library(targets)
source("R/functions.R")
source("R/plotting.R")

list(
* tar_target(cells, c("HepG2", "HuH7", "Hep3B")),
* tar_qs(dds, read_rds(here::here("data", "all.rds")),
         packages = "DESeq2"),
* tar_fst_tbl(annotation, gtf_to_tbl(here::here("data",
                                                "gencode.v36.annotation.gtf")),
              packages = c("tibble", "rtracklayer")),
* tar_qs(sub_dds, subset_dds(dds,
                             filter(annotation, type == "gene"),
                             .cell = cells),
         pattern = map(cells), # dynamic branching
         packages = c("DESeq2", "tidyverse"))
[...]
)
```

]

.pull-right[
.bold[Dynamic branching] makes dependencies easier to read.

.large[
> .orange.bold[Of course, someone has to write for loops, it doesn't have to be you]

.tr[
— _.bold[Jenny Bryan]_]]

![](img/targets_outdated.png)

]

.footnote[Figure from `tar_visnetwork()`]

---

# Running targets

.pull-left[
```
● run target annotation
● run target cells
● run target dds
● run branch sub_dds_3078b1e0
         condition time_h
HepG2_I1   control      0
HepG2_I2      HIL6      2
using pre-existing size factors
estimating dispersions
gene-wise dispersion estimates: 2 workers
mean-dispersion relationship
final dispersion estimates, fitting model and testing: 2 workers
● run branch sub_dds_d05c5da7
        condition time_h
HuH7_I1   control      0
HuH7_I2      HIL6      2
using pre-existing size factors
estimating dispersions
gene-wise dispersion estimates: 2 workers
mean-dispersion relationship
final dispersion estimates, fitting model and testing: 2 workers
● run branch sub_dds_c60d7096
         condition time_h
Hep3B_I1   control      0
Hep3B_I2      HIL6      2
using pre-existing size factors
estimating dispersions
gene-wise dispersion estimates: 2 workers
mean-dispersion relationship
final dispersion estimates, fitting model and testing: 2 workers
● end pipeline
```
]

.pull-right[
Options to display time and object sizes
![](img/targets_dag_done.png)
]

---

# Re-running

.pull-left[
```
✓ skip target annotation
✓ skip target cells
✓ skip target dds
✓ skip branch sub_dds_3078b1e0
✓ skip branch sub_dds_d05c5da7
✓ skip branch sub_dds_c60d7096
✓ skip pipeline
```

.Large[All good, nothing to be done ✔️.

Actually `targets` tracks all objects and so functions

A more complete dependency graph shows .bold[functions]
]
]

.pull-right[

![](img/targets_dag.png)

.Large[Let's add the PCA per cell type now]
]

---
class: hide_logo

# PCA, add 4 targets

.pull-left[
### Smaller targets avoid unnecessary re-running steps

```r
[...]
tar_target(rcounts, vst(sub_dds, blind = TRUE),
           pattern = map(sub_dds),
           packages = c("DESeq2")),
tar_target(pca_df, pca_meta(rcounts),
           pattern = map(rcounts),
           packages = c("DESeq2", "tidyr", "dplyr")),
tar_target(pca_cell, tibble(cell = cells,
                            pca = list(plot_pca_meta(pca_df))),
           pattern = map(cells, pca_df),
           packages = c("ggplot2", "tibble"))
[...]
```

.large[
.bold[Translate into]:

- For every cell data, compute regularized counts (`vst`: variance stabilization)
- For every regularized counts, compute PCA (`df`: data.frame, _i. e_ a table)
- For every cell names / PCA tables, plot PCA in a table for easier labeling

]

.pull-right[

![](img/targets_dag_undone.png)

]

---

# PCA results

.flex[
.w-33.b--green.ba.bw2.br3.shadow-5.ph4.mt2.mr2[
### Running

]
.w-33.b--green.ba.bw2.br3.shadow-5.ph4.mt2.mr2[
### Awesome feature: load results IN a Rmarkdown document

.green.bold[Separate] `code` from content

![](img/targets_tar_read.png)

]
.w-33.b--green.ba.bw2.br3.shadow-5.ph4.mt2[
### How to display a plot
![](img/targets_pca_hep3b.png)
]]

---

# The full picture

#### Adding step by step desired analyses

#### Whole analysis takes 24 minutes and 4.54 seconds

.large[
> .orange.bold[Of course, someone has to</br>remember the dependencies, </br>it doesn't have to be you]

.tl[
— _could be William Landau via .bold[Jenny Bryan]_]]

---

# Is it worth the effort?

.left-column[

.huge.bold.green[Yes]

.Large[
#### For you
- independence
- autonomy
- skills
- _free_ time
- confidence over results
- reproducibility
- fun 🥳
]
]

.right-column[
### Reproducibility

.bold[`targets`] via `git`
```r
> renv::history()
   commit         author_date      committer_date                                    subject
1e8dd2278 2021-02-23 15:29:57 2021-02-23 15:29:57             reformat creating config files
24c1222db 2021-02-15 17:07:01 2021-02-15 17:07:01    highlight gene type in the DEG patterns
326c8a726 2021-02-04 16:16:38 2021-02-04 16:16:38 cluster LRT genes by they dynamic patterns
4c6791796 2021-01-26 13:08:15 2021-01-26 13:08:15      gene types in upset plots for lengths
5865ee70b 2021-01-21 16:36:48 2021-01-21 16:37:08                            add upset plots
6c06b496e 2021-01-20 17:03:29 2021-01-20 17:03:29                         add upset protoype
[...]
```
]

---

# `renv` features

.pull-left[
- `renv` parses your code and finds library calls
- `install` a package including its dependencies
- `snapshot` registers changes, hashes and origin

```r
> renv::snapshot()
The following package(s) will be updated in the lockfile:

# CRAN ===============================
- RcppParallel   [5.0.2 -> 5.0.3]
- cli            [2.3.0 -> 2.3.1]
- pkgload        [1.1.0 -> 1.2.0]
- tint           [0.1.3 -> *]

# GitHub =============================
- targets        [ropensci/targets@main: 598d7a23 -> bdc1b29c]

Do you want to proceed? [y/N]: 
```

`restore` to a certain point in time

]

.pull-right[
`renv.lock` file after a `snapshot`

```
  "R": {
    "Version": "4.0.3",
    "Repositories": [
      {
        "Name": "CRAN",
        "URL": "https://cloud.r-project.org"
      }
    ]
  },
  "Bioconductor": {
    "Version": "3.12"
  },
  "Packages": {
    "AnnotationDbi": {
      "Package": "AnnotationDbi",
      "Version": "1.52.0",
      "Source": "Bioconductor",
      "Hash": "ca5106b296b3aa6af713ce197be547c1"
    },
    "BH": {
      "Package": "BH",
      "Version": "1.75.0-0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "e4c04affc2cac20c8fec18385cd14691"
    },
    "targets": {
      "Package": "targets",
      "Version": "0.1.0.9000",
      "Source": "GitHub",
      "RemoteType": "github",
      "RemoteUsername": "ropensci",
      "RemoteRepo": "targets",
      "RemoteRef": "main",
      "RemoteSha": "598d7a23661d4c760209c7991bf10584eadcf7c8",
      "RemoteHost": "api.github.com",
      "Hash": "ee66061fd5c757ec600071965d457818"
    },
    [...]
```

]

---

# Reports as Rmarkdown documents

.left-column[

`targets`, written by [William Landau](https://wlandau.github.io/) (pictured), is flexible, robust and still allows for a customized report.

All computing is done only when needed, and code is away from writing content.

Once `knitted` the report can be sent to the inquirer.
]

### Targets Markdown

New in `targets` > **0.6**. Instructions at [William bookdown](https://books.ropensci.org/targets/markdown.html)

Test it as the Rmd template (and excellent [video](https://www.youtube.com/watch?v=FODSavXGjYg) from R Lille meetup by **Landau**):

.center[![](https://books.ropensci.org/targets/man/figures/target_markdown.png)]

---

# Bonus: watch the pipeline running live 🍿

.left-column[
- `targets` events watched live 🎞
- Here, after changing a threshold in the LRT step
- `branches` can be monitored too
- 2 videos joined as I fixed an .red.bold[error] at 1'42"
- Option to display functions (unset here)
]
.right-column[
### `tar_watch()` shiny app from `targets`

]

---

# Before we stop (for good)

.flex[
.w-50.bg-washed-green.b--green.ba.bw2.br3.shadow-5.ph3.mt2.mr1[
.large[.gbox[Highlights]

* `targets`, dependencies manager, re-run what's needed
 * `renv`, bundled your packages and versions inside a project
]]
.w-50.bg-washed-green.b--green.ba.bw2.br3.shadow-5.ph3.mt2.ml1[
.large[.bbox[Acknowledgments  🙏 👏]
* **Eric Koncina** early adopter of `targets`
* **Wendkouni N. Minoungou** for the RNA-seq data
* [**William Landau**](https://github.com/wlandau) main developer of `targets`
]
]]
.flex[
.w-50.bg-washed-green.b--green.ba.bw2.br3.shadow-5.ph3.mt2.mr1[
.large[.ybox[Further reading <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M448 360V24c0-13.3-10.7-24-24-24H96C43 0 0 43 0 96v320c0 53 43 96 96 96h328c13.3 0 24-10.7 24-24v-16c0-7.5-3.5-14.3-8.9-18.7-4.2-15.4-4.2-59.3 0-74.7 5.4-4.3 8.9-11.1 8.9-18.6zM128 134c0-3.3 2.7-6 6-6h212c3.3 0 6 2.7 6 6v20c0 3.3-2.7 6-6 6H134c-3.3 0-6-2.7-6-6v-20zm0 64c0-3.3 2.7-6 6-6h212c3.3 0 6 2.7 6 6v20c0 3.3-2.7 6-6 6H134c-3.3 0-6-2.7-6-6v-20zm253.4 250H96c-17.7 0-32-14.3-32-32 0-17.6 14.4-32 32-32h285.4c-1.9 17.1-1.9 46.9 0 64z"/></svg>]]
- [Main website](https://docs.ropensci.org/targets/)
- [Targetopia](https://wlandau.github.io/targetopia/) **Landau** universe of targets-derived
- [Video](https://www.youtube.com/watch?v=FODSavXGjYg) from R Lille meetup by **William Landau**. June 2021 45''
- [Documentation](https://books.ropensci.org/targets/) as bookdown by **Landau**
]
.w-50.pv2.ph3.mt2.ml1[
.huge[.bbox[Thank you for your attention!  
Hope it was useful!]]

]]