creating-analysis-projects

SKILL.md

---
name: creating-analysis-projects
description: Use when setting up a new R or bioinformatics analysis project, scaffolding directory structures, refactoring messy scripts into clean pipelines, or establishing project conventions. Use when the user says "init", "set up a project", "refactor these scripts", or "create a pipeline". Also use when reviewing whether an existing project follows these conventions.
---

# Creating Analysis Projects

## Overview

Scaffold R/bioinformatics analysis projects with a strict separation between code (version-controlled) and data/outputs (not tracked). Git lives at the **project root** with a whitelist `.gitignore` that tracks only `scripts/`. Every notebook is self-contained. Every convention is enforced by structure, not discipline.

**REQUIRED SUB-SKILLS:**
- Use `writing-r-code` for all R code generation
- Use `writing-qmd-scientific` for all QMD document structure
- Use `superpowers:brainstorming` when designing a new pipeline from scratch
- Use `superpowers:writing-plans` when the pipeline has 3+ notebooks to implement

## Directory Structure

The project uses a **read / write / checkpoints** triad as its core data flow architecture. This is non-negotiable — every project follows this layout:

```
project-name/                      # ← Git repo lives HERE
├── .git/
├── .gitignore                     # Whitelist: tracks only scripts/
├── read/                          # Raw input data — IMMUTABLE, never modified by code
│   └── reference/                 # Reference files (marker lists, GTFs, gene sets, etc.)
├── scripts/                       # ← All tracked code lives here
│   ├── .gitignore                 # Script-specific ignores (rendered outputs, caches, etc.)
│   ├── README.md
│   ├── project-name.Rproj         # RStudio project file (sets working directory)
│   ├── utils.R                    # Shared helper functions
│   ├── 01-first-step.qmd
│   ├── 02-second-step.qmd
│   └── ...
├── checkpoints/                   # RDS intermediates — FLAT, shared across scripts
│   ├── 01-merged-harmony.rds
│   └── 02-annotated.rds
└── write/                         # All pipeline outputs — PER-SCRIPT SUBDIRECTORIES
    ├── figures/
    │   ├── 01-first-step/         #   Subdirectory named after the qmd basename
    │   │   ├── 01-qc-violin.pdf   #   Each file still carries the step prefix
    │   │   └── 01-elbow-plot.pdf
    │   └── 02-second-step/
    │       └── 02-umap-condition.pdf
    └── tables/
        ├── 01-first-step/
        │   └── 01-qc-summary.csv
        └── 02-second-step/
            └── 02-condition-markers.csv
```

### The read / write / checkpoints triad

These three directories define how data flows through the project:

- **`read/`** — raw input data and reference files. Code reads from here but **never writes to it**. This directory is the immutable ground truth. Subdirectories organise by sample or data source (e.g., `read/bmls-4/filtered_feature_bc_matrix/`). Reference files like marker databases, gene annotations, and GTFs go in `read/reference/`.
- **`checkpoints/`** — serialised R objects (`.rds`) that pass state between pipeline steps. Each notebook reads its predecessor's checkpoint and writes its own. These are the pipeline's internal handoff mechanism. **Flat layout** with step-prefixed filenames: `01-merged-harmony.rds`, `02-annotated.rds`. Stay flat because every downstream notebook reads checkpoints — subdirectories would obscure the shared-state nature. Within a notebook, intermediate checkpoints enable self-contained chunks: `04-fibro-subset.rds` → `04-fibro-sct.rds` → `04-fibro-harmony.rds` → `04-fibro-states.rds`.
- **`write/`** — all human-facing outputs. Split into `write/figures/` (plots saved via `safe_ggsave()`) and `write/tables/` (CSVs, Excel files saved via `readr::write_csv()`, `openxlsx`). **Per-script subdirectories**: every notebook writes into its own folder named after the qmd basename (`write/figures/<NN-script-slug>/`, `write/tables/<NN-script-slug>/`), and the files inside still carry the matching `NN-` step prefix on their face. Dual provenance (folder + filename prefix) so a folder listing tells you which script produced everything, and an individual file copied elsewhere still self-identifies.

### Why git at root with a scripts/ whitelist

- Git at root means one repo per project — no nested `.git/` confusion, clean `git clone` into the project root
- The whitelist `.gitignore` (`/*` then `!/scripts`) means only `scripts/` is ever tracked — everything else is silently ignored by default
- Adding new data directories (`read/sample-2/`, `outputs/`) never requires updating `.gitignore`
- `read/` is immutable raw data — large, binary, doesn't belong in git
- `checkpoints/` are ephemeral intermediates that change every re-run
- `write/` contains generated outputs that are reproducible from code

## Naming Conventions

### File naming: hyphens

All filenames use **hyphens** (`-`): `01-qc-integration.qmd`, `02-umap-singler.pdf`, `04-fibro-states.rds`

### Code naming: underscores

All R variable and function names use **snake_case** (`_`): `seu_integrated`, `marker_results_df`, `fix_md_rownames()`

Underscores are for code identifiers only — never for filenames or directory names. `naming-conventions` holds the project-wide rule and the dated-asset pattern.

### Output organisation

Outputs use a **dual provenance** scheme: a per-script subdirectory groups everything one notebook produced, and each file inside still carries the script's `NN-` step prefix so it self-identifies if copied elsewhere.

**Per-script subdirectories** for `write/figures/` and `write/tables/`:
- The subdirectory name matches the qmd basename (without `.qmd`)
- Concrete example for `scripts/02-tables-visualisations.qmd`:
  - `../write/figures/02-tables-visualisations/02-hypoxia-marker-violins.pdf`
  - `../write/figures/02-tables-visualisations/02-condition-marker-heatmap.pdf`
  - `../write/tables/02-tables-visualisations/02-condition-markers-significant.csv`
- The script's `setup` chunk `dir.create()`s these directories once, at the top of the document (see `writing-r-code` for the setup chunk template)

**Step-prefixed filenames** inside the subdirectory:
- Figures: `01-elbow-pca.pdf`, `02-umap-singler.pdf`
- Tables: `01-qc-summary.csv`, `04-fibro-markers-all.csv`

**Checkpoints stay flat** in `checkpoints/` — never per-script subdirectories:
- `01-merged-harmony.rds`, `02-annotated.rds`, `04-fibro-states.rds`
- Reason: downstream notebooks all read upstream checkpoints; per-script subdirs would force every reader to know which notebook wrote which file

**Why dual provenance:**
- The folder grouping makes "find every output from script 02" a one-line `ls`
- The filename prefix means an individual `02-condition-marker-heatmap.pdf` copied into a paper draft or shared with a collaborator still announces its origin

### Variable descriptiveness

Variable names must make their type and purpose obvious at a glance:

| Bad | Good | Why |
|-----|------|-----|
| `obj`, `seu` | `seu_integrated`, `seu_fibro` | What Seurat object is this? |
| `objs` | `seu_list` | It's a list of Seurat objects |
| `md` | `metadata_df` | It's a metadata data frame |
| `mat` | `expr_matrix` | It's an expression matrix |
| `p`, `p1` | `plot_umap_clusters`, `plot_violin_state` | What does the plot show? |
| `res` | `go_enrichment`, `marker_results_df` | Results of what? |
| `sid`, `i`, `ct` | `sample_name`, `cluster_id`, `cell_type` | Loop iterators describe what they iterate |
| `df`, `tab` | `composition_df`, `regulon_diagnostic_df` | What data does it hold? |

Keep it practical — 2-3 words max. `seu_fibro` not `seurat_object_containing_fibroblast_subset`.

## Path Conventions

- All paths are **relative** from `scripts/`: `../read/`, `../checkpoints/`, `../write/figures/`
- Paths appear **inline at point of use** — never defined as upfront variables
- Exception: external dependencies the user must provide (e.g., a GTF file path) get a clearly commented variable at the top of the relevant chunk

```r
# Correct — inline at point of use, figures go in the per-script subdir
seu_integrated <- readr::read_rds("../checkpoints/01-merged-harmony.rds")
safe_ggsave("../write/figures/02-second-step/02-umap-singler.pdf",
            plot_umap, width = 8, height = 6)

# Incorrect — upfront path variables
checkpoint_dir <- "../checkpoints/"
fig_dir <- "../write/figures/"

# Incorrect — flat write/figures/ (no per-script subdir)
safe_ggsave("../write/figures/02-umap-singler.pdf", plot_umap, width = 8, height = 6)
```

## Pipeline Structure

### Numbered QMD notebooks

Each analysis step is a separate QMD file. Notebooks are numbered sequentially and pass data via RDS checkpoints:

```
01-first-step.qmd  →  ../checkpoints/01-output.rds
02-second-step.qmd →  reads 01-output.rds, writes ../checkpoints/02-output.rds
03-third-step.qmd  →  reads 02-output.rds, writes ../checkpoints/03-output.rds
```

### Shared utils.R

Helper functions used across multiple notebooks go in `utils.R`:
- Every QMD sources it: `source("utils.R")`
- Functions are grouped by category with `# Category ----------` headers
- No `rm(list = ls())` or side effects — pure utility functions only

### Setup chunk

Every notebook opens with one `setup` chunk, under its own `## Setup` heading, holding the `conflicted` preferences and every output directory the script writes to. See `writing-r-code` for the full template.

```r
# Libraries ----------
library(conflicted)

# Conflicts ----------
conflicts_prefer(
  dplyr::filter, dplyr::count, dplyr::rename, dplyr::slice,
  purrr::reduce,
  base::intersect, base::setdiff, base::union, base::setequal,
  .quiet = TRUE
)

# Output directories ----------
dir.create("../checkpoints", recursive = TRUE, showWarnings = FALSE)

dir.create("../write/figures/02-second-step",
           recursive = TRUE, showWarnings = FALSE)

dir.create("../write/tables/02-second-step",
           recursive = TRUE, showWarnings = FALSE)
```

### Self-contained code chunks

Every code chunk must be independently runnable **once the setup chunk has run**. Follow the pattern from `writing-r-code`. **Processing chunks** save intermediate checkpoints; **visualisation chunks** load the latest checkpoint and save figures into the per-script output subdirectory that setup already created.

Processing chunk (writes to flat `checkpoints/`):

```r
# Libraries ----------
library(Seurat)
library(readr)

source("utils.R")

# Inputs ----------
seu <- readr::read_rds("../checkpoints/01-merged-harmony.rds")

# Processing ----------
seu <- NormalizeData(seu)

# Outputs ----------
readr::write_rds(seu, "../checkpoints/01-normalized.rds")
```

Visualisation chunk (writes to per-script subdir; chunk lives in `02-second-step.qmd`):

```r
# Libraries ----------
library(Seurat)
library(BadranSeq)
library(ggplot2)
library(readr)

source("utils.R")

# Inputs ----------
seu <- readr::read_rds("../checkpoints/01-normalized.rds")

# Processing ----------
plot_umap <- do_UmapPlot(seu, group.by = "celltype")

# Outputs ----------
ggsave("../write/figures/02-second-step/02-umap-celltype.pdf",
       plot_umap, width = 8, height = 6, bg = "white")
```

### Prose format

Section descriptions use **bullet points with bold key terms**, never paragraphs:

```markdown
## Quality Control

- **QC metrics**: `nFeature_RNA`, `nCount_RNA`, `percent.mt`
  - Mitochondrial fraction flags dying cells leaking cytoplasmic mRNA
- **Thresholds**: 200 ≤ nFeature ≤ 6000, nCount ≥ 500, percent.mt ≤ 20
  - Conservative but standard for 10x Chromium data
- **Rationale**: fixed thresholds chosen over MAD-based for reproducibility across batches
```

## Version Control Setup

### Initialise git at project root

```bash
cd project-name/
git init
git branch -m main
```

### Root .gitignore — whitelist approach

Ignores everything by default; only `scripts/` is tracked:

```gitignore
# Ignore everything
/*
/.*

# Whitelist: track only scripts/
!/scripts
```

### Tracking empty data directories with `.gitkeep`

Git does not track empty directories. To make `read/`, `checkpoints/`, and `write/` appear in the repo (so collaborators see the expected structure without committing any data), place an empty `.gitkeep` placeholder file in each:

```bash
touch read/.gitkeep checkpoints/.gitkeep write/figures/.gitkeep write/tables/.gitkeep
```

Then un-ignore only those placeholders in the root `.gitignore`:

```gitignore
# Ignore everything
*
*/

# Whitelist: tracked code and docs
!scripts/
!scripts/**
!docs/
!docs/**

# Expose .gitkeep placeholders so directory structure is visible in the repo
# (directory contents remain gitignored — data never touches the repo)
!read/.gitkeep
!checkpoints/.gitkeep
!write/figures/.gitkeep
!write/tables/.gitkeep

# Belt-and-braces: explicitly ignore directory contents even if rules above change
read/*
!read/.gitkeep
checkpoints/*
!checkpoints/.gitkeep
write/figures/*
!write/figures/.gitkeep
write/tables/*
!write/tables/.gitkeep
```

**Why this works:** `.gitkeep` is just an empty file — git has no special treatment for it. The name is a community convention. Once real data files land in the directory, `.gitkeep` sits there harmlessly and can be deleted if desired.

**Why `!*/` is required:** The `*` rule ignores everything, including directories. When git hits an ignored directory it stops traversing into it entirely — so `!read/.gitkeep` alone would never fire because git would never enter `read/` to see it. The `!*/` rule un-ignores all directories globally, allowing git to walk into all of them. Once inside, `read/*` + `!read/.gitkeep` controls what gets tracked.

**Why tracked `scripts/` needs both `!scripts/` and `!scripts/**`:**

`!*/` already lets git traverse `scripts/`, but it doesn't un-ignore the files inside. You still need:

```gitignore
!scripts/    # un-ignore the directory itself (redundant with !*/ but explicit)
!scripts/**  # un-ignore everything found inside at any depth
```

`!scripts/**` alone is not enough without `!*/` or `!scripts/` first — git won't enter an ignored directory. And `*` (single wildcard) only matches one level deep — `**` is needed for recursive un-ignoring.

### scripts/.gitignore — script-specific ignores

Place a second `.gitignore` inside `scripts/` to exclude rendered outputs and caches:

```gitignore
# Rendered outputs
*.html
*_files/

# Quarto cache and intermediates
.quarto/
_freeze/
_site/

# R intermediates
*.Rhistory
.Rproj.user/
.RData
.Rapp.history

# Knitr cache
*_cache/

# OS files
.DS_Store
Thumbs.db
```

### .Rproj template

```
Version: 1.0

RestoreWorkspace: No
SaveWorkspace: No
AlwaysSaveHistory: No

EnableCodeIndexing: Yes
UseSpacesForTab: Yes
NumSpacesForTab: 2
Encoding: UTF-8

AutoAppendNewline: Yes
StripTrailingWhitespace: Yes
```

### README structure

The README must clearly separate tracked vs untracked content:

1. **Project overview** — biological/scientific context
2. **Pipeline overview** — DAG or numbered list of analysis steps
3. **Per-step descriptions** — what each notebook does
4. **What is tracked** — tree of repo contents with one-line descriptions
5. **What is NOT tracked** — tree of parent directory with setup instructions
6. **Replication steps** — exact `git clone`, `mkdir`, and data placement commands
7. **Dependencies** — R packages with categories
8. **Code style** — brief summary of conventions

### GitHub repo creation

```bash
cd project-name/
git add scripts/
git commit -m "Initial commit: <N>-step <analysis-type> pipeline with shared utils"
gh repo create <descriptive-project-name> --private --source=. --push \
  --description "<one-line description of the analysis>"
```

Repo name should describe the science, not the tool: `capsular-contracture-scrna` not `seurat-analysis-pipeline`.

## Scaffolding Checklist

When setting up a new project:

1. Create parent directory structure (`read/`, `read/reference/`, `scripts/`, `checkpoints/`, `write/figures/`, `write/tables/`)
2. Move/organise raw data into `read/`
3. Move reference files into `read/reference/`
4. Archive any existing scripts to `.original_student_code/` or similar hidden directory
5. Create `.Rproj` in `scripts/`
6. Create `utils.R` in `scripts/` with shared helpers
7. Create numbered QMD notebooks in `scripts/`, each opening with a `## Setup` section and its `setup` chunk
8. Initialise git at project root; create root `.gitignore` (whitelist) and `scripts/.gitignore` (script-specific ignores)
10. Write `README.md`
11. Create GitHub repo and push

## Common Mistakes

| Mistake | Correct Approach |
|---------|-----------------|
| `git init` inside `scripts/` | `git init` at project root with whitelist `.gitignore` |
| Defining path variables upfront | Paths inline at every function call |
| Paragraph prose in QMD sections | Bullet points with bold key terms |
| Generic variable names (`obj`, `df`, `p`) | Descriptive names (`seu_fibro`, `composition_df`, `plot_umap_clusters`) |
| Chunks that depend on previous chunk state | Every chunk loads its own inputs from disk |
| Output files without step prefix | Always prefix: `01-`, `02-`, etc. |
| Saving figures / tables flat under `write/figures/` or `write/tables/` | Use per-script subdirs: `write/figures/<NN-script-slug>/<NN-name>.pdf` and `write/tables/<NN-script-slug>/<NN-name>.csv` |
| Forgetting to `dir.create()` the per-script output subdir | The script's `setup` chunk creates every output directory it writes to, once |
| Scattering `conflicts_prefer()` and `dir.create()` across every chunk | Both belong in the single `setup` chunk at the top of the notebook |
| Per-script subdirs under `checkpoints/` | Checkpoints stay **flat** because downstream notebooks all read them: `checkpoints/02-annotated.rds`, not `checkpoints/02-second-step/02-annotated.rds` |
| `readRDS()`/`saveRDS()` | `readr::read_rds()`/`readr::write_rds()` |
| `%>%` magrittr pipe | `\|>` native pipe |
| `here::here()` for paths | Relative `../` from `scripts/` |
| Mixing hyphens and underscores in filenames | Hyphens for files, underscores for code (see `naming-conventions`) |