Metadata-Version: 2.4
Name: spacerextractor
Version: 0.9.9
Summary: SpacerExtractor: Extract CRISPR spacers from your metagenomic reads
Author-email: Simon Roux <sroux@lbl.gov>, Uri Neri <uneri@lbl.gov>, Brian Bushnell <bbushnell@lbl.gov>
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Project-URL: Home, https://code.jgi.doe.gov/SRoux/spacerextractor

##
![](https://code.jgi.doe.gov/SRoux/spacerextractor/-/raw/main/logo.png)


[![Conda](https://img.shields.io/conda/vn/bioconda/spacerextractor.svg?label=Conda&color=green)](https://anaconda.org/bioconda/spacerextractor)
[![Conda downloads](https://img.shields.io/conda/dn/bioconda/spacerextractor.svg?label=Conda%20downloads&color=orange)](https://anaconda.org/bioconda/spacerextractor)
[![Docker pulls](https://img.shields.io/docker/pulls/simroux/spacerextractor.svg?label=Docker%20pulls)](https://hub.docker.com/r/simroux/spacerextractor)

# SpacerExtractor

## Overview

The SpacerExtractor tool is meant to robustly identify and extract CRISPR spacers from metagenome short reads. Working from a database of known CRISPR repeats, SpacerExtractor quickly scans short reads for the corresponding repeat sequences, extract the potential spacer between two repeats, apply several quality control, denoising, and clustering steps, and provides a full non-redundant complement of spacers for each detected repeat. Because of the high variability observed at CRISPR loci, this read mining approach typically recovers a much larger diversity of spacers than can be found in assembled contigs.
SpacerExtractor also includes commands to run CRISPR-Cas Typer on a new set of genomes or MAGs, and add newly predicted repeats to the repeat database.

**Questions ? Issues to report ?** Send an email to jgi.gitlab.echo+sroux-spacerextractor-633-issue- at gmail.com

## Example of workflows using SpacerExtractor

- From a new metagenome, identify CRISPR spacers, and match them to predicted viruses
1. Install spacerextractor, and download the repeat databases
2. (optional): run ``spacerextractor run_cctyper`` and ``spacerextractor build_database`` on new genomes/MAGs, to automatically predict and add to the database local CRISPR arrays
3. run ``spacerextractor extract_spacers`` and ``spacerextractor filter_hq_spacers`` on the metagenome reads, to identify and QC spacers
4. run ``spacerextractor create_target_db`` on the predicted viruses, and ``spacerextractor map_to_target`` with QC'd spacers and this new virus database to identify hits between spacers and viruses.  

- From a new virus genome (or plasmid, or else), check for spacer hits based on a database of known spacers
1. From https://spacers.jgi.doe.gov/quick_start, download a [fasta file of known spacers](https://portal.nersc.gov/dna/microbial/prokpubs/spacer_database_resources/nr_spacers_hq-all_25-05-10.fna.gz)
2. run ``spacerextractor create_target_db`` on the predicted viruses, and ``spacerextractor map_to_target`` with QC'd spacers and this new virus database to identify hits between spacers and viruses.  
3. see [this notebook](https://spacers.jgi.doe.gov//site/notebooks/hits_info.html) to see how spacer identifiers can then be connected to individual CRISPR repeats, taxon, ecosystem, and samples.  

See below for more detailed instructions on each step.  

## Installation / Requirements

The latest stable version of SpacerExtractor can be installed via bioconda, as follows:
```
$ mamba create -c bioconda -n spacerextractor spacerextractor
$ conda activate spacerextractor
$ spacerextractor --help
                                                                                                                                                                                        
 Usage: spacerextractor [OPTIONS] COMMAND [ARGS]...                                                                                                                                     
                                                                                                                                                                                        
 🚜🧬 Spacer Extractor 🚜🧬 : extract spacers from metagenomic reads.                                                                                                                                             
 The main commands are:                                                                                                                                                                                                                                 
 [*] download_database to download the latest version of the CRISP repeat database                                                                                                                                                                      
 [*] extract_spacers and filter_hq_spacers to extract spacers from metagenome reads and only retain the high-quality ones                                                                                                                               
 [*] run_cctyper and build_database to add repeats predicted de novo (e.g. in new genomes/MAGs) to the repeat database                                                                                                                                  
 [*] create_target_db and map_to_target to compare spacers to a database of potential targets                                                                                                                       
                                                                                                                       
```

Alternatively, the latest development version can be installed from this repo as follows:
```
$ git clone https://code.jgi.doe.gov/SRoux/spacerextractor.git
## or download the zip while from "Code" on top right
$ cd spacerextractor
$ mamba env create -f spacerextractor.yaml
$ mamba activate spacerextractor_dev
$ flit install
$ spacerextractor

 Usage: spacerextractor [OPTIONS] COMMAND [ARGS]...                                                                                                                                     

 🚜🧬 Spacer Extractor 🚜🧬 : extract spacers from metagenomic reads.                                                                                                                   
...
│ build_database                 add new repeats identified by CRISPR-Cas typer to the repeat database                   │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

```

## Download the standard database

SpacerExtractor comes with a database of repeats predicted from GTDBr214, GenBank, and IMG genomes. This can be downloaded as follows:
```
$ conda activate spacerextractor
$ spacerextractor download_database
```

By default, the database will be installed in the local directory (as a folder named "SE_Db_..."). This can be changed with the option "--out_dir".

### (Optional) Example dataset

A test dataset is available at the following url:
```
$ wget https://portal.nersc.gov/dna/microbial/prokpubs/spacer_extractor/SE_Test_data.tar.gz
$ tar -xvf SE_Test_data.tar.gz
```

## Main functions - extract spacers from a read file using an existing database, and refine the predictions

This is the core of SpacerExtractor.
The first command <span style="color:#8e44ad">extract_spacers</span> takes as input the database folder (either the default one downloaded above, or one generated with "build_db", see below), along with a read file in fastq.gz format, and establishes a non-redundant list of candidate spacers linked to one (or more) repeat(s) from the database, with a summary table available as Extracted_spacers.all.nr.wrepeat_info.tsv.
The reads should be in a single fastq file (can be compressed). If paired-end, this should be an interleaved fastq. This can be done with bbtools reformat, i.e. "reformat.sh in1=read1.fq.gz in2=read2.fq.gz out=reads.fq.gz".
Then the second command <span style="color:#8e44ad">filter_hq_spacers</span> will proceed with a denoising of these candidate spacers to minimize the presence of "fake" variants stemming from sequencing error, and provides the sequence and information of the final set of non-redundant spacers in Final_spacers.nr.denoised.fna and Final_spacers.nr.denoised.info.tsv. If possible, this step will also apply default (conservative) quality cut-offs and provide the "best-quality" spacers in Final_spacers.nr.denoised.hq.fna and Final_spacers.nr.denoised.hq.info.tsv. Finally, if some spacers are connected to multiple arrays, these connections will be listed in Final_spacers.nr.denoised.multiarray.info.tsv.

Example:
```
$ conda activate spacerextractor
$ spacerextractor extract_spacers -d SE_Db_r214GBIMG_0.9/ -o SE_example_results/ -f Test_package/Example_reads.fastq.gz -t 4

[13:40:58] INFO     Looks like we have everything to start, so we go ahead                                                                                                              
           INFO     Extracting relevant reads from /.../Test_package/Example_reads.fastq.gz based on the repeat sequence database, using bbduk. Results will go into                    
...                                                                                                                         
           INFO      ... Processing spacer number 4300                                                                                                                                  
[13:46:49] INFO      ... Processing spacer number 4400      

$ spacerextractor filter_hq_spacers -d SE_Db_r214GBIMG_0.9/ -w SE_example_results/ -t 4

[13:49:56] INFO     Loading results from previous steps...                                                                                                                                                                                              
[13:49:58] INFO     First attempt to denoise repeats and remove/adjust the ones that are likely incorrect (originating from another repeat with sequencing error)                                                                                       
           INFO     Denoising repeat sequences based on total predicted array sizes (i.e. repeat coverage)  
...
[13:50:25] INFO     Because they are not really 'high-quality' but also are not necessarily 'low-quality' either..  

```

If you suspect that your input reads may have a higher error rate than the most recent short-read sequencing devices, you can use the options "--denoising_strict" or "--denoising_very_strict" at the filter_hq_spacers step, to set the expected sequence error rate at a higher value and remove more spacers predicted to be sequencing artefacts.

```
$ spacerextractor filter_hq_spacers -d SE_Db_r214GBIMG_0.9/ -w SE_example_results-strict/ -t 4 --denoising_strict
[14:13:24] INFO     Note: the expected sequencing error rate will be set to 0.005 because of the 'denoising_aggressive' flag                                                            
           INFO     Loading results from previous steps...                                                                   
...
[14:26:47] INFO     And exporting separately good quality spacers connected to multiple arrays into /.../SE_results-strict/Final_spacers.nr.denoised.multiarray.fna and                 
                               /.../SE_results-strict/Final_spacers.nr.denoised.multiarray.info.tsv                 
INFO     Because they are not really 'high-quality' but also are not necessarily 'low-quality' either..          
```


## Extra functions - refine repeats predicted by CRISPR-Cas Typer, then create a database combining known repeats with these new de novo predictions

**Note - this is still in beta and has not been extensively tested, use at your own risk / ymmv**

These extra functions can be used if you want to enrich the repeat database with CRISPR arrays detected de novo in your own genomes/MAGs. The first step <span style="color:#8e44ad">run_cctyper</span> is run CRISPR-Cas Typer on these genomes to predict CRISPR arrays. The best predicted arrays are selected, and the exact repeat sequence is refined to avoid cases in which some parts of the repeat in 5' or 3' are missing.
The second step <span style="color:#8e44ad">build_database</span> will then merge these new repeats with an existing database (in this case SE_Db_r214GBIMG_0.9), and create a new custom database including a non-redundant set of all repeats. The path "/.../out_dir_newdb" can then be used as a database folder in <span style="color:#8e44ad">extract_spacers</span> (see above).

**Note - the input must be nucleotide contigs (from MAGs, or selected contigs from a metagenome), in fasta format (no compression). If there are multiple fasta files in the folder, they will all get processed**

```
$ conda activate spacerextractor
$ spacerextractor run_cctyper -i Test_package/Example_MAGs/ -w SE_example_mags_addition/ -p -t 4

[14:37:28] INFO     ## Running CCTyper.. ##
...                                                                                 
[14:39:46] INFO     ## All done ! ##          

$ spacerextractor build_database -d SE_Custom_db/ -c SE_example_mags_addition/all_refined_repeats.tab -r SE_Db_r214GBIMG_0.9/ -t 4

[14:40:50] INFO     Building a repeat dir in SE_Custom_db/ based on /.../SE_Db_r214GBIMG_0.9/Arrays_info.tsv                                                                                      
           INFO     ## Starting to build the database ##
...                                                             
[14:45:58] INFO     All finished, you should now be able to use SE_Custom_db as the --repeat_db_dir to extract_spacers

```

If you want to instead process entire metagenomes (rather than MAGs), our recommendation is to first run a fast tool like Minced (https://github.com/ctSkennerton/minced) on the metagenome contigs, prepare a folder with all contigs encoding a putative CRISPR array in a single fasta file, and use this folder as the "MAG" input. That way, you avoid having to run CRISPRCasTyper on all contigs in your metagenome, and still recover potential CRISPR arrays from your metagenome assembly. Note - This *should* work, but is still to be tested with different datasets.


## Extra functions - map spacers to potential targets at scale

**Note - this has not yet been extensively tested, please report any unexpected issue or result**

These functions can be used if you want to identify potential spacer hits between a set of spacers (provided as a fasta file) and a set of potential targets, e.g. virus genomes (provided as a fasta file). To recover all spacer hits in a timely manner, these functions leverage **bowtie1**, using spacers instead of reads to be mapped against a reference, and a global alignment approach. This means that only hits with up to 3 mismatches across the entire spacer are reported. The first step <span style="color:#8e44ad">create_target_db</span> takes as input the potential targets (as a fasta file) and prepare the corresponding database. The second setp <span style="color:#8e44ad">map_to_target</span> will then map spacers to this target database, and provides a tsv file listing all spacer hits as output.

```
$ conda activate spacerextractor
$ spacerextractor create_target_db -i Test_package/Example_virus_targets.fna -d virus_targets_db -t 4 --replace_spaces

[14:51:54] INFO     Building a target dir in /.../mydir/virus_targets_db
...                                                                                 
           INFO     Preparing the bowtie-1 database (this can be a bit long, please be patient)

$ spacerextractor map_to_target -i SE_example_results/Final_spacers.nr.denoised.hq.fna -d virus_targets_db -o SE_example_spacers_to_viruses/ -t 4

[14:52:12] INFO     Mapping spacers to targets into /.../mydir/spacers_to_viruses                                                                                                     
           INFO     Mapping spacers to potential target
...                                                             
           INFO     Writing output TSV file...

```
Note that you would typically then check for (and remove) edge cases, such as spacer hits due to CRISPR arrays encoded on 
the potential targets, or spacer hits associated with low-complexity regions (already done to an extent here with dustmasker, but could
always be done more thoroughly). You can check an example of post-processing done in [this script](https://github.com/simroux/globalspacers_scripts/tree/main/Analyses/Target_IMGVR_IMGPR#post-process-spacer-matches-and-select-relevant-hits).  

## Copyright

SpacerExtractor Copyright (c) 2024, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Intellectual Property Office at IPO@lbl.gov.

NOTICE.  This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights.  As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit others to do so.

