ART_ILLUMINA README (updated on 06/6/2016) Weichun Huang ART_Illumina (2008-2016), Q Version 2.5.8 (Jun 6, 2016) DESCRIPTION ART (art_Illumina Q version) is a simulation program to generate sequence read data of Illumina sequencers. ART generates reads according to the empirical read quality profile summarized from large real read data. ART has been using for testing or benchmarking a variety of methods or tools for next-generation sequencing data analysis, including read alignment, de novo assembly, detection of SNP, CNV, or other structure variation. art_Illumina can generate single-end, paired-end, mate-pair reads, and amplicon sequencing simulation of Illumina sequencing platform. Its outputs include FASTQ read, ALN and/or SAM alignment files. ALN files can also be converted to UCSC BED files by using aln2bed.pl program included. art_Illumina comes with the tool art_profiler_illumina that can generate quality profiles from Illumina sequencing data in the fastq format. The tool is in the folder ART_profiler_illumina. Please see README in the folder for the details and usage. EXAMPLES In the "examples" subdirectory, the shell script "run_test_examples_illumina.sh" gives four examples of using ART for read simulation. To test these four examples, just run the script "run_test_examples_illumina.sh" USAGE RECOMMENDED USAGES (specifying a sequencing system) art_illumina [options] -ss -sam -i -l -f -o art_illumina [options] -ss -sam -i -l -c -o art_illumina [options] -ss -sam -i -l -f -m -s -o art_illumina [options] -ss -sam -i -l -c -m -s -o OTHER USAGES art_illumina [options] -sam -i -l -f -o art_illumina [options] -sam -i -l -c -o art_illumina [options] -sam -i -l -f -m -s -o art_illumina [options] -sam -i -l -c -m -s -o ===== PARAMETERS ===== -1 --qprof1 the first-read quality profile -2 --qprof2 the second-read quality profile -amp --amplicon amplicon sequencing simulation -c --rcount number of reads/read pairs to be generated per sequence(not be used together with -f/--fcov) -d --id the prefix identification tag for read ID -ef --errfree indicate to generate the zero sequencing errors SAM file as well the regular one NOTE: the reads in the zero-error SAM file have the same alignment positions as those in the regular SAM file, but have no sequencing errors -f --fcov the fold of read coverage to be simulated or number of reads/read pairs generated for each amplicon -h --help print out usage information -i --in the filename of input DNA/RNA reference -ir --insRate the first-read insertion rate (default: 0.00009) -ir2 --insRate2 the second-read insertion rate (default: 0.00015) -dr --delRate the first-read deletion rate (default: 0.00011) -dr2 --delRate2 the second-read deletion rate (default: 0.00023) -k --maxIndel the maximum total number of insertion and deletion per read (default: up to read length) -l --len the length of reads to be simulated -m --mflen the mean size of DNA/RNA fragments for paired-end simulations -mp --matepair indicate a mate-pair read simulation -M --cigarM indicate to use CIGAR 'M' instead of '=/X' for alignment match/mismatch -nf --maskN the cutoff frequency of 'N' in a window size of the read length for masking genomic regions NOTE: default: '-nf 1' to mask all regions with 'N'. Use '-nf 0' to turn off masking -na --noALN do not output ALN alignment file -o --out the prefix of output filename -p --paired indicate a paired-end read simulation or to generate reads from both ends of amplicons NOTE: art will automatically switch to a mate-pair simulation if the given mean fragment size >= 2000 -q --quiet turn off end of run summary -qL --minQ the minimum base quality score -qU --maxQ the maxiumum base quality score -qs --qShift the amount to shift every first-read quality score by -qs2 --qShift2 the amount to shift every second-read quality score by NOTE: For -qs/-qs2 option, a positive number will shift up quality scores (the max is 93) that reduce substitution sequencing errors and a negative number will shift down quality scores that increase sequencing errors. If shifting scores by x, the error rate will be 1/(10^(x/10)) of the default profile. -rs --rndSeed the seed for random number generator (default: system time in second) NOTE: using a fixed seed to generate two identical datasets from different runs -s --sdev the standard deviation of DNA/RNA fragment size for paired-end simulations. -sam --samout indicate to generate SAM alignment file -sp --sepProf indicate to use separate quality profiles for different bases (ATGC) -ss --seqSys The name of Illumina sequencing system of the built-in profile used for simulation NOTE: all built-in sequencing system ID names are: GA1 - GenomeAnalyzer I (36bp,44bp) GA2 - GenomeAnalyzer II (50bp, 75bp) HS10 - HiSeq 1000 (100bp) HS20 - HiSeq 2000 (100bp) HS25 - HiSeq 2500 (125bp, 150bp) HSXn - HiSeqX PCR free (150bp) HSXt - HiSeqX TruSeq (150bp) MinS - MiniSeq TruSeq (50bp) MSv1 - MiSeq v1 (250bp) MSv3 - MiSeq v3 (250bp) NS50 - NextSeq500 v2 (75bp) ===== NOTES ===== * ART by default selects a built-in quality score profile according to the read length specified for the run. * For single-end simulation, ART requires input sequence file, outputfile prefix, read length, and read count/fold coverage. * For paired-end simulation (except for amplicon sequencing), ART also requires the parameter values of the mean and standard deviation of DNA/RNA fragment lengths ===== EXAMPLES ===== 1) single-end read simulation art_illumina -ss HS25 -sam -i reference.fa -l 150 -f 10 -o single_dat 2) paired-end read simulation art_illumina -ss HS25 -sam -i reference.fa -p -l 150 -f 20 -m 200 -s 10 -o paired_dat 3) mate-pair read simulation art_illumina -ss HS10 -sam -i reference.fa -mp -l 100 -f 20 -m 2500 -s 50 -o matepair_dat 4) amplicon sequencing simulation with 5' end single-end reads art_illumina -ss GA2 -amp -sam -na -i amp_reference.fa -l 50 -f 10 -o amplicon_5end_dat 5) amplicon sequencing simulation with paired-end reads art_illumina -ss GA2 -amp -p -sam -na -i amp_reference.fa -l 50 -f 10 -o amplicon_pair_dat 6) amplicon sequencing simulation with matepair reads art_illumina -ss MSv1 -amp -mp -sam -na -i amp_reference.fa -l 150 -f 10 -o amplicon_mate_dat 7) generate an extra SAM file with zero-sequencing errors for a paired-end read simulation art_illumina -ss HSXn -ef -i reference.fa -p -l 150 -f 20 -m 200 -s 10 -o paired_twosam_dat 8) reduce the substitution error rate to one 10th of the default profile art_illumina -i reference.fa -qs 10 -qs2 10 -l 50 -f 10 -p -m 500 -s 10 -sam -o reduce_error 9) turn off the masking of genomic regions with unknown nucleotides 'N' art_illumina -ss HS20 -nf 0 -sam -i reference.fa -p -l 100 -f 20 -m 200 -s 10 -o paired_nomask 10) masking genomic regions with >=5 'N's within the read length 50 art_illumina -ss HSXt -nf 5 -sam -i reference.fa -p -l 150 -f 20 -m 200 -s 10 -o paired_maskN5 READ QUALITY PROFILE TOOL FOR CREATING A NEW QUALITY PROFILE art_profiler_illumina in the folder ART_profiler_illumina can generate quality profiles from Illumina fastq data. Please see README in the folder for the usage. FORMAT A valid quality score profile is tab-delimited and has no specific header line. Headers can be included if each line of extraneous information begins with a number sign(#). Each line of actual quality profile information must begin with an identifier that indicates where the data comes from for the remainder of the line. Identifiers include: . The identifier for the combination of all base information. A The identifier for quality scores associated with A calls only. T The identifier for quality scores associated with T calls only. G The identifier for quality scores associated with G calls only. C The identifier for quality scores associated with C calls only. Following the identifier on a given line must be the position number indicating where the rest of the information on that line applies within a given fragment. The data must be arrayed in pairs such that each line in the pair has the same identifier and position number. The first line in a pair is a list of quality scores in ascending order and the second line are the corresponding cumulative frequencies of the quality scores. EXAMPLE: . 0 3 6 7 8 9 10 11 12 13 14 15 . 0 39375 355755 395136 415685 1227131 1338634 1522001 1851208 2165909 2436839 2608538 ... . 35 4 5 6 7 8 9 10 11 12 13 14 . 35 434262 1341151 1725690 2979293 3478620 3592624 3672807 3873754 4096922 4983957 6111261 A 0 3 6 7 8 9 10 11 12 13 14 15 A 0 560 99637 111485 119727 389899 412150 458066 572958 665153 745916 793532 BUILT-IN PROFILES Under subfolder "Illumina_Profiles" are the read profiles of Illumina GAII sequencers including all ART's built-in profiles as indicated below. 1) Raw quality profiles GA1 36bp reads Emp36R1.txt Emp36R2.txt GA1 44bp reads Emp44R1.txt Emp44R2.txt GA2 50bp reads Emp50R1.txt Emp50R2.txt GA2 75bp reads Emp75R1.txt Emp75R2.txt MiSeq 250bp reads (250bp reads) EmpMiSeq250R1.txt EmpMiSeq250R2.txt HiSeq1000 100bp reads Emp100R1.txt Emp100R2.txt HiSeq2000 100bp reads (the new default profile for 100bp reads) HiSeq2kL100R1.txt HiSeq2kL100R2.txt HiSeq2500 125bp reads HiSeq2500L125R1.txt HiSeq2500L125R2.txt HiSeq2500 150bp reads HiSeq2500L150R1.txt HiSeq2500L150R2.txt HiSeqX PCR free (150bp) HiSeqXPCRfreeL150R1.txt HiSeqXPCRfreeL150R2.txt HiSeqX TruSeq (150bp) HiSeqXtruSeqL150R1.txt HiSeqXtruSeqL150R2.txt MiniSeq TruSeq (50bp) MiniSeqTruSeqL50.txt MiSeq v3 (250bp) MiSeqv3L250R1.txt MiSeqv3L250R2.txt NextSeq500 v2 (75bp) NextSeq500v2L75R1.txt NextSeq500v2L75R2.txt 2) Recalibrated quality profiles (all these are ART's built-in profiles) 36bp reads 1st: EmpR36R1 2nd: EmpR36R2 44bp reads 1st: EmpR44R1 2nd: EmpR44R2 50bp reads 1st: EmpR50R1 2nd: EmpR50R2 75 reads 1st: EmpR75R1 2nd: EmpR75R2 OUTPUT FILES *.fq - FASTQ read data files. For paired‐read simulation, *1.fq contains data of the first reads, and *2.fq for the second reads. *.aln - read alignment files. For paired‐read simulation, *1.aln has read alignments for the first reads and *2.aln for the second reads. FASTQ FILE format A FASTQ file contains both sequence bases and quality scores of sequencing reads and is in the following format: @read_id sequence_read + base_quality_scores A base quality score is coded by the ASCII code of a single character, where the quality score is equal to ASCII code of the character minus 33. Example: @refid-4028550-1 caacgccactcagcaatgatcggtttattcacgat... + ????????????7???????=&?<ref_seq_id read_id aln_start_pos ref_seq_strand ref_seq_aligned read_seq_aligned aln_start_pos is the alignment start position of reference sequence. aln_start_pos is always relative to the strand of reference sequence. That is, aln_start_pos 10 in the plus (+) strand is different from aln_start_pos 10 in the minus (‐) stand. ref_seq_aligned is the aligned region of reference sequence, which can be from plus strand or minus strand of the reference sequence. read_seq_aligned is the aligned sequence read, which always in the same orientation of the same read in the corresponding fastq file. SAM format SAM is a standard format for next-gen sequencing read alignments. The details of the format and examples are available at the links below: 1) http://samtools.sourceforge.net/SAM1.pdf 2) http://genome.sph.umich.edu/wiki/SAM BED format See the format at UCSC http://genome.ucsc.edu/FAQ/FAQformat.html#format1 NOTE: both ALN and BED format files use 0-based coordinate system while SAM format uses 1-based coordinate system. ACKNOWLEDGEMENTS I would like to thanks all ART users for their feedback and contributions, especially the users listed below. Richard Nielson, DNASTAR Bruno Nevado, CRAG in UAB Lee Katz, US CDC