| Table of Contents |
|---|
Overview
Throughout the course we have focused on small data sets, a limited number of samples, and in some cases even purposefully capped the total number of reads you have access to. This has been done for the purpose of time and letting you see the results tick by rather than simply having you come in for 30 minutes, submit a job, and wait an hour (or 6) before it starts running, and have it take another 10 hours to run. The reality is while you will sometimes work with a test sample or a small pilot project, Big Data in Biology means LOTS of data and lots of data means needing to not just identify variants in 1 sample, but to identify commonality across different systems. here we introduce you to bedtools. A program designed to make comparisons across different file types generated from different samples or using different parameters of a given pipeline.
Learning objectives
- Review another situation of the importance of knowing what version you are working with, and take note of how older versions may be better or at least have their place when you dont have access to infinite computational resources
- Become familiar with how to use bedtools intersect and subtract
- Understand when and how bedtools is useful
A note on version control
Previous year's courses dealt a lot with using specific versions of programs, and conflicts that could arise on TACC depending on if you were accessing a program same program from the BioITeam, the TACC module system, or your own installation. With now using conda and being on stampede2 (which doesn't allow access to /corral-repl on idev nodes) all version issues should be eliminated. Friday's closing presentation will detail how to select good tools, what version to select, and when to upgrade to new versions of a tool. The following info box was left in this years tutorial to detail 2 things: 1 if it ain't broke, you probably aren't going to fix it, and 2 newest isn't best if you lack the resources to use it.
...
| Info | ||
|---|---|---|
| ||
As mentioned several times throughout the course, different versions of software behave differently. Once again we have a situation where the BioITeam bedtools is available by default (version 2.20.1), and a different version is available on lonestar (version 2.26.0). A KEY addition to version 2.21 (aka the version after that which is available to you by default through the BioITeam as of this writing) was the ability of bedtools to simultaneously scan multiple files at once rather than having to sequentially scan pairs of files. Using the old version, to identify the common variants of files A, B, C and D, bedtools would have to be invoked a minimum of 3 times:
More broadly, it would have to be invoked a minimum of (number of samples - 1) times. So as you start adding more and more samples, the commands get more and more difficult to write. In some of the other tutorials you will see how we have used command line for loops to systematically deal with these types of situations though this situation would be much more complex likely requiring nested for loops and conditionals. If you had an analysis pipeline worked out to do these types of comparisons for you, upgrading may not be of any interest to you. With the new version however, such complexity isn't required making the tool more useful to more people though the computational requirements increase substantially. This is especially true when looking for convergence (ie common across all), or worse still thresholds (ie what variants are present in at least x% of the total samples). The larger your data set the more likely this can be a problem. Always make sure your programs are actually completing (NOT just erroring out), and remember there are almost always multiple ways to make the program finish running correctly (access different resources like large-mem nodes on TACC, read the documents, post on forums, reach out to former instructors, etc), or downgrade the version to an earlier version that was less computationally intensive. |
Install bedtools version 2.30.0
| Code Block | ||||
|---|---|---|---|---|
| ||||
conda create -n GVA-bedtools -c bioconda bedtools |
You can install to a different environment as you wish. Be wary of error messages as the installation has not been checked for conflicts in other environments.
Comparing the results of different mappers using bedtools
Often you want to compare the results of variant calling on different samples or using different pipelines. Bedtools is a suite of utility programs that work on a variety of file formats, one of which is conveniently VCF format. It provides many ways of slicing, dicing, and comparing the information in VCF files. In this tutorial we will use it on .vcf files generated with samtools after mapping with each of 3 different read mappers (bowtie2, bwa and bowtie) to determine what predictions are the same and which are different from the variant calling on reads mapped with different programs.
Getting some data
For now, create a new directory named GVA_bedtools on scratch and copy the respective VCF files to it from $BI/gva_course/bedtools/:
...
| Warning |
|---|
| Make sure you are on an idev node for the rest of the tutorial. |
Common among all (intersect)
One of if not the most useful links that will be provided in this course is this link to the bedtools intersect page. Not only does it give a very nice graphical interface of what intersect is doing right at the top, but it provides amazing explanations for how to build a good analysis command. If the graphical image is say not what your ideal situation is (i.e. you are interested in seeing where 100% of the samples overlap, not where 1 sample over laps with any of the other samples) there is a command for that as well. Using the link provided, and the bedtools intersect -h command, see if you can identify the regions that were found by all 3 mappers.
...
| Expand | ||
|---|---|---|
| ||
the use of -f 1.0 ensures that only variants in 100% of all the samples are listed the -header to keep the header information moving forward. |
Evaluating the results
Considering the 4 vcf files:
...
See if there are other comparisons that you are interested in.
Differential comparison (subtract)
Imagine a data set comprised of "disease state" and "normal state" individuals where the disease state is caused by any of some number of dominant (yet rare) mutations. Knowing that mutation is dominant, means that variants detected in the "normal state" are not of interest. How would one remove the variants identified in the "normal state" from the "diseased state"? How about bedtools subtract? Use the bedtools subtract -h command to see if you can put together a command.
...
| Expand | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
Rather than putting the answer here, message me on zoom with what you think the answer is.
|
Going further
Hopefully you see how these tools can be useful (if not ask). These are but 2 of the bedtools commands, take a few minutes to look through the other bedtools subcommands that are available using bedtools -h, and bedtools other_command -h
...