https://linker.bio/hash://sha256/be5605e58d2644baedcb160604080d9f02ce528064b7fbb13a5b556dd55cfeb6

Introduction

Billions of biodiversity data records are made openly available by hundreds of Natural History Collections all over the world. Also, since 1982, National Institutes of Health have published versions nucleotide sequences through GenBank. Many specimen described in Natural History Collections have associated GenBank sequence accessions available in GenBank.

During the 2023 Annual Conference of Digital Data in Biodiversity Research hosted by Arizona State University, Jenn Yost expressed a desire to make it easier to link GenBank accession records to the specimen records the helps curate at the The Hoover Herbarium ({ "http://rs.tdwg.org/dwc/terms/institutionCode": "OBI"}), Cal Poly State University, San Luis Obispo, CA (Yost 2023).

This repository is the outcome at a first prototype to help outline a process to discover OBI specimen record references in GenBank. With this, Jenn Yost and collaborators like Kate Pearson can link specimen records to the GenBank accession they are associated with.

Example

The Hoover Herbarium hosts a preserved specimen of type Angelica hendersonii Coult. & Rose that was collected in 1966-07-05 by Tracey & Viola Call at the north end of Tomales Bay and 2 mi south of Tomales in Marin County, California with catalog number: OBI09031, collector number: 2490, occurrence id: 256368e3-f8d7-4028-8010-1a4ff3eb8111, and web reference https://cch2.org/portal/collections/individual/index.php?occid=166203.

GenBank hosts a accession record https://www.ncbi.nlm.nih.gov/nuccore/MT735455 with locus Angelica hendersonii voucher Tracey & V. Call 2490 (OBI09031) internal transcribed spacer 1, 5.8S ribosomal RNA gene, and internal transcribed spacer 2, complete sequence.

Our desire is to develop a method to facilitate the discovery of this preserved specimen and their associated GenBank accession records. The annotated web page screenshots below gives some hints to what information elements may be used to help associated related records.

Methods

Instead of relying on visual inspection of individual html pages for herbarium specimen and GenBank accession records, an data-driven workflow was designed to first acquire and version GenBank and OBI records. Then, using these versioned archives, the records are analyzed and associated record candidates are proposed.

Phase 1. Acquire and Version

Acquire and Version GenBank Accession Records

GenBank publishes their sequence accession records in flat file archives online via https://ftp.ncbi.nlm.nih.gov/genbank/ . Their publications are published grouped by divisions. One of these divisions, the so-called PLN division, covers sequences of plants, fungi and algae.

We used Preston, a biodiversity dataset tracker, to track GenBank PLN sequence records and make them available for versioned archiving, and offline processing (Poelen, Pearson, and Yost 2023).

The following script was used to track the GenBank PLN sequence records:

#!/bin/bash
#
# Lists Genbank Plant sequence entries (including fungi and algae)
#
# For more info, see https://ftp.ncbi.nlm.nih.gov/genbank/README.genbank 

preston track "https://ftp.ncbi.nlm.nih.gov/genbank/gbrel.txt"\
 | preston cat\
 | grep -oE "gbpln+[0-9]+[.]seq"\
 | sed 's+^+https://ftp.ncbi.nlm.nih.gov/genbank/+g'\
 | sed 's+$+.gz+g'

At the time, this produced a list of resources starting with:

https://ftp.ncbi.nlm.nih.gov/genbank/gbpln1.seq.gz
https://ftp.ncbi.nlm.nih.gov/genbank/gbpln10.seq.gz
https://ftp.ncbi.nlm.nih.gov/genbank/gbpln100.seq.gz
https://ftp.ncbi.nlm.nih.gov/genbank/gbpln1000.seq.gz
https://ftp.ncbi.nlm.nih.gov/genbank/gbpln1001.seq.gz
https://ftp.ncbi.nlm.nih.gov/genbank/gbpln1002.seq.gz
https://ftp.ncbi.nlm.nih.gov/genbank/gbpln1003.seq.gz
https://ftp.ncbi.nlm.nih.gov/genbank/gbpln1004.seq.gz
https://ftp.ncbi.nlm.nih.gov/genbank/gbpln1005.seq.gz
https://ftp.ncbi.nlm.nih.gov/genbank/gbpln1006.seq.gz

These files ending with seq.gz were then tracked using command like:

preston track https://ftp.ncbi.nlm.nih.gov/genbank/gbpln1.seq.gz

A Preston package was built using these “track” commands to document where and when genbank resources were accessed, and what they contained. In addition, copies of the resources were made. This package can be uniquely identified by the following content id:

hash://sha256/bc7368469e50020ce8ae27b9d6a9a869e0b9a2a0a9b5480c69ce6751fa4b870e

This resulting Preston package of GenBanks PLN division record was archived offline on an external harddisk and online at ASU’s BioKiC (Biodiversity Knowledge integration Center) and made available via https://linker.bio . The total volume of the GenBank PLN records was a little over 200GB, small enough to fit on a personal computer, or external hard disk.

Acquire and Version OBI Herbarium Specimen Records

Similarly, the OBI specimen records were tracked and archived using Preston (Poelen 2023). Then, this versioned and offline enabled archive was used to query for identifiers found in candidate records.

For instance, GenBank accession record https://www.ncbi.nlm.nih.gov/nuccore/MT735455 references numbers like “2490” and “9031” (from OBI09031) extracted from their locus. These numbers are then used to select records that contain both via query:

preston ls\
 --anchor hash://sha256/be5605e58d2644baedcb160604080d9f02ce528064b7fbb13a5b556dd55cfeb6\
 --remote https://linker.bio\
 --no-cache\
 | preston dwc-stream\
 --remote https://linker.bio\
 --no-cache\
 | grep -E "[^0-9a-zA-Z-](2490)[^0-9a-zA-Z]"\
 | grep -E "[^0-9a-zA-Z-](9031)[^0-9a-zA-Z]"

where the lines with “grep” in is select only records that have the specified number (e.g., 2490, 9031) where the characters preceding and following are not alphanumeric characters. In this example, on only a single record has both numbers in it.

Phase 2. Propose OBI associated GenBank Records

Then the GenBank archive was processed to list all records that mention “OBI” in their (locus, voucher_specimen) descriptions using:

preston ls\
 --anchor hash://sha256/bc7368469e50020ce8ae27b9d6a9a869e0b9a2a0a9b5480c69ce6751fa4b870e\
 --remote https://linker.bio,https://zenodo.org/record/8117720/files/,https://biokic6.rc.asu.edu/preston/gbpln\
 --no-cache\
 | preston gb-stream\
 --remote https://linker.bio,https://zenodo.org/record/8117720/files/,https://biokic6.rc.asu.edu/preston/gbpln\
 --no-cache\
 | grep "OBI"

The first command (i.e., preston ls ... https://linker.bio) lists the content of the package with id hash://sha256/bc7368469e50020ce8ae27b9d6a9a869e0b9a2a0a9b5480c69ce6751fa4b870e, and downloads the necessary data via https://linker.bio if needed.

The second command (i.e., preston gb-stream) analyzes the package content as a stream, and generates metadata objects for each genbank accession encountered.

The third command (i.e., grep "OBI") includes only those datadata records that contain “OBI”.

Results

Capture GenBank Candidate Records

On processing millions of GenBank accession records, 256 candidate genbank accession records with mention of OBI were shared with Katelin Pearson for review. By selecting the PLN division (plants), and selecting the OBI institutions code, we reduced the search space by a couple of order of magnitudes. With only a few hundred records, Kateline Pearson, an OBI curator, was able to make the candidate GenBank accession records that likely referenced OBI specimen records (see genbank-associations-mentioning-OBI.csv or associated online sheet for the table with manual review notes).

Following, Jorrit Poelen used the OBI preston archive and retrieved preserved specimen records that contained numbers and/or other identifying information (e.g., scientific name occurring in the genbank accession record) to select a candidate specimen record for each candidate accession record. In about 1.5 hours, he compiled this list of specimen record / accession records associations in the following format.

Curatorial Candidate Record Review

With this information, Katelin Pearson, a OBI data curator, took about 15 minutes to annotate the specimen records in the CCH2 Symbiota database with their related GenBank Accession number. Most of this time was spent to convert the information provided by Jorrit Poelen into a more convenient format. The full table can be found in Appendix A and the first two lines of the OBI genetics table can be found below. Here, the occid is the record number unique to the CCH2 Symbiota database, url is the reference a GenBank accession, resourcename is the type of resource that Symbiota understands, and locus the optional information supported by Symbiota to include in an associated sequence record.

Adding GenBank Links to Symbiota Records

After Katelin Pearson upload the genbank link table into the CCH2 Symbiota database, she exported the updated records to the published DwC-A. Following, Jorrit Poelen tracked the updated version of the DwC-A and selected the records with associated GenBank sequence records. Following, he created a table (see Appendix B.) including the reference to the original record, a web url to a html record page, and the associated genbank record annotations. The first three lines of Appendix B. can be found below.

Comparing Example Record Before and After Record Linking

In our methods, we keep track of the versions of the datasets we work with. The OBI specimen records were versioned prior and after annotating OBI specimen records with their associated GenBank accessions. This means that the changes in the records, as published via the OBI DwC-A can be measured.

To demonstrate the changes to a specific record related to our example specimen record OBI09031, please consider the record prior to annotating the association:

preston ls\
 --anchor hash://sha256/be5605e58d2644baedcb160604080d9f02ce528064b7fbb13a5b556dd55cfeb6\
 --remote https://linker.bio\
 --no-cache\
 | preston dwc-stream\
 --remote https://linker.bio\
 --no-cache\
 | grep -E "[^0-9a-zA-Z-](2490)[^0-9a-zA-Z]"\
 | grep -E "[^0-9a-zA-Z-](9031)[^0-9a-zA-Z]"\
 | tail -n1\
 | jq --raw-output '.["http://www.w3.org/ns/prov#wasDerivedFrom"]'

which points us to the versioned records with identifier:

line:zip:hash://sha256/b60f9dd7868d6296ddea107219d41e5a92d55f1a5e0e5ee894c6e9977cb872cd!/occurrences.csv!/L2361

The content associated with this content identifier can be retrieved via preston cat 'line:zip:hash://sha256/b60f9dd7868d6296ddea107219d41e5a92d55f1a5e0e5ee894c6e9977cb872cd!/occurrences.csv!/L1,L2361' or accessed via OBI09031@b60f9.csv.

A textual representation of the record is shown below.

id                             166203
institutionCode                OBI
collectionCode                 
ownerInstitutionCode           
collectionID                   3818d95b-b6a4-11e8-b408-001a64db2964
basisOfRecord                  PreservedSpecimen
occurrenceID                   256368e3-f8d7-4028-8010-1a4ff3eb8111
catalogNumber                  
otherCatalogNumbers            9031
higherClassification           Organism|Plantae|Viridiplantae|Streptophyta|Embryophyta|Tracheophyta|Spermatophytina|Magnoliopsida|Asteranae|Apiales|Apiaceae|Angelica
kingdom                        Plantae
phylum                         Tracheophyta
class                          Magnoliopsida
order                          Apiales
family                         Apiaceae
scientificName                 Angelica hendersonii
taxonID                        210544
scientificNameAuthorship       Coult. & Rose
genus                          Angelica
subgenus                       
specificEpithet                hendersonii
verbatimTaxonRank              
infraspecificEpithet           
taxonRank                      Species
identifiedBy                   
dateIdentified                 
identificationReferences       
identificationRemarks          
taxonRemarks                   
identificationQualifier        
typeStatus                     
recordedBy                     Tracey Call; Viola Call
recordNumber                   2490
eventDate                      1966-07-05
year                           1966
month                          7
day                            5
startDayOfYear                 186
endDayOfYear                   
verbatimEventDate              5-Jul-66
occurrenceRemarks              
habitat                        Low bluffs
fieldNumber                    
eventID                        
informationWithheld            
dataGeneralizations            
dynamicProperties              
associatedOccurrences          herbariumSpecimenDuplicate: https://cch2.org/portal/collections/individual/index.php?guid=9afb4a2f-ac2a-4581-b339-9c394ed1163e | herbariumSpecimenDuplicate: https://cch2.org/portal/collections/individual/index.php?guid=1775039a-00d1-4cd3-b40a-3535c7863d93
associatedSequences            
associatedTaxa                 
reproductiveCondition          
establishmentMeans             
lifeStage                      
sex                            
individualCount                
preparations                   
locationID                     
continent                      
waterBody                      
islandGroup                    
island                         
country                        United States
stateProvince                  California
county                         Marin
municipality                   
locality                       North end of Tomales Bay and 2 mi south of Tomales
locationRemarks                
decimalLatitude                
decimalLongitude               
geodeticDatum                  
coordinateUncertaintyInMeters  
verbatimCoordinates            
georeferencedBy                
georeferenceProtocol           
georeferenceSources            
georeferenceVerificationStatus 
georeferenceRemarks            
minimumElevationInMeters       15
maximumElevationInMeters       
minimumDepthInMeters           
maximumDepthInMeters           
verbatimDepth                  
verbatimElevation              50ft.
disposition                    
language                       
recordEnteredBy                
modified                       2011-08-18 00:00:00
rights                         http://creativecommons.org/licenses/by-nc/4.0/
rightsHolder                   
accessRights                   
recordID                       9a370197-6899-4072-8b17-4f2f043fbd54
references                     https://cch2.org/portal/collections/individual/index.php?occid=166203

Similarly, the record seen after the annotation can be retrieved using:

preston ls\
 --anchor hash://sha256/be5605e58d2644baedcb160604080d9f02ce528064b7fbb13a5b556dd55cfeb6\
 --remote https://linker.bio\
 --no-cache\
 | preston dwc-stream\
 --remote https://linker.bio\
 --no-cache\
 | grep -E "[^0-9a-zA-Z-](2490)[^0-9a-zA-Z]"\
 | grep -E "[^0-9a-zA-Z-](9031)[^0-9a-zA-Z]"\
 | head -n1\
 | jq --raw-output '.["http://www.w3.org/ns/prov#wasDerivedFrom"]'

yielding:

line:zip:hash://sha256/cd9de973510975dac3394952bba9c486a482762b3beab05ecb678037b99ab85b!/occurrences.csv!/L2361

Now, we can use a text comparison between the two versioned records, using diff, a widely available linux tool.

diff <(preston cat 'line:zip:hash://sha256/b60f9dd7868d6296ddea107219d41e5a92d55f1a5e0e5ee894c6e9977cb872cd!/occurrences.csv!/L1,L2361' | mlr --icsv --ojsonl cat ) <(preston cat 'line:zip:hash://sha256/cd9de973510975dac3394952bba9c486a482762b3beab05ecb678037b99ab85b!/occurrences.csv!/L1,L2361' | mlr --icsv --ojsonl cat) 

which results in

50c50
<   "associatedSequences": "",
---
>   "associatedSequences": "GenBank Record, Angelica hendersonii voucher Tracey & V. Call 2490 (OBI09031) internal transcribed spacer 1, 5.8S ribosomal RNA gene, and internal transcribed spacer 2, complete sequence., https://www.ncbi.nlm.nih.gov/nuccore/MT735455|GenBank Record, Angelica hendersonii Tracey & V. Call 2490 (OBI09031) ndhF-rpl32 intergenic spacer, partial sequence., https://www.ncbi.nlm.nih.gov/nuccore/MT765790|GenBank Record, Angelica hendersonii Tracey & V. Call 2490 (OBI09031) tRNA-Asp (trnD-GUC), tRNA-Tyr (trnY-GUA), tRNA-Glu (trnE-UUC), and tRNA-Thr (trnT-GGU) genes, complete sequence., https://www.ncbi.nlm.nih.gov/nuccore/MT765975|GenBank Record, Angelica hendersonii Tracey & V. Call 2490 (OBI09031) rpl32-trnL intergenic spacer and tRNA-Leu (trnL) gene, partial sequence., https://www.ncbi.nlm.nih.gov/nuccore/MT766140",

: output of a commonly used programming tool diff as applied to our OBI09031 example.

Additionally, you can find the before/after example records in json (i.e., OBI09031-before.json/ OBI09031-after.json) or csv (i.e., OBI09031-before.csv / OBI09031-after.csv) formats.

Finally, because we have our versioned records available in text formats, the options for re-use, archiving, or other subsequent processing are plentiful, and is consistent with one of the Unix principles (McIlroy, Pinson, and Tague 1978).

Expect the output of every program to become the input to another, as yet unknown, program.

Discussion

We took a systematic approach to independently track natural history collection records and sequence records. Then, we used regular expressions (or queries) to select candidate GenBank accession records. Following, after manual review of candidate records, we extracted identifiers and names to link locate their associated specimen records in the Hoover Herbarium collection as tracked. While the method is not fully automated, our method reduced the number of candidate accession records from millions to hundreds. This many order of magnitude reduction of candidates made manual review was feasible. We expect that periodic revisiting of the available records in GenBank will yield additional associated genbank records. Also, we hope that this example show that links between existing GenBank accessions and their specimen records can be found without major technological investment. And, we hope that this example will help inspire to develop best practices to place identifying information in GenBank records such that collection managers can somehwat easily locate sequences associated to the specimen they keep.

Appendix A

GenBank link table created by Katelin Pearson to link OBI specimen records to their associated GenBank sequences.

See also OBI_genetics.csv.

Appendix B

References to specimen records with associated sequences after application of links of Appendix A.

generated using:

preston cat\
 --remote https://linker.bio\
 hash://sha256/be5605e58d2644baedcb160604080d9f02ce528064b7fbb13a5b556dd55cfeb6\
 | preston dwc-stream\
 | jq -c 'select(.["http://rs.tdwg.org/dwc/terms/associatedSequences"] != null)'\
 | jq '{ derivedFrom: .["http://www.w3.org/ns/prov#wasDerivedFrom"], reference: .["http://purl.org/dc/terms/references"], associatedSequences: .["http://rs.tdwg.org/dwc/terms/associatedSequences"] }'\
 | sed 's+line:zip+https://linker.bio/line:zip+g'\
 | sed 's+occurrences.csv!/+occurrences.csv!/L1,+g'\ 
 | mlr --ijson --ocsv cat

See also specimen-record-with-associated-sequences.csv.

References