A biodiversity dataset graph: https://jhpoelen.nl/bats. 2021. hash://sha256/5150f699411c4433b0a6c111f8e6ec7fbae2c336ab237f3638bbdc9d0b2dda0d
Biodiversity knowledge graph created using Preston v0.3.1 and Jekyll v3.9.0 on 2021-09-22T22:30:13.410Z
Are you looking for a way to have fast, local access to GBIF/iDigBio indexed records and media?
Would you like to have an exact copy of the images in your research dataset?
Do you want to include latest research data while keeping your original data around?
This automatically generated website contains a versioned archive of a custom selection of occurrence/specimen records and associated media. The selection is made using currently available biodiversity search indexes like the iDigBio Search API or the GBIF Occurrence Search API. The indexed data is archived using Preston, a biodiversity data tracker that can version entire biodiversity dataset networks. Finally, the website is generated from the archived content using Jekyll, the static site generator that powers GitHub pages.
This biodiversity data archive website was created with the following steps:
# first, create a new blank jekyll site (needs Jekyll v4+, tested on Jekyll v4.0.1)
jekyll new [site_dir] --blank
cd [site_dir]
# use GBIF/iDigBio API to track specimen, occurrences and their related images.
preston track "https://search.idigbio.org/v2/search/records/?rq=%7B%22order%22%3A%22Chiroptera%22%2c%22hasImage%22%3A%22true%22%7D&limit=100&offset=0"
# generate Jekyll site for archived content
preston copyTo --type jekyll .
# launch website and visit http://localhost:4000 in your browser
jekyll s
You can clone an exact copy of the entire biodiversity data archive using:
# first, create a new blank jekyll site (needs Jekyll v4+, tested on Jekyll v4.0.1)
jekyll new [site_dir] --blank
cd [site_dir]
# clone this existing biodiversity dataset graph
preston clone "https://jhpoelen.nl/bats/data"
# generate Jekyll site for archived content
preston copyTo --type jekyll .
# launch website and visit http://localhost:4000 in your browser
jekyll s
Also, you can query the indexed data available through this site via the api at https://jhpoelen.nl/bats/data.json (one json object per line, https://jsonlines.org/) .
With this, you can programmatically access the data and select the records you are interested in.
For instance, using curl and jq, you can show the first record by executing:
$ curl "https://jhpoelen.nl/bats/data.json" | jq -c 'select(.type == "records")' | head -n1
[{"uuid":"0948c7c9-23fb-428b-9fd5-99e87f87f5d2","type":"records","etag":"e5063e3512c73204ffa467274157586411d669a4","data":{"dwc:identificationRemarks":"Former nature_of_id: expert.","dwc:specificEpithet":"pallidus","dwc:county":"Eddy County","dwc:recordedBy":"Collector(s): Arthur H. Harris","dwc:georeferencedDate":"1979-06-08 00:00:00","dwc:order":"Chiroptera","dcterms:references":"http://arctos.database.museum/guid/UTEP:ES:75-92","dwc:individualCount":"1","dwc:occurrenceID":"http://arctos.database.museum/guid/UTEP:ES:75-92?seid=3401260","dwc:dateIdentified":"1979-06-08","dcterms:language":"en","dwc:earliestEpochOrLowestSeries":"Pleistocene","id":"http://arctos.database.museum/guid/UTEP:ES:75-92?seid=3401260","dwc:minimumElevationInMeters":"1067","dwc:stateProvince":"New Mexico","dwc:locationRemarks":"Pecos River drainage, 24 NW1/4 SW1/4 R25E T23S, Specimen Age Estimate - minimum date: 10730","dwc:eventDate":"1979-06-08","dwc:collectionID":"118","dwc:country":"United States","dwc:earliestPeriodOrLowestSystem":"Quaternary","dwc:collectionCode":"Earth Science","dwc:higherClassification":"Animalia, Chordata, Mammalia, Chiroptera, Microchiroptera, Vespertilionidae, Antrozous,","dwc:kingdom":"Animalia","dwc:decimalLatitude":"32.2875530000","dwc:georeferenceVerificationStatus":"unverified","dwc:maximumElevationInMeters":"1067","dwc:basisOfRecord":"PreservedSpecimen","dcterms:accessRights":"http://vertnet.org/resources/norms.html","dwc:genus":"Antrozous","dwc:continent":"North America","dwc:family":"Vespertilionidae","dwc:identificationQualifier":"A","dwc:identifiedBy":"Arthur H. Harris","dwc:identificationVerificationStatus":"features","dwc:georeferenceSources":"collector","dwc:previousIdentifications":"[{\"scientific_name\":\"Antrozous pallidus\",\"nature_of_id\":\"features\",\"accepted_id_fg\":1,\"made_date\":\"1979-06-08\",\"short_citation\":null,\"publication_id\":null,\"identification_remarks\":\"Former nature_of_id: expert.\",\"idby\":\"Arthur H. Harris\",\"concept_label\":null}]","dwc:organismID":"http://arctos.database.museum/guid/UTEP:ES:75-92","dwc:phylum":"Chordata","dwc:earliestEonOrLowestEonothem":"Phanerozoic","dwc:locationAccordingTo":"Arthur H. Harris","dwc:locality":"Dark Canyon Cave","dwc:institutionID":"UTEP","dwc:associatedMedia":"http://arctos.database.museum/MediaSearch.cfm?collection_object_id=26533983","dwc:institutionCode":"UTEP","dwc:taxonRank":"species","dwc:class":"Mammalia","dwc:catalogNumber":"UTEP:ES:75-92","dwc:nomenclaturalCode":"ICZN","dcterms:type":"PhysicalObject","dwc:higherGeography":"North America, United States, New Mexico, Eddy County","dwc:month":"06","dwc:decimalLongitude":"-104.3542540000","dwc:coordinateUncertaintyInMeters":"1000","dwc:endDayOfYear":"159","dwc:otherCatalogNumbers":"collector number=5228","dwc:verbatimLocality":"USA, New Mexico, Eddy, Dark Canyon Cave, Pecos River drainage, 32.287553 Lat, -104.354254 Lon, 24 NW1/4 SW1/4 R25E T23S, Elev. 1067 M, Strat: Pot hunters' spoil","dwc:georeferencedBy":"Arthur H. Harris","dwc:verbatimEventDate":"1979-06-08","dwc:eventTime":"1979-06-08","dwc:recordNumber":"5228","dwc:preparations":"skull","dwc:georeferenceProtocol":"not recorded","dcterms:modified":"2021-01-15 06:00:00.006453","dwc:scientificName":"Antrozous pallidus","dwc:day":"08","dwc:year":"1979"},"indexTerms":{"individualcount":1,"geopoint":{"lat":32.287553,"lon":-104.354254},"family":"vespertilionidae","recordset":"b7a79601-c07b-46d5-bd09-d4472b0d9431","dqs":0.42028985507246375,"stateprovince":"new mexico","earliestepochorlowestseries":"pleistocene","county":"eddy county","earliestperiodorlowestsystem":"quaternary","phylum":"chordata","catalognumber":"utep:es:75-92","startdayofyear":159,"taxonrank":"species","specificepithet":"pallidus","continent":"north america","datemodified":"2021-02-09T23:20:16.430782+00:00","uuid":"0948c7c9-23fb-428b-9fd5-99e87f87f5d2","countrycode":"usa","basisofrecord":"preservedspecimen","collector":"collector(s): arthur h. harris","institutioncode":"utep","mediarecords":["1020c918-6d65-4474-837f-61dc77d2ac10"],"verbatimlocality":"usa, new mexico, eddy, dark canyon cave, pecos river drainage, 32.287553 lat, -104.354254 lon, 24 nw1/4 sw1/4 r25e t23s, elev. 1067 m, strat: pot hunters' spoil","datecollected":"1979-06-08T00:00:00+00:00","etag":"e5063e3512c73204ffa467274157586411d669a4","recordnumber":"5228","minelevation":1067,"hasImage":true,"kingdom":"animalia","maxelevation":1067,"highertaxon":"animalia, chordata, mammalia, chiroptera, microchiroptera, vespertilionidae, antrozous,","collectionid":"118","taxonid":"2432339","scientificname":"antrozous pallidus","indexData":{"dwc:county":"Eddy County","dwc:georeferenceSources":"collector","dwc:earliestEonOrLowestEonothem":"Phanerozoic","flag_gbif_reference_added":true,"flag_gbif_genericname_added":true,"flag_dwc_datasetid_added":true,"idigbio:parent":"b7a79601-c07b-46d5-bd09-d4472b0d9431","dwc:country":"United States","flag_gbif_vernacularname_added":true,"dwc:maximumElevationInMeters":"1067","dwc:family":"Vespertilionidae","dwc:nomenclaturalCode":"ICZN","dcterms:type":"PhysicalObject","dwc:earliestEpochOrLowestSeries":"Pleistocene","dwc:coordinateUncertaintyInMeters":"1000","gbif:reference":[{"coreid":"2432339","dcterms:source":"catalogue of life","dcterms:bibliographiccitation":"(1996) nodc taxonomic code"},{"coreid":"2432339","dcterms:source":"catalogue of life","dcterms:bibliographiccitation":"(2001) database, comisión nacional para el conocimiento y uso de la biodiversidad"},{"coreid":"2432339","dcterms:source":"catalogue of life","dcterms:bibliographiccitation":"banks, r. c., r. w. mcdiarmid, and a. l. gardner (1987) checklist of vertebrates of the united states, the u.s. territories, and canada: resource publication, no. 166"},{"coreid":"2432339","dcterms:source":"catalogue of life","dcterms:bibliographiccitation":"simmons, nancy b. / wilson, don e., and deeann m. reeder, eds. (2005) order chiroptera: mammal species of the world: a taxonomic and geographic reference, 3rd ed., vols. 1 & 2"},{"coreid":"2432339","dcterms:source":"catalogue of life","dcterms:bibliographiccitation":"simmons, nancy b. / wilson, don e., and deeann m. reeder, eds. (2005) order chiroptera: mammal species of the world: a taxonomic and geographic reference, 3rd ed., vols. 1 & 2"},{"coreid":"2432339","dcterms:source":"catalogue of life","dcterms:bibliographiccitation":"wilson, don e., and deeann m. reeder, eds. (1992) mammal species of the world: a taxonomic and geographic reference, 2nd ed., 3rd printing"},{"coreid":"2432339","dcterms:source":"catalogue of life","dcterms:bibliographiccitation":"wilson, don e., and f. russell cole (2000) common names of mammals of the world"},{"coreid":"2432339","dcterms:source":"catalogue of life","dcterms:bibliographiccitation":"wilson, don e., and f. russell cole (2000) common names of mammals of the world"}],"gbif:vernacularname":[{"coreid":"2432339","dcterms:language":"es","dcterms:source":"integrated taxonomic information system (itis)","dwc:vernacularname":"murciélago-desértico norteño"},{"coreid":"2432339","dcterms:language":"es","dcterms:source":"catalogue of life","dwc:vernacularname":"murciélago-desértico norteño"},{"coreid":"2432339","dcterms:language":"en","dcterms:source":"catalogue of life","dwc:vernacularname":"pallid bat"},{"coreid":"2432339","dcterms:language":"en","dcterms:source":"integrated taxonomic information system (itis)","dwc:vernacularname":"pallid bat"},{"coreid":"2432339","dcterms:language":"en","dcterms:source":"mammal species of the world","dwc:vernacularname":"pallid bat"},{"coreid":"2432339","dcterms:language":"en","dcterms:source":"the paleobiology database","dwc:vernacularname":"pallid bat"},{"coreid":"2432339","dcterms:source":"ncbi taxonomy","dwc:vernacularname":"pallid bat"},{"coreid":"2432339","dcterms:source":"phthiraptera.info","dwc:vernacularname":"pallid bat"},{"coreid":"2432339","dcterms:language":"de","dcterms:source":"german wikipedia - species pages","dwc:vernacularname":"wüstenfledermaus"}],"dwc:identificationRemarks":"Former nature_of_id: expert.","dwc:recordedBy":"Collector(s): Arthur H. Harris","dwc:order":"Chiroptera","dcterms:references":"http://arctos.database.museum/guid/UTEP:ES:75-92","dwc:individualCount":"1","dwc:dateIdentified":"1979-06-08","flag_dwc_taxonomicstatus_added":true,"dwc:basisOfRecord":"PreservedSpecimen","dwc:taxonomicstatus":"accepted","dwc:identificationQualifier":"A","dwc:previousIdentifications":"[{\"scientific_name\":\"Antrozous pallidus\",\"nature_of_id\":\"features\",\"accepted_id_fg\":1,\"made_date\":\"1979-06-08\",\"short_citation\":null,\"publication_id\":null,\"identification_remarks\":\"Former nature_of_id: expert.\",\"idby\":\"Arthur H. Harris\",\"concept_label\":null}]","flag_dwc_multimedia_added":true,"dwc:associatedMedia":"http://arctos.database.museum/MediaSearch.cfm?collection_object_id=26533983","dwc:institutionCode":"UTEP","flag_gbif_taxon_corrected":true,"dwc:parentnameusageid":"2432337","dwc:class":"Mammalia","dwc:taxonid":"2432339","flag_gbif_canonicalname_added":true,"dwc:verbatimLocality":"USA, New Mexico, Eddy, Dark Canyon Cave, Pecos River drainage, 32.287553 Lat, -104.354254 Lon, 24 NW1/4 SW1/4 R25E T23S, Elev. 1067 M, Strat: Pot hunters' spoil","dwc:day":"08","dwc:minimumElevationInMeters":"1067","dwc:scientificName":"Antrozous pallidus","idigbio:dateModified":"2021-02-09T23:20:16.430782","idigbio:uuid":"0948c7c9-23fb-428b-9fd5-99e87f87f5d2","dwc:georeferencedDate":"1979-06-08 00:00:00","gbif:canonicalname":"antrozous pallidus","dwc:occurrenceID":"http://arctos.database.museum/guid/UTEP:ES:75-92?seid=3401260","flag_dwc_originalnameusageid_added":true,"dwc:earliestPeriodOrLowestSystem":"Quaternary","dwc:multimedia":[{"dcterms:license":"public domain mark 1.0","dcterms:references":"http://commons.wikimedia.org/wiki/file:antrozous_pallidus1.jpg","coreid":"2432339","dcterms:identifier":"http://upload.wikimedia.org/wikipedia/commons/3/30/antrozous_pallidus1.jpg","dcterms:source":"english wikipedia - species pages","dcterms:description":"pallid bat (antrozous pallidus)","dcterms:creator":"usgs","dcterms:publisher":"wikimedia commons"},{"dcterms:license":"pd us blm","dcterms:title":"wüstenfledermaus (antrozous pallidus)","dcterms:references":"http://commons.wikimedia.org/wiki/file:antrozous_pallidus.jpg","coreid":"2432339","dcterms:identifier":"http://upload.wikimedia.org/wikipedia/commons/5/5a/antrozous_pallidus.jpg","dcterms:source":"german wikipedia - species pages","dcterms:description":"antrozous pallidus","dcterms:creator":"m. siders (blm)","dcterms:publisher":"wikimedia commons"}],"idigbio:etag":"e5063e3512c73204ffa467274157586411d669a4","flag_dwc_scientificnameauthorship_added":true,"dwc:decimalLatitude":"32.2875530000","dwc:datasetid":"7ddf754f-d193-4cc9-b351-99906754a03b","dwc:identificationVerificationStatus":"features","flag_idigbio_isocountrycode_added":true,"dwc:scientificnameauthorship":"(leconte, 1856)","dwc:locality":"Dark Canyon Cave","dwc:taxonRank":"species","dwc:higherClassification":"Animalia, Chordata, Mammalia, Chiroptera, Microchiroptera, Vespertilionidae, Antrozous,","dwc:catalogNumber":"UTEP:ES:75-92","dwc:higherGeography":"North America, United States, New Mexico, Eddy County","dcterms:accessRights":"http://vertnet.org/resources/norms.html","dwc:decimalLongitude":"-104.3542540000","dwc:month":"06","dwc:georeferencedBy":"Arthur H. Harris","dwc:originalnameusageid":"8720672","dwc:eventTime":"1979-06-08","dwc:preparations":"skull","dwc:specificEpithet":"pallidus","dwc:kingdom":"Animalia","idigbio:recordIds":["b7a79601-c07b-46d5-bd09-d4472b0d9431\\http://arctos.database.museum/guid/utep:es:75-92?seid=3401260"],"dcterms:language":"en","id":"http://arctos.database.museum/guid/UTEP:ES:75-92?seid=3401260","dwc:stateProvince":"New Mexico","dwc:eventDate":"1979-06-08","dwc:collectionID":"118","dwc:collectionCode":"Earth Science","dwc:georeferenceVerificationStatus":"unverified","dwc:genus":"Antrozous","dwc:continent":"North America","dwc:identifiedBy":"Arthur H. Harris","flag_dwc_taxonid_added":true,"idigbio:siblings":{"mediarecord":["1020c918-6d65-4474-837f-61dc77d2ac10"]},"dwc:organismID":"http://arctos.database.museum/guid/UTEP:ES:75-92","dwc:phylum":"Chordata","gbif:genericname":"antrozous","dwc:locationAccordingTo":"Arthur H. Harris","dwc:institutionID":"UTEP","flag_dwc_parentnameusageid_added":true,"dwc:locationRemarks":"Pecos River drainage, 24 NW1/4 SW1/4 R25E T23S, Specimen Age Estimate - minimum date: 10730","dwc:endDayOfYear":"159","dwc:otherCatalogNumbers":"collector number=5228","dwc:verbatimEventDate":"1979-06-08","dwc:recordNumber":"5228","idigbio:isocountrycode":"usa","dwc:georeferenceProtocol":"not recorded","dcterms:modified":"2021-01-15 06:00:00.006453","dwc:year":"1979"},"hasMedia":true,"coordinateuncertainty":1000,"class":"mammalia","occurrenceid":"http://arctos.database.museum/guid/utep:es:75-92?seid=3401260","earliesteonorlowesteonothem":"phanerozoic","institutionid":"utep","country":"united states","locality":"dark canyon cave","collectioncode":"earth science","canonicalname":"antrozous pallidus","eventdate":"1979-06-08","flags":["geopoint_datum_missing","gbif_reference_added","gbif_genericname_added","dwc_datasetid_added","gbif_vernacularname_added","dwc_taxonomicstatus_added","dwc_multimedia_added","gbif_taxon_corrected","gbif_canonicalname_added","dwc_originalnameusageid_added","dwc_scientificnameauthorship_added","idigbio_isocountrycode_added","dwc_taxonid_added","dwc_parentnameusageid_added"],"verbatimeventdate":"1979-06-08","taxonomicstatus":"accepted","recordids":["b7a79601-c07b-46d5-bd09-d4472b0d9431\\http://arctos.database.museum/guid/utep:es:75-92?seid=3401260"],"genus":"antrozous","order":"chiroptera","datasetid":"7ddf754f-d193-4cc9-b351-99906754a03b"}}]
Or, look for an occurrence with images indexed by GBIF:
$ curl "https://jhpoelen.nl/bats/data.json" | jq -c 'select(.media[].type == "StillImage")' | head -n1
[]
Or, select archived iDigBio indexed records with scientific name matching Liphanthus sabulosus:
curl -s "https://jhpoelen.nl/bats/data.json" | jq -c 'select(.data["dwc:scientificName"] == "Liphanthus sabulosus")'
This biodiversity archive contains local copies of remote content.You can access a list of all archived location via the content registry at https://jhpoelen.nl/bats/registry.json ((one json object per line, https://jsonlines.org/)). Here’s a way to get the first one:
$ curl -s "https://jhpoelen.nl/bats/registry.json" | head -n1
{"url":"https://search.idigbio.org/v2/search/records/?rq=%7B%22order%22%3A%22Chiroptera%22%2c%22hasImage%22%3A%22true%22%7D&limit=100&offset=0","verb":"http://purl.org/pav/hasVersion","hash":"hash://sha256/2ebe4283cc704e093fbed0fc49edb64d9873e9eca04c1d31a61961e843f841fd","graphname":"urn:uuid:aa2ce2a1-e330-45dc-adbf-461ca742ecd3"}
The backbone of this biodiversity data archive is their provenance log, or knowledge graph. This knowledge graph is stored in in rdf/ nquad format and can loaded into triple store and queried using SPARQL.
Because the log is stored in a text file you can easily read it. For instance, the first 10 lines of the provenance graph (or knowledge graph) can be seen when running:
curl -s "https://jhpoelen.nl/bats/data/51/50/5150f699411c4433b0a6c111f8e6ec7fbae2c336ab237f3638bbdc9d0b2dda0d" | head
For more information about how to fully take advantage of this biodiversity data graph, please review preston documentation for use cases, concepts and architecture.
Many natural history collections are actively digitizing their collections. These collections are actively indexed by iDigBio as the new data records and media become available. This website can be updated to incorporate newly added or updated data by:
cd [site_dir]
# archive records and related images with criteria specified in some biodiversity search API
preston update "https://search.idigbio.org/v2/search/records/?rq=%7B%22order%22%3A%22Chiroptera%22%2c%22hasImage%22%3A%22true%22%7D&limit=100&offset=0"
# update Jekyll site with archived content
preston copyTo --type jekyll .
This archived dataset includes 100 iDigBio indexed specimen records, 460 iDigBio indexed media records, 0 GBIF indexed occurrence records, and 0 GBIF indexed images.
The first 10-20 records and their associated media included in this data archive are:
