BSI Hub

Topics:

BSI Hub

BSI Hub is a platform for the translation and loading of data from multiple systems, each of which uses different terminology, into a single database for collective access. For example, if a community of repositories want to gather data into a centralized location to promote data sharing amongst them, they may encounter difficulties with data sets being from a wide array of legacy systems. Data from those systems would most probably be stored in vastly different formats. Instead of forcing any participating groups to migrate to a single platform, BSI Hub allows for continued use of legacy systems while providing the ability to automatically translate data into a single, uniform format in a BSI database so that data can be viewed collectively.

Example: A cohort of 3 biobanks decide to use BSI as the central repository for sharing their collected data with researchers. Each biobank works with IMS to build a translation which will transform the file’s data into a format readable by BSI.

  • Biobank #1 exports a master file containing all records from their legacy system once a week. A technician logs into BSI Hub and submits the single file.
  • Biobank #2 programmatically exports files from their legacy system to a network drive nightly. They use BSI Hub APIs to automatically submit new files stored in this network location to BSI Hub for processing each morning. They do not login to BSI Hub.
  • Biobank #3 exports a file containing all new or updated subject records and a second file containing all new or updated specimens stored with the legacy system. A technician logs into BSI Hub and submits both files.

Multiple translations can be created to facilitate translation of data from different legacy formats. These translations are coded ahead of time by IMS staff using a specific set of requirements agreed upon by all involved parties.

APIs are available to programmatically process files using a selected BSI Hub translation instead of submitting files via the website user interface.

Users can submit files to BSI Hub containing only specimen data or containing specimen data in conjunction with subject and/or location data. Subject and location data cannot be submitted independent of specimen data to BSI Hub. BSI Hub accepts .csv and .xlsx file types for submission.Once translations have been completed, the data is pushed into BSI. In the BSI database, the loaded data is committed via Data Entry batches.

During scheduled system maintenance windows, file processing is paused. Once a maintenance window is over, BSI Hub resumes processing any paused submissions.

To process a file on BSI Hub:

  1. Login to BSI Hub.
    1. If you have access to more than one repository, select the one to which you are submitting data.
  2. Select a value for the Type field. This will be the translation used to systematize the data from the legacy system into vocabulary matching that of BSI.
  3. Using the file explorer, upload the file to be submitted.
  4. If desired, enter Notes about the file upload.
  5. Select the Submit

BSI_Hub_interface.png

Submission Progress Tab

When viewing a submission’s Details page, the Submission Progress tab will open first. This tab gives an overview of the current status of the whole submission and the individual stages each file goes through during processing.

For each submitted file, BSI Hub will attempt the following processing stages:

  1. Transforming
  2. Analyzing File
  3. Processing Subjects
  4. Saving New Subjects
  5. Saving Modified Subjects
  6. Processing Specimens
  7. Adding New Specimens to Batches
  8. Adding Modified Specimens to Batches
  9. Committing New Specimen Batches
  10. Committing Modified Specimen Batches
  11. Finalizing Submission

At the top of the tab, a header indicated which stage, if any, BSI Hub is currently processing for this submission. If processing is complete, stopped due to errors encountered, or paused, then the header will indicate that instead.

The progress bar, in combination with the status key above it, give a visual indication of what stage BSI Hub is at in processing. Below that, the Submission Details table gives date/time stamps for the beginning and end of each stage. A count of any errors that are encountered (processing errors or BSI errors) will be included with the stage, as well as an indication of where to look for more details. Additionally, for subject saves and specimen commits, a count of successfully processed items out of the total number BSI Hub attempted to process during that stage is included.BSI_Hub_submission_progress.jpg

BSI Results Tab

The BSI Results tab is displayed for submissions which have a status of “Completed”. It includes counts for the following items which were processed during the submission:

  • New Specimens
  • Modified Specimens
  • New Subjects
  • Modified Subjects

The tab also includes 2 tabs, which list the Add and Modify batch IDs in which specimens were updated, if applicable.

BSI Errors Tab

The BSI Errors tab is displayed for submissions which have a status of “Completed”. It includes counts for the following types of errors encountered during the submission:

  • Add Vial
  • Mod Vial
  • Add Subjects
  • Mod Subjects

Clicking on any of the aforementioned counts will display the corresponding details table in the tab. The details table lists information about each specific error. The following fields are shown for all errors: Study ID, Subject ID, Field, and Error. Additionally for specimen errors, the Current Label field is shown.

An Export Errors button is available on this tab. Selecting the option will download an .xlsx file containing all errors for the submission.

Processing Errors Tab

The Processing Errors tab is displayed when non-BSI errors were encountered during file processing. Examples of such errors include: incorrect file format, missing required fields, and incorrect date formats.