Frequently Asked Questions
DIGGS Background
- What is DIGGS?
DIGGS is a data transfer protocol. It allows each piece of data to be connected to key metadata such as test reference, geographic location, geologic references, etc. By using a common and robust transfer protocol, the end use (interpretation, storage, presentation) is separated from data transfer. - Why “data transfer” and not “database”?
Everyone has a database that meets their needs, or maybe multiple databases for different needs and departments. DIGGS is a way to allow data to be easily transferred from one storage “box” to another without human interaction. - Why should I adopt DIGGS?
Many agencies have struggled with getting data delivered in a format that will work with their database and software. DIGGS is autonomous of this and begins with data collection. So once data are generated it does not have to be manipulated manually at any point. Thus, the redundant or wasted efforts of data manipulation and the risk of data corruption is eliminated. An example of a similar process is browsing the internet. Because the world has agreed to HTML as a standard for transmitting website data, we can all use whatever browser we want, because all browsers can read HTML and present it on your computer screen. Once DIGGS is the worldwide standard for geotechnical data, we won’t have to worry about translating or manipulating data when we get it or need to share it, and we can use whatever tool we want to manipulate and view the data. Just as most people who use the internet do not need to understand HTML, as users of geotechnical data, most geotechnical specialists will not have to be DIGGS “coders” and get into the details within the DIGGS files. Multiple software vendors will hopefully support DIGGS and develop tools to import and export from DIGGS. - Are there other “options” to DIGGS?
The Association of Geotechnical & Geoenvironmental Specialists (AGS) data format has been used in the United Kingdom and internationally for several years. While it has the benefit of “standardization” the technology used in the AGS system (i.e., comma separated values) is dated and limiting in its functionality. DIGGS, on the other hand, is an XML schema which is much more flexible than comma separated values. XML is THE standard used by the software community for data transfer. The Geotechnical Virtual Data Center concept originally developed as part of COSMOS utilized an early version of DIGGS to demonstrate functionality and capability but was unsuccessful in garnering financial support for industry wide sustainable development. - Where do I find the files and reference materials?
https://www.geoinstitute.org/special-projects/diggs - The website includes the DIGGS schema files and information regarding history, format, training, management, references, etc. - Why are there multiple schema files?
The schema files are separated to make them more easily manageable.- Complete Schema: http://diggsml.org/schemas/2.0.b/Complete.xsd (Ties the files together)
- Geotechnical Schema: http://diggsml.org/schemas/2.0.b/Geotechnical.xsd - (Contains the vetted test objects for data transfer)
- Kernel Schema: http://diggsml.org/schemas/2.0.b/Kernel.xsd (link to OpenGIS and W3 as well as the object structure for headers, testing, geology, samples, etc.)
- Is DIGGS tied to the Open Geospatial Consortium?
Not yet. This is something that we anticipated looking into further this year. The original development consultant (Galdos Systems, Inc.) and our current development partner (Image Matters) were intimately involved in this organization and the intention has always been to become connected once we got DIGGS functional. - What tests have been vetted and are ready for use/adoption?
The following tests are included in the current schema:Atterberg Limits Lab includes both Fall Cone and Casagrande test for liquid limits Driven Penetration test In situ includes SPT and other hammer-based penetration tests Particle Size Lab includes both sieve and hydrometer tests Chemical tests Lab Compaction Lab includes Proctor, Modified Proctor, Porter Static, California Test 216 Density Lab PhotoIonization Detector test Lab Specific Gravity Lab Shrinkage Limit Lab Static Cone Penetration test In situ e.g., CPT Water Content Lab Wireline logs In situ includes any kind of downhole sensors, including suspension velocity logs The following tests are being considered for future inclusion in the schema. Their procedure objects require review by subject matter experts before adoption:
Aggregate Abrasion Value
Aggregate Crushing Value
Aggregate Elongation Index
Aggregate Flakiness Index
Aggregate Impact Value
Aggregate Polished Stone Value
Aggregate Slake Durability
Aggregate Soundness
Aggregate Ten Percent Fines
Aggregate Water Absorption
Cation Exchange
Chalk Crushing Value
Compressive Strength
Consolidation
Direct Shear
Flame Ionization Detector
Flat Dilatometer
Frost Susceptibility
Hand Vane
In Situ CBR
In Situ Density
In Situ Penetrometer test
In Situ Permeability
In Situ Redox
In Situ Resistivity
In Situ Vane Shear
Lab CBR
Lab Penetrometer test
Lab Permeability
Lab Redox
Lab Velocity
Los Angeles Abrasion
MCV
Micro Deval
Point Load
Pressuremeter test
Pumping
Relative Density
Rock Porosity-Density
Schmidt Rebound Hardness
Shore Scleroscope Hardness
Suction
Triaxial test
DIGGS Implementation
- OK, I adopt DIGGS, now what do I get?
The question really is how do we obtain files, input them into the agency database, create tools to export DIGGS files as needed to generate reports and plots, etc. - My organization now uses gINT (or other software), do I have to buy new software and train everyone?
No, as vendors see and respond to the demand from users, they will integrate import and export functions for DIGGS into their software. As vendors further down the chain see the demand, they will integrate DIGGS output or input into their software to deliver products that will import DIGGS files and then manage the DIGGS-transferred data to generate graphs and reports of the data. - I get a DIGGS file from my consultants (because I require it). How do they generate it?
As noted, we are dependent on the software vendors to update their software to generate these files when there is a demand from their clients. Right now, we are discussing various options with the vendors and potentially will be developing example files and tools for conversion to make this process easier for vendors to implement. - What do I do with the DIGGS file?
That’s up to your internal agency or company procedures. Do you have a common geographic reference database that you want this data read into? Do you just store this for future retrieval? Do you want it read into your geotechnical software so you can generate your own graphs, reports, profiles? With DIGGS and support from software vendors, all of these are possible. - Can I simply dump the DIGGS file into my agency’s system (GIS, database, etc.)?
With a mapping tool, the data in a DIGGS file can be imported directly into your database. Even without setting up a mapping between the DIGGS schema and your agency’s database, you can also store the DIGGS file directly into your SQL database or save DIGGS files on a network server. - Can I dump the PDF presentation of the DIGGS file into my database and get any benefit?
No, a pdf file is information, not data. Continued reliance on pdf files will require manual manipulation of the information, which violates one of the first tenants of good data management, “only input data once.” - What will I do once I have the DIGGS files?
Archiving the data in the DIGGS files is at the discretion of the agency. Ideally, they will become the cornerstone of a geotechnical data management system encouraged by the agency. However, like a database, these files can be searched directly for information. This search could be for a single project file or across multiple projects. - Once I Integrate the DIGGS file into my database, do I need to keep the DIGGS file?
If you transfer all the data in the DIGGS file into a database and that database software can export DIGGS files, then there would not be a need to save these files other than for archival records, if desired. However, if you do not import ALL of the data in the original DIGGS file you would lose some data once the DIGGS file is eliminated. Remember that DIGGS is all about data “transfer.” - When will software vendors be able to export and import DIGGS files?
The short answer is – when the users demand it. Let your software vendors know that you are interested in being able to use DIGGS files with their software. Keynetix has already developed the DIGGS feedback tool which allows users to export data from CSV/Excel/gINT/HoleBASE SI to DIGGS. Dataforensics has developed an importer for CPT data in the DIGGS format. More development by all software vendors is needed as the geotechnical community begins using and mandating DIGGS. - Once adopted, how do I confirm that a DIGGS file is valid and complete?
Scott Deaton has demonstrated a “round trip” for CPT files where he exported CPT data in a DIGGS format, then re-imported it and compared the results. Users can also use XML validation tools which will automatically confirm whether the file is a valid DIGGS file. We can also have tools developed to provide summaries or tables of DIGGS file content to assist with such a review. One of the potential goals of the DIGGS Committee is to develop a “DIGGS Validation Tool” in the event that users eventually develop their own DIGGS files. - Are there tools currently available to natively search DIGGS files across a directory?
Yes, this is available and we can relatively easily develop this, but it will have limited functionality. - Are there tools currently available to quickly see what is in a DIGGS file?
Yes, this is available and we can relatively easily develop this, but it will have limited functionality. - Are there tools currently available to quickly see plots of some DIGGS data?
This has been done on a very simple scale as a demonstration, but again we see this as having limited functionality. - If consultants provide DIGGS files for a field or lab test, should I ask for anything else (spreadsheet, database, text file, pdf file, etc.)?
For the purist, no. In the interim and until software vendors support DIGGS, you may want some redundancy. - If I am requested to send data to someone, do I send a DIGGS file?
Yes, once DIGGS is the worldwide standard for geotechnical data. That is one of the real benefits of DIGGS. The advantage of this is that once everyone can easily read a DIGGS data file, you are seamlessly transmitting the data without any cut-and-paste requirements. In addition, you are not transmitting any interpretation of the data and thus not presupposing what the data means to the person making the request. - If I am requested to send data to someone, do I send them a copy of my database?
No, as this would mean that the requestor would then have to map your database to their database or “cut-and-paste” various fields. By sending a DIGGS file, you are essentially being “database agnostic” in that you do not need to know the structure or the software vendor for their database. - How do I export from my database to DIGGS format?
This will require export routines for your database to generate the DIGGS format. There will be some necessity to develop mapping algorithms to generate DIGGS files from an existing database that was never developed with the DIGGS schema in mind. - What is all of this going to cost my agency?
While we recognize that converting existing software to DIGGS will require work on the part of the software vendors, we hope that the volume and demand from a large user base will promote incentives for the vendors to include DIGGS in new releases of their software programs with minimal financial impact on the current users/clients. In addition, if your agency currently devotes little to no effort on data management, this transition will ultimately have a cost, as the agency will likely need to identify a “Data Manager” to help maintain the integrity of the new system. - How do I “sell” this adoption of DIGGS to my management?
Besides the flexibility to capture and store DATA as required, DIGGS presents the benefit of providing more capabilities and flexibility to use the data with whatever tools the agency has or will have. Data is the key to Asset Management. Ultimately, by having ready access to historic data, there should be less time and effort spent to find the old files and to minimize drilling of redundant borings.
Sustaining DIGGS
- Who manages DIGGS?
The Geo-Institute of ASCE is the current primary sponsor. We propose the development of a DIGGS Advisory Board to help provide direction and management of Technical Committees, Corporate Sponsors. We will discuss the role and scope of the Technical Committees. - How do I know the DIGGS organization will remain viable in the future if I adopt it?
ASCE has been around for more than 150 years. Like agencies, if our users want this, the Geo-Institute will be there to support and advance it. Ultimately, we have an opportunity to help make a transformational change to our practice. Think of this as similar to the work of committees for TRB, AASHTO, ASCE, ASTM, etc. - When will the remaining AGS data format be converted to DIGGS?
This will be a function of user interest and demand - How do we get others into DIGGS?
This will be a function of user interest and demand, but we see that this is will be one of the functions of the Technical Committees - What do we do if we find a problem with a file or a standard?
There will be a DIGGS Validator to verify the file against the schema. Compliance and development of tools will be within the purview and domain of one of the Technical Committees. - Can I have a say in the development and maintenance of DIGGS?
Yes. We are looking for volunteers to assist with various aspects of DIGGS and its development. If you are interested in participating in helping, please use the contact form to get in touch with us. - How can I support development and the long-term sustainability of DIGGS?
We are working on a Business Model to identify some of the financial realities associated with this effort.