Skip to main content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Research Data Management (RDM): Preservation Formats - What You Need To Know When Creating A DMP

UCT Libraries Research Data Services provide guidance and support for all aspects of the data lifecycle, from planning your data management strategy during the proposal phase through preserving your data at the conclusion of your project.

Overview

Different types of data are acquired, processed and stored (preserved and/or archived) in different ways and can be discipline specific. When starting a new project and creating a Data Management Plan - DMP, one of the first considerations to make should be to decide, in advance, which file formats to use. Many proprietary file formats are “containers” for standard file formats. By packaging them into these containers, a software and/or hardware developer can provide additional functionality, usually by streamlining a process, to analyse data acquired on their platform. However, this has the negative consequence of making these data less interoperable.

Moreover, file formats can be either lossless or lossy: that is, whether data is uncompressed (such as TIFF for images) or compressed (such as JPEG for images) to remove redundant information and thus reduce file size. It is common practice to do analyses on lossy data but this does not necessarily mean that these data should be the ones that should be kept for long-term storage. In this context, it is highly likely that the most important file to consider for long-term storage through its curation lifecycle is either the first file (that which was initially captured from an instrument) or a direct lossless standard file format version from this one (see also guide on raw data (to be available soon)).

Why Is It Necessary?

The key consideration to make is longevity and interoperability and to be as FAIR as much as possible - proprietary versus standard formats: equipment used, brand of software/hardware and research discipline may all be contributing factors. There is no guarantee that existing proprietary file formats will exist in the future. For example, many Microsoft file formats such as Word and Excel may be in common use now, but this does not negate the possibility that they will become obsolete in the future. Software and file format obsolescence will be even more pronounced for bespoke file formats that were created as part of an individual project.

How To Deal With This?

As an example, in the biomedical imaging field, a realisation of the huge variety of file formats that exist led to an initiative to make these interoperable. As part of the OMERO project, Bioformats is a software plugin which allows the conversion of multiple established proprietary and standard file formats. Image analysis software such as ImageJ (free and open source) have adopted Bioformats as a plugin to allow users to read and write their image data without having to consider their origin. However, such tools may not always exist for different disciplines, and a researcher should consider storing their acquired data in a standard format at the earliest available opportunity. Many (most?) commercial and open source software packages allow conversion of data into standard formats and this should be exploited.

During the course of the digital revolution, a number of file formats have been recognised to be the file formats of choice for longevity and interoperability.

As an example, the following table describes a variety of file formats for different disciplines that are either recommended or acceptable (from the UK Data Service):

When writing a DMP, researchers are advised to refer to tables such as this to help decide the best file formats to use for their project and to state this clearly.

Type of data Recommended formats Acceptable formats

Tabular data with extensive metadata

variable labels, code labels, and defined missing values

  • SPSS portable format (.por)
  • delimited text and command ('setup') file (SPSS, Stata, SAS, etc.)
  • structured text or mark-up file of metadata information, e.g. DDI XML file

proprietary formats of statistical packages: SPSS (.sav), Stata  (.dta), MS Access (.mdb/.accdb)

Tabular data with minimal metadata  column headings, variable names
  • comma-separated values (.csv)
  • tab-delimited file (.tab)
  • delimited text with SQL data definition statements
  • delimited text (.txt) with characters not present in data used as  delimiters
  • widely-used formats: MS Excel (.xls/.xlsx), MS Access (.mdb/.accdb), dBase (.dbf), OpenDocument Spreadsheet (.ods)
Geospatial data  vector and raster data
  • ESRI Shapefile (.shp, .shx, .dbf, .prj, .sbx, .sbn optional)  geo-referenced TIFF (.tif, .tfw)
  • CAD data (.dwg)
  • tabular GIS attribute data  Geography Markup Language (.gml)
  • ESRI Geodatabase format (.mdb)
  • MapInfo Interchange Format (.mif) for vector data
  • Keyhole Mark-up Language (.kml)
  • Adobe Illustrator (.ai), CAD data (.dxf or .svg)
  • binary formats of GIS and CAD packages
Textual data
  • Rich Text Format (.rtf)  plain text, ASCI (.txt)
  • eXtensible Mark-up Language (.xml) text according to an appropriate Document Type Definition  (DTD) or schema
  • Hypertext Mark-up Language (.html)
  • widely-used formats: MS Word (.doc/.docx)
  • some software-specific formats: NUD*IST, NVivo and ATLAS.ti
Image data TIFF 6.0 uncompressed (.tif)
  • JPEG (.jpeg, .jpg, .jp2) if original created in this format
  • GIF (.gif)
  • TIFF other versions (.tif, .tiff)
  • RAW image format (.raw)
  • Photoshop files (.psd)
  • BMP (.bmp)
  • PNG (.png)
  • Adobe Portable Document Format (PDF/A, PDF) (.pdf)
Audio data Free Lossless Audio Codec (FLAC) (.flac)
  • MPEG-1 Audio Layer 3 (.mp3) if original created in this format
  • Audio Interchange File Format (.aif)
  • Waveform Audio Format (.wav)
Video data
  • MPEG-4 (.mp4)
  • OGG video (.ogv, .ogg)
  • motion JPEG 2000 (.mj2)
AVCHD video (.avchd)
Documentation and scripts
  • Rich Text Format (.rtf)
  • PDF/UA, PDF/A or PDF (.pdf)
  • XHTML or HTML (.xhtml, .htm)
  • OpenDocument Text (.odt)
  • plain text (.txt)
  • widely-used formats: MS Word (.doc/.docx), MS Excel  (.xls/.xlsx)
  • XML marked-up text (.xml) according to an appropriate DTD or  schema, e.g. XHMTL 1.0