ASAsense Binary Table (ABT) format: Difference between revisions

From ASAsense Documentation (staging)
← Older edit

Thomas (talk | contribs)

4 user groups
427 edits
VisualWikitext
No edit summary
Marked this version for translation
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
The ASAsense binary table format is an alternative for the CSV format, but for binary data with fixed row lengths. The structure is as follows:
<languages/>
<translate>
<!--T:1-->
The ASAsense Binary Table (ABT) format is an alternative to the CSV format, but for binary data with fixed row lengths. The structure is as follows:
 
<!--T:2-->
{| class="wikitable"
{| class="wikitable"
|+
! name
!name
! length (bytes)
!length (bytes)
! type
!type
! allowed values
!allowed values
! explanation
!explanation
|-
|-
|file_type
| file_type
|1
| 1
|u8
| u8
|1
| 1
|The internal number of the file type and version
| The internal number of the file type and version
|-
|-
|n_cols
| n_cols
|4
| 4
|u32
| u32
|1-u32::max
| 1-u32::max
|the amount of headers
| The number of columns
|-
|-
|col_widths
| col_widths
|1 x n_cols
| 4 × n_cols
|u32[n_cols]
| u32[n_cols]
|1-u32::max per column
| 1-u32::max per column
|n_cols u32 integers describing the byte-width of the columns
| n_cols u32 integers describing the byte width of the columns
|-
|-
|metadata
| metadata_length
|until '\0'
| 4
|char[]
| u32
|C-style string (ends when '\0' character is encountered, encoded in UTF8)
| 0-u32::max
|JSON-encoded metadata
| The length of the metadata field (can be 0 if it is to be skipped)
|-
|-
|data
| metadata
|variable, until EOF
| metadata_length
|byte[]
| char[]
|any
| string of fixed length (so no '\0' character marks the end of the string)
|The actual data
| JSON-encoded metadata
|-
| data
| variable, until EOF
| byte[]
| any
| The actual data
|}
|}
The JSON-metadata should be an object with the following structure
 
<!--T:3-->
The JSON metadata should be an object with the following structure:
 
<!--T:4-->
{| class="wikitable"
{| class="wikitable"
|+
! field
!field
! required
!required
! type
!type
! explanation
!explanation
|-
|-
|n_rows
| n_rows
|false
| false
|number
| number
|The amount of rows in the file, if known upfront (can be omitted for streaming data)
| The number of rows in the file, if known upfront (can be omitted for streaming data)
|-
|-
|comment
| comment
|false
| false
|string
| string
|a free string
| A free string
|-
|-
|columns
| columns
|true
| true
|a list of column definition objects
| a list of column definition objects
|an list of objects describing the columns in more details (should have n_cols entries)
| A list of objects describing the columns in more detail (should have n_cols entries)
|-
| extra
| false
| JSON object
| A JSON object that can be specified freely
|}
|}
A "column definition object" should have the following format
 
<!--T:5-->
A "column definition object" should have the following format:
 
<!--T:6-->
{| class="wikitable"
{| class="wikitable"
!field
! field
!required
! required
!type
! type
!explanation
! explanation
|-
|-
|name
| name
|false
| false
|string
| string
|an optional name of the column
| An optional name of the column
|-
|-
|comment
| comment
|false
| false
|string
| string
|a free string
| A free string
|-
|-
|datatype
| datatype
|true
| true
|string
| string
|An identifier for the datatype of the column
| An identifier for the datatype of the column
|}
|}
The datatype identifiers can be anything, but the following should at least be supported by the reader:
 
<!--T:7-->
The datatype identifiers can be anything, but the reader should support at least the following:
 
<!--T:8-->
{| class="wikitable"
{| class="wikitable"
|+
! identifier
!identifier
! description
!description
|-
| utf8
| Interpret the data as a UTF-8 string (with fixed length; it does not need to be terminated with '\0')
|-
| int
| Interpret the data as a signed integer
|-
|-
|utf8
| uint
|interpret the data as an UTF8-string (with fixed length, does not need to be terminated with '\0')
| Interpret the data as an unsigned integer
|-
|-
|int
| float
|interpret the data as signed integer
| Interpret the data as floating-point data (only possible for 4- or 8-byte fields)
|-
|-
|uint
| bool
|interpret the data as unsigned integer
| Boolean value (0 = false, anything else = true)
|-
|-
|float
| numtype/div
|interpret as floating point data (only possible for 4 or 8 byte fields)
| "div" should be replaced by any floating-point number. The data is first interpreted as "numtype" (int, uint or float) and then divided by "div". This is mainly useful for storing decimal numbers in integer fields.
|-
|-
|numtype/div
| unix_timestamp(numtype/div)
|"div" should be replaced by any floating point number. The data is then first interpreted as "numtype" (int, uint or float) and then divided by "div". This is mainly useful for storing decimal numbers in integer fields.
| Interprets the data as specified within the brackets (so any numtype, with optional divider), but takes into account that it is a Unix timestamp (seconds since the Unix epoch, 1 January 1970 at UTC)
|}
 
<!--T:9-->
Other, unknown datatype identifiers should be treated by the reader as opaque binary data (bytes).
 
== Common extra metadata fields == <!--T:10-->
 
<!--T:11-->
In many ABT files used by ASAsense, the following extra metadata fields will be found:
 
<!--T:12-->
{| class="wikitable"
! Field name
! Field type
! Default value
! Description
|-
|-
|unix_timestamp
| sorted_by_timestamp
|Interprets the data as "uint" , but takes into account that it is a unix_timestamp (offset seconds since the Unix Epoch on January 1st, 1970 at UTC)
| bool
| true
| Indicates whether the data in the ABT file is sorted by timestamp (column 1)
|-
|-
|bool
| frame_width_s
|boolean value (0 = false, anything else = true)
| float
| NaN
| Indicates the frame width of one sample (for example the sampling period or spectrogram frame width)
|}
|}
Other, unknown datatype identifiers, should be treated as opaque binary data (bytes) by the reader.<blockquote></blockquote>
 
== CLI tool for ABT handling == <!--T:13-->
 
<!--T:14-->
To make the ABT format easier to use, a few precompiled binaries are available:
</translate>
* Linux
** [http://asasense.com/abt-binaries/abt-cli-ubuntu-22.04-x86_64.zip Ubuntu 22.04 x86_64]
** [http://asasense.com/abt-binaries/abt-cli-ubuntu-20.04-x86_64.zip Ubuntu 20.04 x86_64]
* [http://asasense.com/abt-binaries/abt-cli-win11-x64.zip Windows 11 x64]
<translate>
<!--T:15-->
The MD5 sums of the ZIP files are as follows:
</translate>
adcd6a227861bb6b188ef8c5748faae8  abt-cli-ubuntu-20.04-x86_64.zip
6f4c63cbd7416ead026f170c97b3ab89  abt-cli-ubuntu-22.04-x86_64.zip
39db54827d2634262d9d666b4262b212  abt-cli-win11-x64.zip

Latest revision as of 10:23, 14 October 2026

Other languages:

The ASAsense Binary Table (ABT) format is an alternative to the CSV format, but for binary data with fixed row lengths. The structure is as follows:

name length (bytes) type allowed values explanation
file_type 1 u8 1 The internal number of the file type and version
n_cols 4 u32 1-u32::max The number of columns
col_widths 4 × n_cols u32[n_cols] 1-u32::max per column n_cols u32 integers describing the byte width of the columns
metadata_length 4 u32 0-u32::max The length of the metadata field (can be 0 if it is to be skipped)
metadata metadata_length char[] string of fixed length (so no '\0' character marks the end of the string) JSON-encoded metadata
data variable, until EOF byte[] any The actual data

The JSON metadata should be an object with the following structure:

field required type explanation
n_rows false number The number of rows in the file, if known upfront (can be omitted for streaming data)
comment false string A free string
columns true a list of column definition objects A list of objects describing the columns in more detail (should have n_cols entries)
extra false JSON object A JSON object that can be specified freely

A "column definition object" should have the following format:

field required type explanation
name false string An optional name of the column
comment false string A free string
datatype true string An identifier for the datatype of the column

The datatype identifiers can be anything, but the reader should support at least the following:

identifier description
utf8 Interpret the data as a UTF-8 string (with fixed length; it does not need to be terminated with '\0')
int Interpret the data as a signed integer
uint Interpret the data as an unsigned integer
float Interpret the data as floating-point data (only possible for 4- or 8-byte fields)
bool Boolean value (0 = false, anything else = true)
numtype/div "div" should be replaced by any floating-point number. The data is first interpreted as "numtype" (int, uint or float) and then divided by "div". This is mainly useful for storing decimal numbers in integer fields.
unix_timestamp(numtype/div) Interprets the data as specified within the brackets (so any numtype, with optional divider), but takes into account that it is a Unix timestamp (seconds since the Unix epoch, 1 January 1970 at UTC)

Other, unknown datatype identifiers should be treated by the reader as opaque binary data (bytes).

Common extra metadata fields

In many ABT files used by ASAsense, the following extra metadata fields will be found:

Field name Field type Default value Description
sorted_by_timestamp bool true Indicates whether the data in the ABT file is sorted by timestamp (column 1)
frame_width_s float NaN Indicates the frame width of one sample (for example the sampling period or spectrogram frame width)

CLI tool for ABT handling

To make the ABT format easier to use, a few precompiled binaries are available:

The MD5 sums of the ZIP files are as follows: 

adcd6a227861bb6b188ef8c5748faae8  abt-cli-ubuntu-20.04-x86_64.zip
6f4c63cbd7416ead026f170c97b3ab89  abt-cli-ubuntu-22.04-x86_64.zip
39db54827d2634262d9d666b4262b212  abt-cli-win11-x64.zip
Retrieved from "https://docs-staging.asasense.com/index.php?title=ASAsense_Binary_Table_(ABT)_format&oldid=561"