Supporting CUE files in MPD: syntactic and semantic consistency with standard formats for audio metadata

Support for CUE files in MPD and its clients is not as trivial and simple as it may appear, with respect to the syntactic and semantic consistency with the involved formats for audio metadata.

As I am interested in this subject, I have done an extensive and detailed work on this issue, which is available for download: SUPPORTING CUE FILES IN MPD (pdf, 319KB).

In this work, the specifications of the main formats for audio metadata (CCDB (freedb and MusicBrainz), TOC/CD-TEXT, CUESHEET, Vorbis comment and, obviously, MPD tags) were first examined with the aim of overcoming the critical issues.

Having considered the results of the analisys carried out on these formats, it was possible to deepen the syntactic and semantic knowledge of the CUESHEET format, also in relation to the use conventions generally adopted in CUE files.

Then, the correlations between CUE and MPD metadata were discussed, identifying a common set of audio metadata to be supported, exactly specifying their meaning and use, and finally understanding how to establish correlations so as to ensure syntactic and semantic consistency.

Also, this has meant to precisely specify the syntactic and semantic use of the involved (standard and non-standard) CUE commands, and to define a proposal to extend the MPD specification introducing several new extra “extended” tags, that were necessary to achieve the optimal correlations.

All of this has been reported in a reference summary for CUE and MPD audio metadata (both standard and extended), which also contains the proposals to standardize and extend both the CUE commands and the MPD tags.

Finally, a pseudocode has been proposed, which implements the parsing of CUE file commands in relation to the MPD tags so as to realize the appropriate correlations between the medatates. Some synthetic examples showing what result should be obtained with this implementation were also given.

The hope is that the results and proposals of this work could be useful for improving CUE files support in MPD and its clients, so allowing users to better manage and use their CUE files to organize and listen to their digital audio collection.

The proposed pseudocode is also listed here below (please, refer to the previously indicated work for the full understanding of the code flow).

Comments and suggestions are welcome!


// 
// cuesheet-parsing
// 
// Pseudocode -- CUE file metadata parser for MPD
// 
// Version: 1.1  2018-11-10
// 
// Copyright (c) 2018+ alexus. Released under the GNU General Public License, Version 3.
// 
// This program is free software: you can redistribute it and/or modify it under the 
// terms of the GNU General Public License as published by the Free Software Foundation, 
// either version 3 of the License, or (at your option) any later version.
//
// This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; 
// without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 
// See the GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License along with this program. 
// If not, see .

// NOTES:
// - A 'special_genres' option is supposed to be available through the MPD configuration or 
//   the MPD client configuration, containing the list of genres for which the artist may not 
//   correspond to the performer, such is for the 'Classical' genre (e.g. 'Classical, Opera, 
//   Chamber Music, Jazz'). If this feature is not available then the procedure should be adapted 
//   as indicated.
// - A 'various_equivs' option is supposed to be available through the MPD configuration or 
//   the MPD client configuration, containing a list of synonyms or equivalent alternatives for 
//   'Variuos' (e.g. 'Various artist, AA.VV., VVAA'). If this feature is not available then the 
//   procedure should be adapted as indicated.
// - The CUE file being parsed is referenced here as 
// - The MPD database is referenced here as 'mpd-db[][]'
// - The GETCUECOMMAND() subprocedure and the ISCONTAINED() function are defined at the bottom 
//   of the main pseudocode
// - Some subprocedures are referenced (called) in this pseudocode although they are not defined 
//   here (but just simply described)
// - The '&&' operator is intended for string concatenation
// - Variables in this pseudocode are supposed to be craeted when set for the first time, being 
//   initially empty (not defined or null)
// - Indexes of all the arrays in this pseudocode are supposed starting from 1
// - Runtime exceptions are generally not handled here
// - More details and code explanations are among the comments to the pseudocode

// IMPORTANT WARNING:
// Please, to avoid uncoherent semantic results, **DO NOT** alter the "cascading" way parsing is done!
// Be aware that changes that may appear as possible optimizations of the code flow, could easily 
// lead to assignments that lose the semantic connections between the tags.
//

BEGIN

// initialize the cuelist[] array, containing the list of "clean" CUE tags
SET cuelist[] = ('FILE', 'TRACK', 'PREGAP', 'INDEX', 'POSTGAP', 
                 'COMPOSER', 'PERFORMER', 'SONGWRITER', 'TITLE', 'GENRE', 'DISCID', 'ALBUMID', 
                 'CATALOG', 'ISRC', 'DATE', 'ORIGINALDATE', 'DISCNUMBER', 'TOTALDISCS', 'DISCTOTAL', 
                 'MEDIA', 'SOURCEMEDIA', 'ORGANIZATION', 'LABEL', 'COPYRIGHT', 'LOCATION', 'REM')


// open the CUE file for reading
OPEN  FOR READ
// read the CUE file line by line, storing CUE commands and respective tags values
SET lineno = 0  // initialize variable for CUE lines count
SET ntracks = 0 // initialize variable for CUE tracks count
SET headerlastlineno = -1 // initialize variable for store the last line number of the CUE header
WHILE NOT EOF 
    READ cueline FROM 
    SET lineno = lineno + 1
    // look for the last line in the CUE file header section (which is the line just above 
    // the *first* 'FILE' CUE command; the track section of the CUE file starts just below)
    IF ((headerlastlineno IS -1) AND (cuecmd[lineno]->tag IS 'FILE')) THEN
        SET headerlastlineno = lineno - 1
    ENDIF
    // count the 'TRACK' commands in the CUE file
    IF (cuecmd[lineno]->tag IS 'TRACK') THEN
        SET ntracks = ntracks + 1
    ENDIF
    // call the GETCUECOMMAND() subprocedure to parse the current CUE line and store the 
    // contained CUE command to a bi-dimensional indexed array cuecmd[][] where:
    // - the containing CUE tag in the current cueline being assigned to cuecmd[lineno]->tag
    //   examples: 'TITLE', 'GENRE', 'COMPOSER' etc.
    // - the corresponding value (if not empty) being assigned to cuecmd[lineno]->val 
    //   examples: 'Autobahn', 'Classical', 'Mozart' etc.
    // examples:
    //      'REM GENRE "Alternative Rock"' line parsing would give:
    //              cuecmd[lineno]->tag = 'GENRE'
    //              cuecmd[lineno]->val = 'Alternative Rock'
    //      'REM TRACK 03 AUDIO' line parsing would give:
    //              cuecmd[lineno]->tag = 'TRACK'
    //              cuecmd[lineno]->val = '03'
    //      'INDEX 01 01:17:43' line parsing would give:
    //              cuecmd[lineno]->tag = 'INDEX'
    //              cuecmd[lineno]->val = '01  01:17:43"
    CALL GETCUECOMMAND(cueline, cuelist[], lineno, cuecmd[][])
ENDWHILE
// once read close the CUE file
CLOSE 

// initialize an array list of genres for which the artist may not correspond to the performer 
// by getting items through a GETCONFOPT() function subprocedure
// NOTE: a 'special_genres' option is supposed to be available through the MPD configuration or 
//       the MPD client configuration
SET specialgenres[] = GETCONFOPT('special_genres')  // NOTE: this subprocedure is not defined here
// initialize a flag to indicate if the album genre is in the specialgenres[] array list
SET isspecialgenre = FALSE

// initialize an array list of synonyms or equivalent alternatives for 'Variuos' by getting items 
// through a GETCONFOPT() function subprocedure
// NOTE: a 'various_equivs' option is supposed to be available through the MPD configuration or 
//       the MPD client configuration
SET variousequivs[] = GETCONFOPT('various_equivs')  // NOTE: this subprocedure is not defined here

// now parsing the CUE header section...
// go down throught the cuecmd[][] array staying in the range of the CUE file header section 
// (from the 1st line down to the last line in the header section)
FOR lineno = 1 TO headerlastlineno
    // store in a header[] array the values which will be need later when parsing the track section
    IF NOT (IS EMPTY cuecmd[lineno]->val) THEN  
        CASE cuecmd[lineno]->tag IS
            'REM':  
                // note: there may be more instances of REM in the header
                SET header[comment] = header[comment] && ' ' && cuecmd[lineno]->val 
                BREAK
            'GENRE':
                SET header[genre] = cuecmd[lineno]->val
                // check if the header[genre] value is contained in the specialgenres[] array list
                //  NOTE: if the 'special_genres' option is not supported then the next code line 
                //        should be just:  IF (header[genre] IS 'Classical') THEN
                IF (ISCONTAINED(header[genre], specialgenres[]) IS TRUE) THEN
                    SET isspecialgenre = TRUE
                ENDIF
                BREAK
             'TITLE':
                SET header[album] = cuecmd[lineno]->val
                BREAK
            'PERFORMER':
                SET header[performer] = cuecmd[lineno]->val
                // check if header[performer] is in the variousequivs[] array list
                // NOTE: if the 'various_equivs option' is not supported then the whole contition 
                // in the next code lines should be removed
                IF (ISCONTAINED(header[performer], variousequivs[]) IS TRUE) THEN 
                    SET header[performer] = 'Various'
                ENDIF
                BREAK
            'COMPOSER':
                SET header[composer] = cuecmd[lineno]->val
                BREAK

            // ... put here similar assignments for other standard CUE commands: 
            //     'DATE, 'CATALOG', 'SONGWRITER'
            
            // ... put here similar assignments  for other extended non-standard CUE commands:
            //     (if they will be managed): 'DISCID', 'ALBUMID', 'ORIGINALDATE', 'MEDIA', 
            //     'DISCNUMBER', 'TOTALDISCS', 'ORGANIZATION', 'COPYRIGHT', 'LOCATION'
            
        ENDCASE
    ENDIF
    //  NOTE: if the 'special_genres' option is not supported then the next code line should be:
    //        IF ((header[genre] IS 'Classical') AND (header[composer] IS NOT EMPTY)) THEN
    IF ((isspecialgenre IS TRUE) AND (header[composer] IS NOT EMPTY)) THEN
        SET header[albumartist] = header[composer]
    ELSE
        SET header[albumartist] = header[performer]
    ENDIF
ENDFOR

// now parsing the CUE tracks section...
// continue go down throught the cuecmd[][] array but now starting from the top of the track section
SET trackno = 1  // inizialize variable for track indexing 
FOR lineno = headerlastlineno + 1 to nlines
    // if command is 'FILE': 
    // get the current audio file and restart timecode positioning for its tracks
    IF (cuecmd[lineno]->tag IS 'FILE') THEN
        // call the SETFILE() subprocedure to parse cuecmd[lineno]->val and set the audio file path 
        // and type (i.e. WAVE) in the MPD database
        CALL SETFILE(cuecmd[lineno]->val)  // NOTE: this subprocedure is not defined here
        // call the RESTARTPOSITIONTIME() subprocedure to restart timecode positioning counts 
        // for the tracks related to the current audio file to which they belong
        CALL RESTARTPOSITIONTIME()  // NOTE: this subprocedure is not defined here
        // no further action is needed now, so jump directly to read next line
        NEXTFOR
    ENDIF
    // if command is 'TRACK':
    IF (cuecmd[lineno]->tag IS 'TRACK') THEN
        // assign the track number as is specified in the CUE file
        SET mpd-db[track[trackno]]->track = cuecmd[lineno]->val
        // assign tags taken from the header section to the current track
        SET mpd-db[track[trackno]]->comment = header[comment]
        SET mpd-db[track[trackno]]->genre = header[genre]
        SET mpd-db[track[trackno]]->album = header[album]
        SET mpd-db[track[trackno]]->albumartist = header[albumartist]
    
        // ... put here similar assignments for all the other CUE commands as already stored 
        //     in the header[] array, when parsing the CUE header section before ...
        
        // now go ahead to the next track...
        SET trackno = trackno + 1
        // no further action is needed now, so jump to read next line
        NEXTFOR
    // if command is neither 'FILE' nor 'TRACK':
    ELSE
        // still being inside the current track, now set the remaining specific tags for the track...
        IF ( (cuecmd[lineno]->tag IS 'PREGAP') OR 
             (cuecmd[lineno]->tag IS 'INDEX') OR 
             (cuecmd[lineno]->tag IS 'POSTGAP') ) THEN
            // call the UPDATEPOSITIONTIME() subprocedure to update the timecode position 
            // calculation for the current track (trackno) inside the current audio file 
            // which it belongs
            CALL UPDATEPOSITIONTIME()   // NOTE: this subprocedure is not defined here
            // no further action is needed now, so jump to read next line
            NEXTFOR
        ENDIF
        CASE cuecmd[lineno]->tag IS
            'REM':  
                // note: append the track comment...
                IF (cuecmd[lineno]->val IS NOT EMPTY) THEN 
                    SET mpd-db[track[trackno]]->comment = 
                        mpd-db[track[trackno]]->comment && ' ' && cuecmd[lineno]->val
                ENDIF
                BREAK
            'TITLE':
                IF (cuecmd[lineno]->val IS NOT EMPTY) THEN 
                    SET mpd-db[track[trackno]]->title = cuecmd[lineno]->val
                ENDIF
                BREAK
            'PERFORMER':
                IF (cuecmd[lineno]->val IS NOT EMPTY) THEN
                    SET mpd-db[track[trackno]]->performer = cuecmd[lineno]->val
                ELSE
                    IF (mpd-db[track[trackno]]->albumartist IS NOT 'Various') THEN  
                        SET mpd-db[track[trackno]]->performer = mpd-db[track[trackno]]->albumartist
                    ENDIF
                ENDIF
                BREAK
            'COMPOSER':
                IF (cuecmd[lineno]->val IS NOT EMPTY) THEN 
                    SET mpd-db[track[trackno]]->composer = cuecmd[lineno]->val
                ENDIF
                BREAK

            // ... put here similar cases for assign the remaining specific tags for the 
            //     track: 'ISRC', 'SONGWRITER'

        ENDCASE
        // now set artist...
        //  NOTE: if the 'special_genres' option is not supported then the next code line should be:
        //        IF (mpd-db[track[trackno]]->genre IS 'Classical') THEN
        IF (isspecialgenre IS TRUE) THEN
        // artist when genre is 'Classical':
            IF (mpd-db[track[trackno]]->composer IS NOT EMPTY) THEN
                SET mpd-db[track[trackno]]->artist = mpd-db[track[trackno]]->composer
            ELSE
                IF (mpd-db[track[trackno]]->albumartist IS NOT 'Various') THEN  
                    SET mpd-db[track[trackno]]->artist = mpd-db[track[trackno]]->albumartist
                ENDIF
            ENDIF
        ELSE
        // artist when genre is NOT 'Classical':
            IF (mpd-db[track[trackno]]->performer IS NOT EMPTY) THEN
                SET mpd-db[track[trackno]]->artist = mpd-db[track[trackno]]->performer
            ELSE
                IF (mpd-db[track[trackno]]->albumartist IS NOT 'Various') THEN  
                    SET mpd-db[track[trackno]]->artist = mpd-db[track[trackno]]->albumartist
                ENDIF
            ENDIF
        ENDIF
        // and finally set the fallback assignments 
        // (for genres, see: Sony CDTEXT genres list; for album, albumartist, artist and title,
        // 'Unknown' is as used in freedb, MusicBrainz etc.)
        IF (IS EMPTY mpd-db[track[trackno]]->genre) THEN SET mpd-db[track[trackno]]->'Not Defined'
        IF (IS EMPTY mpd-db[track[trackno]]->album) THEN SET mpd-db[track[trackno]]->'Unknown'
        IF (IS EMPTY mpd-db[track[trackno]]->albumartist) THEN SET mpd-db[track[trackno]]->'Unknown'
        IF (IS EMPTY mpd-db[track[trackno]]->title) THEN SET mpd-db[track[trackno]]->'Unknown'
        IF (IS EMPTY mpd-db[track[trackno]]->artist) THEN SET mpd-db[track[trackno]]->'Unknown'
    ENDIF
ENDFOR

// Subprocedure GETCUECOMMAND()
//      parse a CUE line (cueline) corresponding to the lineno-th line of the CUE file, looking for 
//      a CUE command (listed in cuelist[]) and store the contained CUE tag and value to the 
//      corresponding element (cuecmd[lineno][]) in the cuecmd[][] array
// NOTES:
//      - the REGEXREPLACE(string, test, options, replace) is intended to be a function where:
//          + string: input text
//          + test:  matching regexp 
//          + replace: substitution regexp
//        and the return value is:
//          + NULL, if test DOES NOT match string
//          + the result of the regexp match and substitution, if test DOES match string
//      - the && operator is intended for string concatenation
//      - the REM command must be handled using a different regexp than the other commands, 
//          this is to distinguish e.g. 'REM hey world' from 'REM COMPOSER hey world'
//
PROCEDURE GETCUECOMMAND(cueline, cuelist[], lineno, cuecmd[][])
    FOR i IN cuelist[]
        // set the regexp test (REM needs a different test than the other commands)
        IF (cuelist[i] IS NOT 'REM') THEN
            // - firstly look for all the leading white spaces and tabs, if present
            // - then look for 0 or 1 occurrences of 'REM' followed by a white space or a tab
            // - finally look for 1 occurrence of cuelist[i] followed by a white space or a tab
            SET test = '/^[ \t]*(REM[ \t]){0,1}' && cuelist[i] && '[ \t]+/'
        ELSE
            // - firstly look for all the leading white spaces and tabs, if present
            // - then just look for 1 occurrence of 'REM' followed by a white space or a tab
            SET test = ^[ \t]*(REM[ \t])+/'
        ENDIF
        // now do the regexp matching & substitution calling the REGEXREPLACE function with:
        //   - options='i' (case insensitive, non global)
        //   - replace = NULL
        // and assign the result to command
        // for example:
        //   when:       cueline=' REM COMPOSER "Mozart"   ' and cuelist[i]='COMPOSER'
        //   will be:    REGEXREPLACE(' REM COMPOSER "Mozart"   ', 
        //                            '/^[ \t]*REM[ \t]+){0,1}"COMPOSER[ \t]+/', 'i', NULL)
        //   and this will return "Mozart" (with the final white spaces)
        SET command = REGEXREPLACE(cueline, test, 'i', NULL)
        // when a match is successful (the CUE command in the cueline was found)...
        IF (command IS NOT EMPTY) THEN 
            // assign the current CUE tag (which is cuelist[i])
            SET cuecmd[lineno]->tag = cuelist[i]
            // assign the current CUE command value after stripping all the leading and final white 
            // spaces and tabs and also the quotation marks (")
            // (note this involves the use of the global option ('g') with the regexp test)
            SET test = '^[ \t]*\"{0,1}[ \t]*|[ \t]*\"{0,1}[ \t]*$'
            SET cuecmd[lineno]->val = REGEXREPLACE(command, test, 'g', NULL)
            // done! no further action is needed... so exit...
            ENDFOR
        ENDIF
    ENDFOR
END PROCEDURE

// Function ISCONTAINED(item, itemsarray[])
//      check if the value of item matches the value of one element in the itemsarray[] 
// NOTES:
//      Return: TRUE if a match is found; FALSE if a match is NOT found.
//
FUNCTION ISCONTAINED(item, itemsarray[])
    FOR i IN itemsarray[]
        IF (item = itemsarray[i]) THEN RETURN TRUE
    ENDFOR
    RETURN FALSE
END FUNCTION

END

// end of cuesheet-parsing pseudocode

disktemp

The disktemp script can output and notify disk devices temperatures (reading SMART data) on a GNU/Linux system.

The script output disk devices temperatures (echoing smartctl — so you do not need other tools or deamons such as hddtemp) and send a notification (using notify-send) when the temperature is over its limit. Outputs and notifies could be used by desktop panels, applets and notifiers to monitor disks temperatures.

See inside the script code for more details.

The disktemp script is free software, released under the terms of the GNU General Public License version 3 or (at your option) any later version.

Comments, suggestions and bug reports are welcome.

#!/bin/bash
#
# NAME
# 	disktemp
# 
# SYNOPSIS
# 	Output and notify disk devices temperatures reading SMART data
# 
# DESCRIPTION
# 	This script is intended to output and notify disk devices temperatures 
# 	reading SMART data. Outputs and could be used by desktop panels, applets 
# 	and notifiers to monitor disks temperatures.
# 	The script output disk devices temperatures (echoing smartctl) and send 
# 	a notification (using notify-send) when the temperature is over its limit.
# 	Disk device list with respective labels and temperature limits should be
# 	configured inside the script itself.
# 	Temperature are read in Celsius degrees (the 'RAW_VALUE' of the SMART 
# 	vendor attribute 'Temperature_Celsius').
# 	If smartctl cannot read disk temperature (also if due to errors in the 
# 	configuration data inside the script) then the script output a 0 value and 
# 	sends an error notification.
# 	The script must run with superuser rights.
# 
# VERSION
# 	2.0 (2018-01-25)
# 
# AUTHOR
# 	Alexus
# 	
# COPYRIGHT
# 	Copyright (c) 2016+ Alexus
# 	License: GPLv3+ (GNU GPL version 3 or later)
# 	This program is free software: you can redistribute it and/or modify
# 	it under the terms of the GNU General Public License as published by
# 	the Free Software Foundation, either version 3 of the License, or
# 	(at your option) any later version.
# 	This program is distributed in the hope that it will be useful,
# 	but WITHOUT ANY WARRANTY; without even the implied warranty of
# 	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# 	GNU General Public License for more details.
# 	You should have received a copy of the GNU General Public License
# 	along with this program.  If not, see .
# 	

## @file
## @name       disktemp
## @brief      Output and notify disk devices temperatures reading SMART data
## @author     alexus
## @copyright  GPLv3+
## @version    2.0
## @date       2018-01-25
## @pre        smartctl
## @pre        notify-send
## @note       Must run with superuser rights


# Declarations
declare -a disk_dev
declare -a disk_lbl
declare -a disk_tmax
declare -i temp  # °C
declare -i alarm_ms  # milliseconds
declare -i errcode
declare smartdata smartstat temp
declare out out_pre out_post
declare item

# ### Script configuration ###
disk_dev[0]='/dev/sda'
disk_lbl[0]='a:'
disk_tmax[0]=55
disk_dev[1]='/dev/sdb'
disk_lbl[1]='b:'
disk_tmax[1]=55
alarm_ms=3000
out_pre='| '
out_post=''
#disk_dev[2]='--device=3ware,0 /dev/twe0'
#disk_lbl[2]='c0:'
#disk_tmax[2]=55
#disk_dev[3]='--device=3ware,1 /dev/twe0'
#disk_lbl[3]='c1:'
#disk_tmax[3]=55
alarm_ms=3000
out_pre='| '
out_post=''
# ### End script configuration ###

# Outputs and notifications
for item in "${!disk_dev[@]}" ; do
	# read SMART data
	smartdata="$(sudo smartctl -A ${disk_dev[item]} 2>/dev/null)"
	smartstat=$?
	# read and output disk temp (awk used to strip the line feed)
	temp="$(echo "$smartdata" | grep -m 1 -i Temperature_Celsius | awk "{print \$10}")"
	out+="${disk_lbl[item]}$temp "
	# ckeck, notify, output
	if [[ $smartstat -ne 0 ]] || [[ $temp -le 0 ]] ; then 
		notify-send -u "critical" -t "$alarm_ms" "ERROR READING DISK!" "${0##*/}: cannot read temperature from device '${disk_dev[item]}'.\nPlease:\n- check if device supports SMART, or\n- check disk's health, or\n- check internal script configuration."
		errcode=1
	elif [[ "$temp" -gt "${disk_tmax[item]}" ]] ; then 
		notify-send -u "critical" -t "$alarm_ms" "DISK TEMPERATURE ALARM!" "Device ${disk_dev[item]}: $temp C (limit is: ${disk_tmax[item]} C)."
	fi
done
echo -n "$out_pre$out$out_post"
# Exit
if [[ -z $errcode ]] ; then exit 0 ; else exit 1 ; fi

mdalerts, mdcheck

Here below are two useful Bash scripts to monitor mdadm software RAIDs on a GNU/Linux system.

The mdalerts script is intented to output events detected by mdadm –monitor, thus it should be set in the PROGRAM line of the mdadm.conf file in order to be run by mdmadm. Arguments are passed by mdmadm.
When mdadm detect any event, the script output the event details (using echo) and also send a notification (using notify-send) to desktop applets (panels and notifiers).

The mdcheck script is intented to check and output the status of mdadm devices. It check and output (using echo) the status of mdmadm RAID devices and also send a notification (using notify-send) to desktop applets (panels and notifiers) when a device is not clean.

For both scripts, the mdadm devices list should be configured inside the code. Superuser rights are not required.

For more details and settings, please look inside the code. See also: man mdadm and man mdadm.conf.

The mdalerts and the mdcheck are free software, released under the terms of the GNU General Public License version 3 or (at your option) any later version.

Comments, suggestions and bug reports are welcome.

mdalerts

#!/bin/bash
#
# NAME
# 	mdalerts
# 
# SYNOPSIS
# 	Output (using echo and notify-send) events detected by mdadm monitor.
# 
# DESCRIPTION
# 	This script is intented to output events detected by 'mdadm --monitor', 
# 	thus it should be set in the PROGRAM line of the 'mdadm.conf' file in order 
# 	to be run by mdmadm. Arguments are passed by mdadm.
# 	When mdadm detect any event, the script output the event details (using 
# 	echo) and also send a notification (using notify-send) to desktop applets 
# 	(panels and notifiers).
# 	The mdadm devices list should be configured inside the script itself.
# 	The script do not need superuser rights.
# 	See also: 'man mdadm' and 'man mdadm.conf'.
# 
# VERSION
# 	1.0 (03/06/2017)
# 
# AUTHOR
# 	Alexus
# 	
# COPYRIGHT
# 	Copyright (c) 2016+ Alexus
# 	License: GPLv3+ (GNU GPL version 3 or later)
# 	This program is free software: you can redistribute it and/or modify
# 	it under the terms of the GNU General Public License as published by
# 	the Free Software Foundation, either version 3 of the License, or
# 	(at your option) any later version.
# 	This program is distributed in the hope that it will be useful,
# 	but WITHOUT ANY WARRANTY; without even the implied warranty of
# 	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# 	GNU General Public License for more details.
# 	You should have received a copy of the GNU General Public License
# 	along with this program.  If not, see .
# 	

## @file
## @name       mdalerts
## @brief      Output events detected by 'mdadm --monitor'
## @author     alexus
## @copyright  GPLv3+
## @version    1.0
## @date       03/06/2017
## @pre        mdadm, notify-send
## @note       Do not need superuser rights
## @param      string  $1  event (passed by mdadm)
## @param      string  $2  md device (passed by mdadm)
## @param      string  $3  related component device (possibly passed by mdadm)
## @return     int     1   parent command was not mdadm

# Declarations
declare -i alarm_ms  # milliseconds
declare    parent_cmd='mdadm'
declare    msg

# ### Script configuration ###
alarm_ms=10000
# ### End script configuration ###

# Check parent command
if [[ $parent_cmd != "$(ps -o comm= $PPID)" ]] ; then echo -e "ERROR! This script should be run by mdadm.\nSet it in the PROGRAM line of the 'mdadm.conf' file.\See: 'man mdadm'.\n"; exit 1 ;fi

# Outputs and notifications
msg="Event: $1, Device: $2"
if [[ -n "$3" ]] ; then msg+=", Component: $3" ; fi
echo "MD MONITOR EVENT!" "$msg$"
notify-send -u "normal" -t "$alarm_ms" "MD MONITOR EVENT!" "$msg"

# Exit
exit 0

mdcheck

#!/bin/bash
#
# NAME
# 	mdcheck
# 
# SYNOPSIS
# 	Check and output mdadm devices status.
# 
# DESCRIPTION
# 	This script is intented to check and output the status of mdadm devices.
# 	The script check and output (using echo) the status of mdadm devices and 
# 	also send a notification (using notify-send) to desktop applets (panels and 
# 	notifiers) when a device is not clean.
# 	The mdadm devices list should be configured inside the script itself.
# 	The script do not need superuser rights.
# 	See also: 'man mdadm' and 'man mdadm.conf'.
# 
# VERSION
# 	1.0 (03/06/2017)
# 
# AUTHOR
# 	Alexus
# 	
# COPYRIGHT
# 	Copyright (c) 2016+ Alexus
# 	License: GPLv3+ (GNU GPL version 3 or later)
# 	This program is free software: you can redistribute it and/or modify
# 	it under the terms of the GNU General Public License as published by
# 	the Free Software Foundation, either version 3 of the License, or
# 	(at your option) any later version.
# 	This program is distributed in the hope that it will be useful,
# 	but WITHOUT ANY WARRANTY; without even the implied warranty of
# 	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# 	GNU General Public License for more details.
# 	You should have received a copy of the GNU General Public License
# 	along with this program.  If not, see .
# 	

## @file
## @name       mdcheck
## @brief      Output the status of mdadm devices for monitoring applets
## @author     alexus
## @copyright  GPLv3+
## @version    1.0
## @date       03/06/2017
## @pre        mdadm, notify-send
## @note       Do not need superuser rights
## @return     int  1  one or mode mdadm devices is not clean working

# Declarations
declare raid
declare raid_status='ok'
declare raids
declare unit_state
declare unit_recovery
declare unit_resync
declare out_pre
declare out_post
declare -i alarm_ms  # milliseconds
declare -i exitCode=0

# ### Script configuration ###
raids="md1 md2 md3 md4"
alarm_ms=5000
out_pre='| MD:'
out_post=''
# ### End script configuration ###

# Checking...
for raid in $raids ; do
	mdstat="$(grep -A2 ^$raid "/proc/mdstat")"
	if [[ $(echo -e $mdstat | grep '\[.*F.*\]') ]] ; then unit_state="failed" ; raid_status='KO'
	elif [[ $(echo -e $mdstat | grep '\[.*_.*\]') ]] ; then unit_state="degraded" ; raid_status='KO'
	else unit_state="working" ; fi
	unit_recovery=$(echo -e $mdstat | grep recovery | awk '{print $16}')
	unit_resync=$(echo $mdstat | grep resync | awk '{print $16}')
	if [[ $raid_status == "KO" ]] ; then 
		notify-send -u "critical" -t "$alarm_ms" "MD RAID ALARM!" "${0##*/} - Device '$raid' has $unit_state unit(s). Recovery: $unit_recovery. Resync: $unit_resync."
		exitCode=1
	fi
done

# Output
echo "$out_pre$raid_status$out_post"

# Exit
exit $exitCode

hddetach

The hddetach script is a Bash script to safely detach (umount+sync+stop) an external hard disk on a GNU/Linux system.

The only argument $1 must be the label of a volume (partition) in the hard disk (e.g. MYDATA) which is mounted on the file system.
The script gets the device path (e.g. /dev/sdb) of the disk which holds the volume labeled $1 and then try to umount all the disk’s volumes, sync the internal cache and stop the disk and finally to removes it from the system (delete volume icon(s), file manager, desktop etc.).
Superuser rights are requested.

The hddetach script is free software, released under the terms of the GNU General Public License version 3 or (at your option) any later version.

Comments, suggestions and bug reports are welcome.

#!/bin/bash
#
# NAME
# 	hddetach
# 
# SYNOPSIS
# 	hddetach 
# 
# DESCRIPTION
# 	This script safely detach an external hard disk.
# 	The only argument $1 must be the label of one volume (partition) of 
# 	the hard disk (e.g. MYDATA) which is mounted on the file system.
# 	The script gets the device path (e.g. /dev/sdb) of the disk which holds 
# 	the volume labeled $1 and then try to umount all the disk's volumes, sync 
# 	the internal cache and stop the disk and finally to removes it from the 
# 	system (delete volume icon(s), file manager, desktop etc.).
# 	Superuser rights are requested.
# 	
# AUTHOR
# 	Alexus
# 	
# COPYRIGHT
# 	Copyright (c) 2014+ Alexus
# 	License: GPLv3+ (GNU GPL version 3 or later)
# 	This program is free software: you can redistribute it and/or modify
# 	it under the terms of the GNU General Public License as published by
# 	the Free Software Foundation, either version 3 of the License, or
# 	(at your option) any later version.
# 	This program is distributed in the hope that it will be useful,
# 	but WITHOUT ANY WARRANTY; without even the implied warranty of
# 	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# 	GNU General Public License for more details.
# 	You should have received a copy of the GNU General Public License
# 	along with this program.  If not, see http://www.gnu.org/licenses/.

# Function desc
desc() {
echo ""
echo "${0##*/}: safely detach an external hard disk"
echo "$COPY, version $VER $DATE"
echo ""
}

# Function usage
usage() {
echo "  Usage: ${0##*/} "
echo "  : label of one volume (partition) of the disk to detach"
press_key
exit 1 
}

# Function press_key
press_key() { echo ""; read -rsn 1 -p "Press any key to exit..."; echo ""; echo "";}

# Function umount_all
umount_all() { local dev="$DEVNAME"; mount | cut -d' ' -f1 | while read vol; do if [[ $vol =~ /dev/$dev. ]]; then umount "$vol"; fi; done; } 

# Intro
VER="1.0"
DATE="18/11/2014"
COPY="Copyright (c) 2014+ Alexus, GPLv3+"
desc

# Check argument
if [[ $# -eq 0 ]]; then echo "  *** Fatal error: Missing argument."; usage; fi

# Get disk device path and device name
LABEL=$1
PATTERN="\[$LABEL\]"
DEVPATH=$(mount -l | grep $PATTERN | cut -d' ' -f1)
DEVNAME=$(echo $DEVPATH | cut -d'/' -f3 | grep -Eo "[[:alpha:]]*")

# Check disk presence
if [[ ${#DEVNAME} -eq 0 ]]; then echo "  *** Fatal error: Not found any device with a volume labeled '$LABEL'."; usage; fi

# Ask user confirmation
echo "  Disk /dev/$DEVNAME holding a volume labeled '$LABEL' will be detached."
echo "  Superuser rights might be requested."
read -p "  Continue? (y/N) " -n 1 -r
if [[ ! "$REPLY" =~ ^[Yy]$ ]]; then echo "  Nothing done."; press_key; exit 0; fi
echo ""

# Unmount partitions on disk
echo "  Trying to umount all the volumes on $DEVPATH..."
umount_all

# Detaches (sync, stop, forget) disk and exit
echo "  Trying to sync, stop and forget $DEVNAME disk..."
sudo sdparm --command=sync /dev/$DEVNAME
sudo sdparm --command=stop /dev/$DEVNAME
sudo sh -c "echo 1 >/sys/block/$DEVNAME/device/delete"
echo "  Done."
press_key
exit 0

Back again

Hey!

Still alive and back again.
Next days I will try to publish some useful (?) Bash scripts.

GNU IceCat 14.0.1 prebuild binaries tarball

Image: GNU IceCat logo.

A tarball containing the prebuild binaries of GNU IceCat 14.0.1, compiled by me for a GNU/Linux i386 system based on Debian stable, is available at the GNU IceCat downloads page of this site.

Language files should be available at the Gnuzilla FTP site: ftp://ftp.gnu.org/gnu/gnuzilla/lang/.

Also see how to install GNU IceCat tarballs.

Enjoy!

GNU IceCat 13.0.1 prebuild binaries tarball

Image: GNU IceCat logo.

A tarball containing the prebuild binaries of GNU IceCat 13.0.1, compiled by me for a GNU/Linux i386 system based on Debian Squeeze, is available at the GNU IceCat downloads page of this site.

Language files should be available at the Gnuzilla FTP site: ftp://ftp.gnu.org/gnu/gnuzilla/lang/.

Also see how to install GNU IceCat tarballs.

Enjoy!

GNU IceCat 10.0 prebuild binaries tarball

Image: GNU IceCat logo.

A tarball containing the prebuild binaries of GNU IceCat 10.0, compiled by me for a GNU/Linux i386 system based on Debian Squeeze, is available at the GNU IceCat downloads page of this site.

Also see how to install GNU IceCat tarballs.

Enjoy!

How to install GNU IceCat tarballs

To install a generic prebuild binaries tarball package just unpack it in a folder (which is usually already defined decompressing the file itself) and copy or move it (also renaming it if necessary) in a suitable position, usually into your /home directory or in the /opt system directory. Might also be necessary to change the permissions of the directory and/or the contained files (see the instructions which usually come with the package).

I suggest this way to install the GNU IceCat prebuild binaries tarball packages provided by this website:

prepare the system

  • update your system (this is not strictly necessary, but it is better before removing or installing software):

    $ sudo apt-get update
    $ sudo apt-get upgrade
    
  • if you are updating a previous version already installed in /opt/icecat, then it is better to rename this folder before to install the new package:

    $ sudo mv /opt/icecat /opt/icecat-old
    

    so that you might eventually delete it later (be very careful using the rm command!):

    $ sudo rm -R /opt/icecat-old
    

download and install

(notice: below you should substitute x.y.z with the actual version number of the tarball package to install)

  • download the wanted GNU IceCat x.y.z tarball package from the GNU IceCat downloads page of this site
  • rename the downloaded package to fix the file extension (this is need as, due to bugs in WP, the file extensions of the tarballs have been changed from ‘.tar.gz’ to ‘tar_.gz’):

    $ mv icecat-x.y.z.tar_.gz icecat-x.y.z.tar.gz
    
  • unpack it (you will have a icecat-x.y.z folder):

    $ tar -jxvf icecat-x.y.z.tar.gz
    
  • move the icecat-x.y.z folder to the /opt system directory, renaming it at the same time to icecat:

    $ sudo mv icecat-x.y.z /opt/icecat
    
  • now you can finally run IceCat:

    $ /opt/icecat/icecat
    

Enjoy!

GNU IceCat 9.0.1 prebuild binaries tarball

Image: GNU IceCat logo.

A tarball containing the prebuild binaries of GNU IceCat 9.0.1, compiled by me for a GNU/Linux i386 system based on Debian Squeeze, is available at the GNU IceCat downloads page of this site.

Enjoy!

Update: Also see how to install GNU IceCat tarballs.