Acropolis Family Firmware Overview
The Acropolis family firmware consists of several open-source libraries managed by purpose-coded classes and libraries that allow for a consistent interface when programming firmware for novel artefacts. The Acropolis family classes can be categorised by their high-level function where there are utility, audio playback, audio analysis, environmental sensing, and visual feedback classes. When compiling code for an Acropolis species, the code leverages preprocessor macros and directives to dynamically compile the code according to the hardware configuration and artistic use-case of the artefact.
To control the hardware components presented in the previous chapter, and to expedite the creation of new Acropolis artefacts a shared "Acropolis code base" was created. The firmware provides support for common artefact behaviour functionality while accounting for the unique environmental challenges Acropolis devices must withstand when exhibited in outdoor environments. This discussion introduces what exactly the Acropolis codebase is and how its object-orientated design paradigm helps to facilitate a consistent coding interface while developing novel Hyper-Soundwalk Series installation scenarios. The structure of the Acropolis codebase is introduced before an overview of the dynamic compilation process.
Dynamic Compilation
It is usually preferable for artefact's to include physical mechanisms to control vital run-time behaviour and artefact settings. However, as not every setting can be guaranteed a physical user control due to size and budget constraints, a code interface is needed to ensure important settings can safely and easily be modified quickly and safely. For Acropolis artefacts, a group of human-readable configuration files satisfies this need.
Within this group of configuration files, Configuration.h holds the most commonly adjusted parameters and serves as the primary user-facing configuration file. There are several additional files which contain parameters that pertain to specific operating modes or hardware configurations such as pin numbers for different PCB revisions, and audio-engine routing but these files are minimally edited for artefacts which are not in active development.
Audio Classes
As seen in the above Figure, user configured settings within the configuration files determines what classes the AudioEngine.h file imports into the firmware. While as depicted by the dotted black lines, the inclusion of the ZCrossings, ClipCounter, AutoGain, and OnsetDetector classes is dependent of the artefact's artistic use-case, the audio system always imports an instance of the open source Audio library for the Teensy 3.2, and the purpose coded FFTManager class to provide frequency-domain audio features, and the purpose coded FeatureCollector class to manage time-domain audio features. Along using time and frequency domain audio features to calculate novelty functions that drive artefact behaviour, one of the primary responsibilities of the Acropolis audio system is dynamically creating and routing Audio library objects and create the appropriate connections to support the artefact's audio processing, analysis, and playback requirements.
FeatureCollector
Once artefact prototyping began in late 2019, the need for a centralised interface which can manage the collection and tracking of audio features became urgent as individual unit firmware began to diverge and managing a code-base for each artefact became difficult to manage. To address this need, The HSS FeatureCollector class was created --- drawing inspiration from a class with the same name with the ChucK programming language. The FeatureCollector collects raw data from several sources to calculate high-level time-domain audio features (such as spectral flux, centroid, and rolloff) which can be leveraged by artefact-specific code to drive unit behaviour. All frequency-domain feature extraction and analysis functionality is included in the FFTManager. The FTManager class together with the FeatureCollecter class together provide audio feature values scaled from 0.0 to 1.0 through a standardised process of generating feature data.
During setup(), FeatureCollector is passed references to all active time domain AudioAnalyzeRMS, AudioAnalyze-Peak, and AudioAmplifier objects. AudioAnalyzePeak and AudioAnalyzeRMS objects are linked to the FeatureCollector via a pointer passed to the class during either setup() or main(). FeatureCollector.update() should be called within main() to ensure the firmware polls Audio library objects for the most up to date analysis data. If new data is available, the FeatureCollector.update() updates the RMS and Peak ValueTrackers.
FFTManager
The FFTManager operates on the same principles as the FeatureCollector but is concerned with the calculation of frequency-domain, instead of time-domain, audio features. The FFTManager acts as a wrapper class for the AudioAnalyzeFFT1024 object, which as the name implies, calculates a FFT with 1024-bins using a linear scale to produce a frequency resolution of approximately 43 Hz. Within setup(), any present AudioAnalyzeFFT1024 Teensy Audio library objects should be linked to the FFTManager.
Within the artefact's main() loop, the FFTManager.update() function is called to update the features contained within the FFTManager class. If a new analysis is not available from the FFT object, or the FFT object is inactive, the function immediately exits while returning false to indicate that no update took place. If the adaptive whitening option is active, the raw FFT values will be scaled according to each bins runtime maximum. A combination of the current and past FFT values are used to calculate the active frequency-domain features to be utilised by the visual feedback system.
AutoGain
Human biology allows us to adjust the sensitivity of our ears so to extend our hearings dynamic range. This ability automatically increases the sensitivity of our hearing in quiet environments and decrease the sensitivity in loud environments. From the early stages of the iterative process, it became apparent that a similar mechanism is desired for Acropolis artefacts to help them to adjust to the varying sonic environments they are installed within. Automatic gain control is a common feature within consumer electronics such as smart speakers and conference call devices. Accordingly, there is a wide variety of computational and hardware approaches which can be used to provide this feature, but as the comparative needs for the Acropolis gain-calibration is minimal, there is no need for an overly complicated approach.
When the AutoGain class is instantiated it is passed a pointer to a FeatureCollector object and, depending on if frequency domain features will be included in the calculations, optionally a FFTManager. AutoGain conducts gain adjustments through the linked FeatureCollector class. Three additional variables are passed into the instantiation function, a minimum gain value, a maximum gain value, and a maximum gain adjustment factor. The AutoGain object can be used either actively, or passively, depending on the use-case.
If the AutoGain functionality is intended to be used passively, within setup() the setUpdateRate() function must be called to establish the autogain routines update rate. Furthermore, trackAvgRMS(), trackOnRatio(), trackClips() or any other desired feature tracking function must be called to establish AutoGain's criteria. Each of the ``tracking'' functions require four variables. A tolerance threshold around the target value, and the minimum and maxiumum expected values for the tracked feature. Once these functions have been called within the setup() loop, the AutoGain class is updated by calling the update() function within each run through the main() loop.
When used actively, the forceCalibration() function is used to force a run of the auto-calibration routine. As seen in the class flowchart, the forceCalibration() function is typically run within the firmware setup() loop to establish working initial gain levels for artefact microphones but can also be periodically called within main() if undesirable feedback conditions are observed. For instance, with the Speculātor units, if the LED on/off ratio falls below around 5% (meaning the units are providing no visual feedback 95% of the time) the auto-gain routine is triggered to establish more productive microphone gain levels.
Early within the firmware base's development cycle, the auto-gain routines were often overused or implemented with overly aggressive settings. At the time, it was assumed that the devices would need to continually adjust the sensitivity of their microphones to optimise the expressive capability of their soundscape augmentations. However, as Acropolis devices are both concerned with periods of relative quiet and periods of relative loudness, through exploratory testing it was discovered that for most Acropolis installations, unit soundscape augmentations should embrace these amplitude variations and not try to compensate for them. Overly aggressive auto-gain routines can undermine the long term effectiveness of the soundscape augmentations. Alternatively, an approach which seeds gain stages within setup() but is supplemented by user control overrides to adjust gain sensitivity during runtime, is often the most flexible and safest configuration.
Clipping Counter
A useful time-domain metric which the FeatureCollector can optionally calculate for artefacts is the number of clipped audio frames which occur over a period of time. This information can be used by AutoGain objects to help establish healthy gain levels for microphones as well as provide useful diagnostic information which can be used for unit evaluation. The ClipCounter class is a child class of the general purpose AudioStream class provided by the Teensy Audio library. Instead of being strictly contained to the FeatureCollector class, ClipCounter can be used in the same manner as any other AudioStream object. Any other AudioStream object with an active output can be connected to the ClipCounter using an AudioConnection objects. Once connected, ClipCounter will automatically call its update() function along with the other AudioSteam objects --- incrementing an internal counter whenever a clipping audio sample is detected.
The clipping threshold can be changed from the default of 32767 to any value by using the setThreshold() function (causing near clipping events to be registered by the class). The clipping counter can be reset to 0 by calling resetCounter() while the number of clips registered since the last reset can be retrieved using getNumClips(). The class also provides deactivate() and activate() functions which can be called before and after artefact sonic actuations to prevent unit actuations from triggering clip events.
Auxiliary Classes
Cochlear Feedback Classes
Rhythm
The playback information such as note on times, note lengths, and rhythmic relationships to each other are stored within the Rhythm class instances, of which one is generated for each newly transcribed rhythm. In addition, multiple Rhythm instances are organised within RhythmBank classes which allow for expanded rhythm selection options.
Rhythm() instances are loaded with notes one at a time using one of four functions depending on the type of note added: addPitchedNote(), addUnpitchedNote(), addDampenedNote(), and addMotorMove(). The Rhythm class is dynamic allowing sonic events to be added, removed, replaced, or modified at any time.
The information provided for each sonic event added depends on the sonic event type as, for instance, an "unpitched note" does not possess frequency information while a "pitched note" does. Within the Rhythm class, the necessary information needed to recreate the sonic event are contained. This involves the note type of which the options are pitched, dampened, percussive, and motor. The frequency of the event, the length of the event, and the velocity (scaled from 0.0 to 1.0) of the event are also stored. If multiple pitched events are stored, a medium_frequency for the Rhythm will be calculated to aid in rhythm selection for some Explorators. The onset_time array contains the start time for each sonic event contained within the Rhythm object and is how the notes are temporally organised. The onset_time is calculated concerning the start of the rhythm in question and is expressed using milliseconds. If the onset_time is set at 3000 for a sonic event when the rhythm is played by a PlaybackEngine that sonic event will start three seconds after the rhythm playback begins.
In addition to containing the information for individual sonic events, the Rhythm class aids the PlaybackEngine by keeping track of whether it is currently being played through the active boolean. If the rhythm is currently in playback, it will update the current_note variable to reflect the next note which will be played. By keeping track of the current playback note, the Rhythm class can provide a simplified interface
Visual Feedback Classes
NeoGroup
Several custom classes have been written to manage Speculātor's visual feedback system. The NeoGroup class in Speculātor's firmware is a wrapper class for the non-blocking WS2812Serial NeoPixel control library~\cite{stoffregenPaulStoffregenWS2812Serial2020. NeoGroup handles the logic for setting the final brightness and colour of each \gls{led element according to real-time sonic and environmental conditions. Additionally, NeoGroup tracks diagnostic data concerning the feedback system's runtime operations for affecting visual feedback as well as datalogging and troubleshooting purposes.
The NeopixelManager is a class that contains one or more instances of NeoGroup, performs data collection, and coordinates LED operation with the other firmware classes when needed. Once a NeopixelManager object is created, settings such as the colour mapping strategy, the onset feedback colour, and the brightness scalers can be configured using the class' setting functions. Within main(), the NeoGroup is primarily controlled through the use of the colorWipe() function. The colorWipe() function is a high-level interface to the NeoPixel manager that provides control over the display brightness and colour of LED groups or individual LEDs.
Environmental Sensing Classes
During runtime, the WeatherManager class is updated within the main() loop using the update(). As the update() function blocks updates to the visual feedback system and consistently takes 16 ms to complete, the WeatherManager.update() polling rate is set to twice a minute. When the current, average, maximum, or minimum recorded temperature or humidity values are needed by a use-case specific portion of the firmware (primarally for the visual feedback system), they are accessed through WeatherManager's getter functions. If the temperature or humidity values are higher than the threshold set during class instantiation, the artefact enters into a sleep-like hibernation state.
LuxManager
The inclusion of lux sensors in Speculātor's first artefact specific design consideration: adaptability to varied and unpredictable lighting conditions. The lux sensors allow Speculātor to provide more perceptually consistent feedback while maximising run-time by conserving energy by exhibiting behaviour such as dimming its feedback brightness when clouds pass overhead.
First the LuxManager wrapper class is initialised with a minimum and maximum polling time. As polling the ambient light sensors require the visual feedback system to deactivate, resulting in a perceivable "glitch" in the visual feedback the sensors should not be polled too frequently. The minimum polling time value passed into the LuxManager class addresses this concern by preventing Speculātor from polling the sensors too often. Alternatively, the maximum polling time value determines the duration allowed between polling events before a forced-shutdown event is triggered to collect new readings. To minimise the period in which the visual feedback system remains inactive in order for new lux readings to be collected the ambient light sensors are configured to utilise the shortest available integration time of 25 ms.
After these initialisation routines are complete and at the end of the main() loop, the LuxManager.calibrate() function is called. LuxManager.calibrate() first attempts to gather readings from the initialised sensors. Then the function reads each sensor ten times with 3 seconds between each reading, to calculate an initial global lux value. If the sensor returns an error, is returning chaotic values, or is returning unexpected readings (0.0 or extremely high values) it is flagged as inactive and polling ceases for that sensor for the remainder of runtime and the other sensor's readings are used exclusively. After the initialisation function tests the sensors and calculates an initial global lux value, pointers to the NeoPixelManagers are passed to the LuxManager object so brightness scalers can be updated within the NeoPixelManager objects directly.
For the final v3 iteration of Speculātor, the higher lux value is used to determine brightness scalers while the lower value is ignored. During initial testing this was not the case. For the v0 iteration, the two lux sensors provided independent brightness scalers for each side of the artefact. When this strategy was tested in-situ, movement exhibited by artefacts required the brightness scalers to be updated several times a second to keep up with unit rotation.
After determining that independent scalers for each side of the artefact served inadequate, averaging the readings was tested as an alternative approach. In practice, this strategy usually provided more useful brightness scalers than independent operation. However, under circumstances when one side of the artefact is shaded while the other side is exposed to direct sunlight the averaged lux values produce brightness scalers which produce overly dim feedback which can not be seen on one side. As the visibility of Speculātor's feedback is more important than the artefact's run-time, a third approach was conceived. The third approach dismissed the lower lux reading and calculates brightness scalers based solely on the higher value alone. During in-situ testing, this approach avoided overly dim feedback. While there were occasions when the artefacts were brighter than they needed to be, and thus were consuming too much energy, the consistently visible feedback this mapping provides outweighs the power-efficiency drawbacks. While all three options are available within the unified firmware, the Speculātor firmware utilises the third approach by default within the Speculātor branch.
WeatherManager
During runtime, the WeatherManager class is updated within the main() loop using the update() function. As the update() function blocks updates to the visual feedback system and consistently takes 16~ms to complete, the WeatherManager.update() polling rate is set to twice a minute. When the current, average, maximum, or minimum recorded temperature or humidity values are needed by a use-case specific portion of the firmware (primarally for the visual feedback system), they are accessed through WeatherManager's getter functions. If the temperature or humidity values are higher than the threshold set during class instantiation, the artefact enters into a sleep-like hibernation state.
OrientationManager
When engaged in soundscape augmentation, Legatus periodically polls the gyroscope and accelerator sensors. Once the new sensor data is collected, the readings are loaded into a two dimensional FIFO buffer. The current and previous sensor readings are compared to determine if any significant change occurred to the artefact's orientation as determined by thresholds within firmware configuration files. If a significant change in artefact orientation is detected, usually triggered by an angular shift of at least forty-five degrees, the artefact triggers an emergency state. This firmware state is intended to discourage theft while alerting the installation facilitator to the artefact's peril. Until rebooted, Legatus plays a loud and shrill alarm while flashing its lights red for several seconds before resting for thirty seconds and repeating. Alternatively, low magnitude readings that occur when the artefact is exposed to wind and rainfall in some test installations have been mapped to the artefact's random number generation routines to influence vocalisation times and qualities. Alternately, low magnitude readings have been mapped to the colour and brightness of the visual feedback system for some tests. With white light occurring when the artefact experiences minor shaking and red when significant events occur.