Chroma Temporal Surveilance Bot

This video provides an example of some experimentations I conducted with creating C++-based slit scan processing effects. This particular example shows the use of the modulo mathmatic operator to repeate the temporal effects across the images height and width.

Chroma Temporal Surveilance Bot

While technically a hardware and software project, the Chroma Temporal Surveilance Bot project is included on this page as the project served as a showcase for the slitscan-based programs I developed during the final year of my MFA studies. Exhibted in the 2017 Digital Arts Expo at the California Institute of the Arts, the Chroma Temporal Surveilance Bot was leveraged as the participant interface for the installation "6 * 9 = 42".

The Chroma-Temporal Surveillance Bot (CTSB) consists of a simple hardware interface consisting of only two controls housed in a small metal box connected to a iMac computer running a few hundred lines of Processing27 code running on an iMac computer. CTSB serves as the functional portion of the installation 6 * 9 = 42.

While there is simple firmware running on an Arduino Uno inside of the CTSB hardware interface, it only serves to report button presses and switch positions to the computational core of the system which is written in Processing and is running on an iMac host computer. Besides using the state of the hardware interface to determine its internal state as expanded upon below, CTSB gains control over the iMac’s webcam which it turns on. The images captured from the webcam provide the source material for the remainder of the program.

Although the true purpose of the CTSB is to run its political profiling subroutine, this can only be done when someone is interacting with the CTSB hardware. Thus, the first thing CTSB does after initializing the webcam is to create some interesting “art” with the hope of drawing a crowd of users it can proceed to tweet about. This is done by means of a specialized type of slit-scan video processing algorithm which operates on the red, green, and blue color channels with separate images masks providing color independent slit-scan processing.

Upon initialization, the system creates three contrasting greyscale “master color masks” - one for each of the primary colors. Next, the program begins storing the 256 most recently captured images from the host computer’s webcam in a FIFO28 buffer. After updating the FIFO CTSB calculates the color information for each pixel it will display on the iMac screen one at a time. The program references the value stored in each of the three master image mask’s corresponding pixel which is used to look up a corresponding image stored in the FIFO buffer. For instance, if the first pixel returns 0, 50, and 200 from the three master color masks it will take its red value from the first frame in the FIFO buffer, its green value from the 51st frame, and its blue value from the 201st frame. This process is repeated for every single pixel of each image displayed on the iMac.

The switch and button give the user control over the master color masks. When the switch is turned to a new position, the program chooses a different algorithm for drawing the master color masks and redraws the masks, resulting in a different aesthetic quality. When the pushbutton is pressed, the program reconstructs the masks using the rule set dictated by the rotary switch creating a different variation of the currently selected aesthetic.

While the video produced by the CTSB is entertaining its enjoyment comes at a cost to the users privacy and possible reputation. The hidden purpose of the CTSB is not to create beautifully psychedelic videos, but instead to gather data about the political affiliation of the gallery visitors. Ten percent of the time, chosen randomly, the pushbutton is pressed the CTSB executes its surveillance subroutine. First, the program analyzes the color content of the frame it is about to display on screen. It adds up all of the red and blue values for all of the pixels and determines if the image overall contains more red or blue. Next, according to the results of this “chroma analysis”, the CTSB determines the interactee is either Republican (if there is more red) or Democrat (if there is more blue). Although the system for determining the users political affiliation is obviously naive and erroneous, the CTSB proclaims the results of its findings along with its evidence (the image) on Twitter in the form of a tweet announcing the users allegiance to the affiliated political party.

The CTSB has gone through two major hardware revisions. The first version of the interface comprised of six N.O. push-buttons on the top of a BB aluminum project enclosure box. Inside the enclosure, an Arduino Nano recognizes button presses and reports the events to a Processing sketch running on a host computer. The Processing program handles communication with the hardware, reading images from the computers webcam, computing the slit-scan processing, and displaying the video output. Each of the buttons on the controller gave the user an axis of control over the slit-scan processing algorithm. This includes: changing the buffer length; reconstructing a mask; cycling through the mask categories; toggling between red, green, and blue channel separation; inverting the color processing; tweeting a photo; and cycling through different frame rates. This multitude of control results in millions of unique optional program states. After some testing, it became apparent that the interfaces large number of controls distracted attention away from the software system which the user is intended to focus on. As interactees spent more time figuring out what each of the button do than playing with the system they controlled, the need for a simplified interface motivated the design and construction of the second version of the CTSB.

The second, and final, version of the interface features only two controls instead of the six found on the prototype. The first control is a button which redraws the color masks when pressed. The second control is a twelve-position, continuous rotary switch which allows users to choose between the different algorithms used to build the color masks. The simplified design provided users a more consistent experience in a curated, yet dynamic, manner. This was done partly by removing many of the axis of control once available to the user. For instance, by removing user control over the oversampling rate of the FIFO buffer, which is interesting with some effects but can destroy others if set too low or high, and hard coding its value for each of the program states reduces the chance the user enters into a “dead state” where nothing interesting is happening. This additionally reduces confusion about each controls’ effect on the displayed image, while still inviting exploration and allowing for countless variations through the generation of new masks. As the interface needs to attract participants to collect its data, it is important that the device appears friendly, innocent, and easy to use. The hardware interface is straightforward, familiar, and easy to understand to hide the software’s more sinister purpose.

Chroma Temporal Surveilance Bot

Chroma Temporal Surveilance Bot

Related Works and Further Reading