Tiltification

Requirements to follow along

This manual contains all steps necessary to build the Android app on a Windows 10 computer using Flutter in combination with Android Studio, as well as the steps necessary to replace our psychoacoustic sonification with your own sound design using Pure Data. Follow this instruction manual to participate in the ICAD2023 Sonit Tilt Competition. Besides the Pure Data sound design, no programming skills are necessary.

This means if you are working on Windows 10 or 11, you can follow step by step. Mac and Linux might have different namings for some options we mention. Besides the Git installation and parts of our troubleshooting however, the steps should essentially be the same on those systems.
We explain ONLY Android builds in this manual. Building for iOS devices is only possible using a Mac with XCode installed (XCode comes with Git, so we recommend to install it on Mac either way). You can of course check our repository's readme for some advice on iOS builds, but we can't give you further help here.

All in all the hardware requirements to continue are:

• a Windows 10 or 11 computer with an internet connection and 5GB of free space on your hard drive (a different OS supported by Android Studio and Flutter works as well, but keep in mind that some steps might need adjustments there)
• an Android smartphone (a tablet may work, too)
• a USB data cable to connect the computer and the smartphone (a "charge-only" cable will not work)

Git Installation

According to the official Flutter installation guide, Git is a system requirement to develop apps with Flutter. Besides that it is our choice for version control and collaborative programming and designing.
For Windows download git from here. During the installation process you can leave all options on default. But be sure the option "Git from the command line and also from 3rd-party software" is selected.

Project Download

Now that Git is installed, create a folder with a short path, such as C:\tiltification.
Here, you right-click and choose "Git Bash Here" to open a terminal window.



Inside the terminal window you type git clone https://github.com/Tiltification/sonic-tilt.git or paste it by right-clicking on the terminal window and selecting "paste". Hit the return key and the repository should download and become visible as a directory called sonic-tilt.

Flutter Download

We developed Tiltification using Flutter 1.22.0 from the 1st of October 2020. As newer versions may produce errors when compiling, we recommend downloading this dated version from their website.



Extract files to C:\tiltificationso that the tiltification folder contains the two sub-folders sonic-tilt and flutter_windows_1.22.0-stable.
This short file path will come in handy, as you will have to find and copy Flutter's file path later on.

Android Studio Installation and Setup

Android Studio comes along with all further development kits we need in order to build our app, and also provides us an IDE to work with. We have written this manual based on Android Studio version 2021.1.1 Patch 3, and also verified that Version 2021.3.1.1 is working.

Warning: The project is not running under Android Studio Electric Eel, Flamingo or Giraffe!

After double clicking the linked Android Studio installer, it takes some seconds to respond. After clicking "Next" on the welcome message, you will be asked to choose the components to install. Feel free to uncheck the "Android Virtual Device" to save some space, as you will be using your phone as a physical device to build on. Click "Next".



As the installation location for Android Studio, choose a folder named AndroidStudio (without a space character) in C:\tiltification so that the tiltification folder contains the three sub-folders AndroidStudio, sonic-tilt and flutter_windows_1.22.0-stable.
You can click through the next options. When clicking the final "Finish" button you can start Android Studio directly. Otherwise and in the future you can always start it from C:\tiltification\AndroidStudio\bin\studio64.exe.

You will need an internet connection upon the first start of Android Studio. In the first pop up window choose "do not import settings" and click "OK".



In the Android Studio Setup Wizard change the "Install Type" from "standard" to "custom" to be able to set your own paths for where to install things.



You will be asked to choose the Java Development Kit (JDK) location to use, which should be the one installed with Android Studio under C:\tiltification\AndroidStudio\jre.



After choosing the eye-friendly and energy-efficient "Darcula" theme you will get to the "SDK Components Setup". Here, you can leave all boxes checked as they are. What we are interested in is to set the "Android SDK Location" at the bottom of the window to a new folder named C:\tiltification\AndroidSDKand click "Next".



The last steps can be left to default. The installation wizard will finish by asking you to accept all licenses before you can click on "Finish".

Now that Android Studio starts, you will already find an option to install the plugins needed to work with Flutter.



If this window does not pop up, you can later in Android Studio also go to "File" -> "Settings..." and select "Plugins" from the list in the left column.
Under Plugins search the list for "Flutter", click the "Install" button next to it and accept the privacy note.
A pop up window informs you about "Dart", the next plugin to install. Click on "Install" inside that pop up window. The "Dart" plugin could otherwise also be installed over the same list like the "flutter" one.
After some installation progress, the button next to Flutter should change to "Restart IDE". Click it to finish the plugin installations.

Now that the plugins are installed you can click the folder icon "Open" to search your machine for our "sonic-tilt" project. It should exhibit an Android face icon , as Android Studio recognizes it as an Android Studio project. Open it, click "Trust Project" if asked, and give the project some time to load. If your firewall pops up, allow "adb.exe" to access the network.
Even though the interface is visible early, any inputs will be very laggy until the last progress bars at the bottom stops moving, so give Android Studio its time.

To tell Android Studio where to find the Flutter SDK, go to "File" -> "Settings..." found in the menu bar at the top of Android Studio's main window. Inside the Settings menu click on "Languages & Frameworks" where you can find "Dart" and "Flutter".



Click on "Flutter" and you will see the option to enter the path to your Flutter installation. Enter the path to where you unpacked Flutter into, which in our case is C:\tiltification\flutter_windows_1.22.0-stable\flutter (pay attention that the last folder of the path should be named "flutter" in all lowercase). Then click "OK".

Another crucial part to make your own Pure Data sonifications usable is to install the Android NDK, as we are using code that isn't supported by just the normal Android SDK. At the top under "Tools" you need to click on the "SDK Manager". A window pops up.



Here, you need to select the tab "SDK Tools". This should bring up a big list of tools available to install.
What we need are "NDK (Side by side)" and "CMake", so check the boxes next to them and click "OK" at the bottom right.

First Test Build

To be able to run a test build on your Android device, you first have to go into it's settings and enable the developer mode to make debugging on it possible. This is done differently depending on the Android version, and you may want to look for a detailed description on the official Android Website.
Most likely you need to go to "Settings" on your Android Phone and scroll all the way down to "About phone". Here, you scroll down to "Build number" and tap it 7 times. You might have to type in your password. This unlocks the developer options. Now go back to "System", tap on "Advanced", look for "USB debugging" and activate it. Sometimes, this option is hidden under "Developer options".
After connecting the phone via USB cable to your PC with Android Studio open, your phone now asks you to allow debugging. Allow it (you might have to allow it multiple times) and your phone should load as a device at the top of Android Studio near a green "play button".



If you can't see the drop down menu to select a device at all, check our trouble shooting further down this page.
If you can see it saying "<no devices>" and your device won't show up when refreshing or reconnecting the phone, Flutter might need to be informed explicitly where it can find the Android SDK. Go to the "Terminal" tab in Android Studio. It can be found on the bottom of the window in the same line as "Git", "TODO", "Problems", (sometimes "Messages") and "Dart Analysis".
This opens a terminal window right above the tab. Here, type C:\tiltification\flutter_windows_1.22.0-stable\flutter\bin\flutter.bat config --android-sdk C:\tiltification\AndroidSDK\. This tells Flutter where it can find the Android SDK. After pressing "Return", it should only take a second or so and then the text "You may need to restart any open editors for them to read new settings." should be displayed in the terminal window. Do so, exit Android Studio and open it again.
If everything went right you are now ready to click said green button next to your phone's name to create a test build of the app on your phone.
Especially on the first build you can expect it to load for several minutes. But unless the process stops, you do not have to worry about any warnings on the console. And in case it does unfortunately stop on an error, you should check our troubleshooting section further down this page, as it is likely a development kit not being found.

Pure Data

The sonification of our app is designed using Pure Data. Download Pure Data from their website, install it, and associate .pd files with it.
Your Pure Data version (like Vanilla or Extended) does not matter at this point as it is just an editor for you to work on pd patches and not part of the build process.
What matters is to know the correct paths to the files you want to edit with Pure Data. For Android builds this is C:\tiltification\sonic-tilt\android\app\src\main\res\raw, where all Pure Data files can be found in a ZIP file called streamingassets.zip.
Extract all files, which should produce a folder named streamingassets. Cut the ZIP file and paste it to C:\tiltification\streamingassets.zip as a backup.

In the extracted streamingassets folder (not the zip file) the shepardGuide.pd is your entry point. Open this file with Pure Data. You will see several inlets, some wild routing and subpatches, and the [dac~] output.



Delete the wild routing and subpatches and replace it with your own sonification. The parts to delete are marked by a red rectangle and a comment on the right will give you some hints on how to start implementing some sound designs yourself.
The inlet labeled sound is the volume control that can take values from 0 to 1.
The inlet pink is either 0 or 1. It tells you whether the smartphone is tilted by more than 3° or not. Tiltification uses this information to (un-)mute pink noise as a confirmation to the user that the mobile phone is almost leveled.
The inlet tarX is the tilt angle along the left-right dimension. Here, angles from -45° to 45° are linearly scaled to values from -0.5 to 0.5, telling you by how much the smartphone is tilted to the left or right. Angles that have a larger absolute value than 45° will still be transferred to tarX as -0.5 or 0.5, respectively.
The inlet tarY is the tilt angle along the front-back dimension, again, with angles between -45° and -45° scaled to values between -0.5 and 0.5. It tells you by how much the smartphone is tilted towards or away from you. Again, larger tilt angles will still send a -0.5 or 0.5 to tarY.
These are all inputs that your sonification needs. The output of your sonification has to be routed to both inlets of [dac~].
Even though you can use stereo, we recommend to stick to mono, i.e., route the same signal to the left and the right inlet of [dac~]. Furthermore, all important auditory information should be contained in the frequency region between 200Hz and 4kHz. This is because many smartphones only have a mono tweeter, and most users will certainly not use headphones while leveling furniture or alike. Consequently, stereo information and frequencies outside the bandwidth of the tweeter will be lost.
The objective of the user will be to reach a value of 0 for tarX and tarY, and your responsibility as a sonification designer is to guide the user there. If you use any externals, please copy the respective PD files into the streamingassets folder.
To test your sonification, open receiverLibPD.pd. This patch opens the shepardGuide.pd as a subpatch. In receiverLibPD.pd you can manually manipulate tarX using the green, horizontal slider, tarY using the blue, vertical slider, pink (un-)checking the toggle box, and volume using the white, vertical slider.



To test your sonification inside the Sonic Tilt app on your mobile phone, use the Windows explorer, navigate to C:\tiltification\sonic-tilt\android\app\src\main\res\raw\streamingassets, right-click and choose "New" -> "Compressed (zipped) folder" that you name streamingassets.zip. Then, copy all files from the edited streamingassets folder to the ZIP file. Place said ZIP in C:\tiltification\sonic-tilt\android\app\src\main\res\raw. This way of zipping files seems unnecessarily complicated. But if you simply zip the complete folder directly, the ZIP file will not contain all files directly, but within a subfolder, in which case the file structure is wrong and the built app will stay silent.
With your phone connected via USB (and USB debugging enabled) it is now just another click on the green "play button" in Android Studio to build the app with your new sonification. App builds may take a minute. After the successful build, the Sonic Tilt app opens automatically on your Android phone.

App Build

While you develop your own bullseye spirit level sonification in Sonic Tilt, you will make changes in the Pure Data files over and over, create the streamingassets.zip file, run the app on your smartphone and explore how the sonification responds to your actions. Note that in the Sonic Tilt menu there is an option to use the spirit level only in one dimension. This option will set tarY to 0 and the only variables controlled by the smartphone sensors are tarY, pink and mute.
When you are happy with the sound inside the app, and its response to the smartphone orientation and motion, you should build a distributable APK. To do so, navigate to "Build" -> "Flutter" -> "Build APK". The compilation may take a minute. You can find the APK file under C:\tiltification\sonic-tilt\build\app\outputs\flutter-apk\app-release.apk. Note that the app you create is yours. You can use it and even publish it, as long as you do this under an appropriate license, like the MIT license, and mention that you created this app using Flutter and Sonic Tilt. Please refer to our open source Git project and our Tiltification paper wherever you present your Sonic-Tilt app. Note that you are also free to modify other parts of Sonic Tilt, such as the Graphical User Interface (GUI), the app icon, you are free to include menu sounds or a screen reader, apply additional signal processing on the sensors, include the compass as a third dimension, develop a sensor calibration routine or whatever you like.

For example, you can replace the icons in the path C:\tiltification\sonic-tilt\android\app\src\main\res and the respective subfolders from mipmap-hdpi to mipmap-xxxhdpi. Make sure that the icons have the correct resolution (from 72x72 to 192x192 pixels).

Now it is time for you to prepare a documentation of your sonification using our downloadable LaTeX-Template or the online Overleaf Template and submit via Easy Chair for the sonification design competition at ICAD 2023. An exemplary documentation of my "Mixing Levels" version of Sonic Tilt can be found on ArXiv. Before you hand them in, please make sure that the APK file contains your latest changes: Uninstall Sonic Tilt from your smartphone and use the Windows Explorer to copy the APK file from your computer to your smartphone. On your smartphone, open the APK file to install the app. If prompted, toggle "Allow from this source". You may need to allow APK installs in your Android settings first. Depending on your Android version this can be done under "Settings" -> "Apps" -> "..." -> "Special access" -> "Install unknown apps" or under "Settings" -> "Apps & notifications" -> "Advanced" -> "Special app access" -> "Install unknown apps".

Troubleshooting

Even when you strictly follow our instructions, issues can occur. Here are some common problems and potential solutions:

1. Problems with Pure Data:
   Sometimes it looks as if the ShepardGuide.pd file was empty. In this case, make sure to scroll to the upper left corner of the Pure Data window using the two scroll bars.
2. Unable to Build:
   Sometimes, the Flutter device selection menu does not show in the Android Studio IDE at all.
   
   
   
   To resolve the problem, go to "File" -> "Settings" -> "Languages & Frameworks" -> "Dart". Here, check the box "Enable Dart support for the project ’sonic-tilt’" at the top and the boxes "Project ‘sonic-tilt’", "sonic-tilt" and "sonic-tilt.Pd.Core" at the bottom. After pressing "OK" the device selection menu should appear.
   
   
   
   
3. Failing to Build:
   
   • Environment Variables:
     Most errors in the build process occur, because Android Studio doesn't find the correct paths to it's development kits. One can try to rename some directories as a solution, but the most robust solution for all the development kits is to set an "environment variable" for them in the operating system.
     The fastest way to do this is to use the Windows shortcut [Win] + [R], type in sysdm.cpl and click "OK" to get to the System Properties window. Here, select the tab "Advanced". In that Tab click on "Environment Variables..." in the bottom right corner. Here you can click on "New..." either just for your account or for all accounts on your machine. Environment variables need a name like JAVA_HOME and a value to point to like C:\tiltification\AndroidStudio\jre. Click "OK" to save the environment variable.
     
     
     
     With environment variables like this, programs like Android Studio can easily find what they are looking for. Make sure to restart Andoid Studio after setting them, for it to register the newly set environment variables. In case environment variable are a possible solution, the error messages down in the "Run"-console of Android Studio will mention their names.
     Below are examples of environment variables often needed for Android Studio to find everything important:
     1. ANDROID_NDK_HOME is the environment variable name to set the NDK location with. This value should become C:\tiltification\AndroidSDK\ndk\some.version.number depending on the ndk version.
     2. JAVA_HOME is the environment variable name to set the JDK location with. This value should be set to the Java Development Kit that came with Android Studio. According to our naming conventions that value would be C:\tiltification\AndroidStudio\jre.
   • Flutter and Dart:
     Make sure you downloaded Flutter 1.22.0. We have also tested Flutter 1.22.4 successfully. But other versions may not work, especially Flutter 2 and newer versions. Make sure you enabled the plugins for Flutter and Dart in Android Studio under "File" -> "Settings" -> "Plugins". Make sure you entered C:\tiltification\AndroidSDK as the Android SDK location under "Tools" -> "SDK Manager".
   • streamingassets:
     Flutter does not like unexpected files inside the project. Make sure to create the streamingassets.zip file in C:\tiltification\sonic-tilt\android\app\src\main\res\raw and remove any other zip files from that directory before you run the app.
   • Error: Entry point isn't within the current project: This error occurs when you've installed an incompatible Android Studio Version, like Electric EEl, Flamingo or Giraffe. Please try again using Android Studio version 2021.1.1 Patch 3, or Version 2021.3.1.1, i.e., Chipmunk or Dolphin.
4. No Audio in App:
   Naturally, you should ensure that your phone is not muted. Keep the "volume up" button on the side of your smartphone pressed for several seconds to maximize the smartphone volume. Apart from that we identified four common reasons for the Sonic Tilt app to stay silent:
   1. The Mute Button: On default the built Sonic Tilt app will be set to mute and you will have to tap the speaker icon at the bottom left of the app to unmute it. If unmuted, the speaker symbol is not crossed out anymore.
   2. streamingassets.zip: Firstly, it can easily happen that your latest changes of shepardGuide.pd are only contained in the streamingassets folder and not yet in the ZIP file. Make sure to create a new streamingassets.zip file every time you want to test a new build.
      Secondly, it is easy to mess up the folder structure of the streamingassets.zip file. Make sure the Pure Data files are directly inside that zip and not in another folder beneath it or they won't be found.
   3. Incompatible Device: Download and run Tiltification from the Google Play Store or from this website. If the sound doesn't work either, Sonic Tilt may be incompatible with your device. In this case you need to find another smartphone or tablet to test your app.
   4. Erroneous shepardGuide.pd: Maybe your Pure Data file(s) contain an error. Try replacing your sonification by a simple pure tone that is constantly playing by putting an object [osc~ 440] in your file and connecting its outlet to the [dac~] inlets, save the file, zip all files and compile a new test build. If this works, try to debug your Pure Data files.
5. Audio Artefacts in the App:
   Audio artefacts can have several causes.
   
   • Clipping: It is important to know that the audio output in Pure Data only allows values from -1 to 1. This is true for the sum of all inputs to the [dac~]. All values outside this range will cause clipping, which may sound as distortion, crackling, clicks or noise. Try connecting every audio to an object [*~ 0.01], route that to the [dac~] and listen carefully, if the artefacts are gone.
   • Unknown Pure Data Externals: Sonic Tilt uses libpd to use Pure Data as a sound engine on Android. It is possible that libpd does not know all the externals that you installed on your computer. To be on the safe side, install Pure Data without any additional externals. Every object that does not work with the native install of Pure Data should be added to the streamingassets.zip to make sure that "iemlib" can call it.
     For example, to use the [pink~] object from the "iemlib" external, you should either copy the pink~.pd file to streamingassets.zip, or you create your own pink~.pd file that carries out all the expected operations or signal processing.
   • Computational Demands: Smartphones tend to have a quite high computational power. But if your Pure Data files contain a lot of real-time signal processing, like convolutions, Fourier transforms, and dozens of oscillators, this may be too much for some smartphones, and they fail processing it in real time, which leads to audible artefacts. In this case try to reduce the signal processing demands of your sonification.
   • Errors in Pure Data Signal Chain: Signal processing in Pure Data can produce many artefacts. Sometimes, no artefacts occur when you change values using sliders, especially because you slowly modify one value after another. But then, when you use the smartphone sensors, several values may alter at once, and the alterations may be much faster and contain more jitter as compared to using a slider. Use the app in one-dimensional mode to see whether the artefact still occurs. If not, tarY may cause the artefact. Try tilting the smartphone slowly to check whether fast motions may cause the artefact. Tilt the smartphone near 3° to hear whether the [pink] object causes the artefact. Use the app with headphones and listen carefully. If the artefacts are gone, low frequencies may cause them. These frequencies are practically inaudible when played via the internal smartphone speakers. Nonetheless, they deflect the speaker membrane and, together with the audible sounds, may produce artefacts. In the case of low frequency artefacts, either try to eliminate the cause of low frequencies in your signal processing, or add a high pass filter to the signal chain.
     Note, for example, that the [osc~] object in Pure Data is a cosine function that can produce an audible click when switched on and/or off. This may also be true for other oscillators. Here, a ramp at the onset and offset may help.

Citation

When you use Sonic Tilt for your own project, please cite our paper:
Asendorf, M. et al., “Tiltification: An accessible app to popularize sonification”, in: 26th International Conference on Auditory Display (ICAD 2021), 2021, https://doi.org/10.21785/icad2021.025