Audio visualization in JavaFX - AudioSpectrumListener

Hello everybody!

In this article I would like to show you how to visualize audio in JavaFX without additional frameworks or libraries, for example, to create a visualizer for an mp3 player. [go to german version]

ATTENTION!
I assume that it is already known how to play audio files over the MediaPlayer in JavaFX. Otherwise, simply search on Google ;). Also, this does not work on a raspberry because the media API of JavaFX does not work there!

Okay, let's go!

Where can I get the data?

To visualize audio, we need data to display it. For this, JavaFX has the so-called AudioSpectrumListener. This can be assigned to a MediaPlayer object using the setAudioSpectrumListener() method.

 mediaPlayer.setAudioSpectrumListener(new SpektrumListener());  
   
 private class SpektrumListener implements AudioSpectrumListener {  
     @Override  
     public void spectrumDataUpdate(double timestamp, double duration,   
 float[] magnitudes, float[] phases) {}
 }  

In this case, mediaPlayer is a MediaPlayer object and to improve the overview, I use an internal class instead of an anonymous.

As you can see, the AudioSpectrumListener contains a spectrumDataUpdate() method. This gives us the time stamp, the period over which the data was collected, the amplitude and the phase every 100ms (standard value). Note that this is done for each frequency band (default value: 128), which are evenly distributed by JavaFX and therefore we get an array of amplitudes and phases.

But now the question is, what can we do with these values? Well, I am unfortunately not a musician, so I personally only use the amplitudes to visualize audio. Unfortunately, in most cases the obtained values can not be used easily since these values are less than or equal to 0 (decibels). For all who now think: "Okay, then I simply take these values *(-1)." This is not the case. Frequency bands which are currently playing no tones have no value of 0, but of -60! This means that if you simply convert these to positive values by *(-1), we get a value of 60, although no sound is played at all! A musician could now explain exactly why this is so, but since I have no idea in this area, we leave it first. But the more important question is: Where does the -60 come from?

The -60 is the default value for the threshold in JavaFX, which is the limit from when JavaFX limits the values. Of course, we can change this at any time using the setAudioSpectrumThreshold() method. In this case, it should be remembered that no tones have this changed value, so if the new threshold value is -100, then no tones also have a value of -100! We can now use this to get correct positive values. To do this, we simply have to subtract the threshold value from the values obtained, so -60 becomes 0 for no tones. The following source code shows this again on the basis of the first frequency band:

 mediaPlayer.setAudioSpectrumListener(new SpektrumListener());  
   
 private class SpektrumListener implements AudioSpectrumListener {  
     @Override  
     public void spectrumDataUpdate(double timestamp, double duration,   
 float[] magnitudes, float[] phases) {  
   
     correctedMagnitude[0] = magnitudes[0] - mediaPlayer.getSpektrumThreshold();  
 }  

However, since we want to have not only the value of the first frequency band but of all, we still use a for loop:

 mediaPlayer.setAudioSpectrumListener(new SpektrumListener());  
   
 private class SpektrumListener implements AudioSpectrumListener {  
     @Override  
     public void spectrumDataUpdate(double timestamp, double duration,   
 float[] magnitudes, float[] phases) {  
   
     for (int i = 0; i < magnitudes.length; i++) {
         correctedMagnitude[i] = magnitudes[i] - mediaPlayer.getSpektrumThreshold(); 
     } 
 }  

Now we can use the values in correctedMagnitude[] to visualize audio since these are now positive values, which can be easily visualized. Later I will show an example!

Why the Threshold?

But before a little thing: For those who now look at these values will find that the higher frequency bands are getting smaller values. These are sometimes so low that they are too low in relation to the other values to be displayed effectively. To show you a bit better here are the values from the song Money for Nothing by Dire Straits (1:20 to 1:30) for 32 frequency bands with a default threshold of -60:

Positive values of the frequency bands at a threshold of -60.

As you can see, the last band is not displayed at all, since it is in a range below 1, even if there are actually sounds. But if we now reduce the threshold to -120, our graph looks quite different:

Positive values of the frequency bands at a threshold of -120.

As you can see, the last frequency band is now clearly visible. Therefore I would recommend to play with the threshold value to see how strongly you want to display the last frequency bands. My general recommendation is a threshold of -100, if you want to display all exactly, or -80 if it does not necessarily have to be all.

Example: Visualizer

So now to a small example how to use these values to visualize audio in JavaFX. Here I would like to point out that I am not a big fan of additional frameworks or libraries, if one can make the same also with the JavaFX own libraries. Therefore we use an AreaChart to create a visualizer!

I do not want to show the FXML code here for an AreaChart, so I just assume that this has the fx:id="spectrum" and is known in the controller. Data in an AreaChart is displayed in series, in our case we only have a series of data, which change regularly. Therefore we create a new series in the initialize() method of the controller:

 public void initialize(URL location, ResourceBundle resources) {  
   
     XYChart.Series<String, Number> series1 = new XYChart.Series<>();  

}  

However, this series also requires data that is represented by an XYChart.Data array. Since we also want to have the values at the beginning all to 0, we still need a for loop, which does this and then assigns to the series. In the end we add the series to the AreaChart "spectrum". The constant BANDS represents the number of frequency bands, which is basically mediaPlayer.getAudioSpectrumNumBands(). Integer.toString(i +1) only serves to make the frequency band 1 also appear as 1 and not as 0, but this does not affect the visualization.

 public void initialize(URL location, ResourceBundle resources) {  
   
     XYChart.Series<String, Number> series1 = new XYChart.Series<>();  
     XYChart.Data[] series1Data = new XYChart.Data[BANDS];  
     for (int i = 0; i < series1Data.length; i++) {  
       series1Data[i] = new XYChart.Data<>(Integer.toString(i + 1), 0);  
       series1.getData().add(series1Data[i]);  
     }  
     spektrum.getData().add(series1);  
  
}  

Now we need to change the data in the series in the AudioSpectrumListener and JavaFX will display it automatically in the AreaChart. Here the corresponding code:

 private class SpektrumListener implements AudioSpectrumListener {  
   float[] buffer = createFilledBuffer(BANDS, mediaplayer.getAudioSpectrumThreshold());  
   
   @Override  
   public void spectrumDataUpdate(double timestamp, double duration, float[] magnitudes, float[] phases) {    
     for (int i = 0; i < magnitudes.length; i++) {
        series1Data[i].setYValue(magnitudes[i] - mediaplayer.getAudioSpectrumThreshold());
     }  
   }  
 }  

Well, that was it. Now we have a visualizer, which visualizes audio. However, this looks a bit choppy, this is due to the fact that it directly displays the new values without flowing transition. If you activate animations with the AreaChart, this does not improve, as the value changes are now too fast for the animation. Therefore, we add a "buffer" manually, which is a little bit behind to make it look smoother. The following code shows the buffer:

 private class SpektrumListener implements AudioSpectrumListener {  
   float[] buffer = createFilledBuffer(BANDS, mediaplayer.getAudioSpectrumThreshold());  
   
   @Override  
   public void spectrumDataUpdate(double timestamp, double duration, float[] magnitudes, float[] phases) {  
   
     for (int i = 0; i < magnitudes.length; i++) {  
       if (magnitudes[i] >= buffer[i]) {  
         buffer[i] = magnitudes[i]; 
         series1Data[i].setYValue(magnitudes[i] - mediaplayer.getAudioSpectrumThreshold());  
       } else { 
         series1Data[i].setYValue(buffer[i] - mediaplayer.getAudioSpectrumThreshold());  
         buffer[i] -= 0.25;  
       }  
     }  
   }  
 }  

private float[] createFilledBuffer(int size, float fillValue) {  
   float[] floats = new float[size];  
   Arrays.fill(floats, fillValue);  
   return floats;  
}

Basically, this code is quite self-explanatory, if the current value is above the buffer, the current value is taken, otherwise the value from the buffer is reduced by a small value, here 0.25.

To see the whole in action here is a small video of my  FXPlayer, from which this code basically originates:

Here is another video from my other JavaFX application AudiPic, which also uses the AudioSpectrumListener to create pictures of songs:

I hope I could help you a little and wish you a lot of fun visualizing audio in JavaFX!

Until next time!