Introduction
In this article, we will describe how to utilize the Fast Fourier Transform (FFT), which can be most powerfully employed with incomparable performance in "Time Series DBMS". Furthermore, through samples provided by Machbase Neo, the following practical exercises are possible, and we are confident that anyone can easily modify and utilize them in the most convenient and rapid manner for their specific purposes.
Upon completing this article, you will be able to:
Visualize real-time 3-axis vibration data simultaneously at a sampling rate of 1000 samples per second.
Visualize the Fast Fourier Transform (FFT) results for each of the 3 axes in real-time without additional programming.
Instantly visualize and plot trends of Peak, RMS, and Trough values of vibration data in real-time without additional programming.
Obtain FFT results most rapidly for billions of vibration data entries in Machbase Neo by specifying arbitrary sensors, time periods, and analysis conditions.
If you wish to quickly view an operational demo video, you can navigate directly to this link, or proceed to the prepared practical exercise link.
Fourier Transform
For detailed information about this Fourier Transform, please refer to the following link. If you want to understand the historical significance of the Fourier Transform in an engaging way, I strongly recommend the video titled "The Most Important Algorithm in History" from the 3Blue1Brown channel. (It's fascinating!)
The simplest illustration for beginners to understand what a Fourier Transform is is shown below.
Note: I've translated the content as requested, maintaining the reference to external links and videos. However, as an AI, I can't access or view these external resources. If there's an image that was meant to be included with "The simplest illustration... is shown below," I'm unable to see or describe it. If you'd like to share that image directly in our conversation, I'd be happy to describe or discuss it.
It would be easiest to understand this algorithm as one that allows you to visualize at a glance which frequency bands are included in waveforms such as voice or vibration that appear as time series on the left, based on the X-axis frequency. It's worth noting that the Fast Fourier Transform (FFT) is an improved algorithm that accelerates this transformation process.
Where is the Fast Fourier Transform (FFT) utilized?
As referred to in the previous video as "the most important algorithm in history," the applications of this algorithm are virtually limitless and can be considered to be utilized in almost all data analysis systems.
Communications: FFT plays a crucial role in digital signal processing, used for data compression, error detection and correction, signal modulation and demodulation.
Audio and Acoustics Technology: Spectrum analysis of audio signals, sound quality enhancement, noise reduction, etc.
Image and Video Processing: FFT is used for image filtering, compression, enhancement, and pattern recognition.
Radar and Satellite Systems: Processing and interpretation of radar signals, spectrum analysis of satellite data.
Medical Imaging: Processing medical imaging data such as MRI and CT scans.
Financial Analysis: Analysis of financial data, such as periodicity analysis of time series data.
Geophysics and Exploration: Wave analysis in seismology, resource exploration.
Scientific Research: Analysis of experimental data in various scientific fields including physics, chemistry, and engineering.
Speech Recognition: Analysis and recognition of speech signal characteristics.
Issues with Fast Fourier Transform Utilization in Professional Fields
With such an excellent algorithm already in existence, one might assume it's being well utilized across all industrial sectors without major difficulties. So, what exactly is the problem here? As briefly mentioned in the algorithm description above, since the algorithm itself is already complete, issues related to further improving or developing it are not included in this domain.
As you might have guessed, the real problem lies in the scale and utilization of data for FFT processing. For a long time, relevant industries have been analyzing FFT in real-time, and various tools and visualization software have evolved to allow anyone to easily analyze data.
However, the issue is where this analyzed "data" resides.
This might sound like a ridiculous question, but it has begun to emerge as a significant problem in professional fields. Looking briefly at the previous examples of applications, the most basic data is audio or vibration data. Generally, vibration data collection starts from as low as 500Hz and can go up to 100KHz. In other words, in a field where multiple sensors are installed, each generating 100,000 data points per second, FFT analysis can be performed briefly (30 seconds to 1 minute) through optimized specialized equipment, but how can the following requirements be addressed with this alone?
I want to analyze FFT continuously. I can't trust the accuracy of analyzing for just a few minutes a day.
I want to perform FFT analysis later based on historical data under conditions I specify.
I want to visualize not only this FFT but also analysis results such as Peak, RMS (root mean square) on the screen in real-time, and set up real-time alarms.
I want to use this data later as training data for AI, and extract and train desired data without additional coding during learning. The cost of data transformation is too high.
I want to use the raw vibration data being input in real-time, along with the FFT transformation results, as input for AI inference. In real-time!
In the real industrial field, where the world is changing, customer requirements are increasing, and leaps through new technologies are necessary, encountering such difficult-to-solve barriers is the reality.
The Fantastic Convergence of Time Series DBMS and FFT
The synergy between FFT and "time series databases" is so perfect that it wouldn't be an exaggeration to say that time series databases were "Born to be a rockstar!" for FFT. The technologies on both sides are a match made in heaven.
The reason for this is that the input speed of vibration and acoustic data, which FFT targets, can reach thousands to hundreds of thousands of entries per second. Moreover, to support real-time analysis, extraction performance of over 1 million entries per second is required. When such data accumulates, it can easily exceed hundreds of millions of entries per day.
The database that perfectly solves this problem is the "time series database". Before this technology emerged, in most cases where FFT computation was needed in industry, data was discarded rather than stored. Recalling this fact helps us imagine how serious this data problem must have been.
As a result, memes like "there's no need to store raw vibration data" or "raw vibration data is garbage" are widespread in vibration/acoustic and industrial fields. However, this is equivalent to saying "there's no use in going to the moon" or "those grapes must be sour". This is because it's extremely difficult to properly collect such vast amounts of data and utilize it in real-time.
Nevertheless, professionals in the new technology generation often say, "What? You're still discarding data? Why? I'd like to collect and analyze it... Is that impossible?" Even today, they are deeply contemplating the usefulness and analyzability of this data.
Therefore, this article was written to demonstrate actual examples of performing various real-time analyses on high-frequency audio or vibration data on a powerful data infrastructure called a time series database. It's worth emphasizing again that these examples include no coding except for a few lines of JavaScript within HTML documents, and are based purely on the functions of Machbase Neo.
Hands-on Practice with Real-time FFT Analysis
0. YouTube Video for Those in a Hurry
The following section contains practical instructions for developers to install Machbase Neo, generate simple vibration sensor data for X, Y, and Z axes (each producing 1000 samples per second) using prepared scripts, and perform real-time visualization of FFT transformations, Peak, RMS, and other real-time statistical measures. It also covers how to conduct detailed FFT analysis on desired sensors at specific times from hundreds of billions of collected data points.
Meanwhile, for upper management and planning teams who may not need to follow all the technical details, we've prepared a roughly 5-minute video below for a quick overview.
1. Installing Machbase Neo (since 8.0.17)
Access the system through a web browser as shown above, and log in using the initial credentials:
username: sys
password: manager
2. Download the neo-apps
Neo provides a method to directly download source code from GitHub internally. As a result, you can install it quickly and immediately as shown in the image below. We'll assume that the latest version of Neo is installed and running.
Click on the third icon in the upper left corner, and a Git Clone menu will appear in the center. Enter the source code address https://github.com/machbase/neo-apps.git here, and click OK. The app will then be automatically installed.
Note: As an AI, I can't see the image you're referring to, but I've translated the description of the process as you've outlined it.
Once the installation is complete, a top-level directory named 'neo-apps' will be created as shown in the image below. When you click on the subdirectory named 'realtime-fft', you can view the necessary source code as follows.
3. Executing Data Schema and Data Input Timer
Configuring the FFT Sensor Table
For this demo, we've simulated actual vibration sensor data, manipulating it to create various patterns for maximum demonstration effect. We've assumed one vibration sensor with X, Y, and Z axes, each generating 1000 data points per second. This generation is accomplished by preparing Machbase Neo's input TQL and using an internal timer to call it once every second, inputting 1000 data points per second for each axis. From the database perspective, this results in high-speed input of 3000 sensor data points per second.
insert-auto-fft-x.tql (X-axis vibration data, 150Hz with amplitude 2.0 + 300Hz with amplitude 50)
insert-auto-fft-y.tql (Y-axis vibration data, 100Hz with amplitude 1.0 + 0.1Hz with amplitude 10)
insert-auto-fft-z.tql (Z-axis vibration data, 0.5Hz with amplitude 20 + 15Hz with amplitude 10 + 35Hz with amplitude 3 + 220Hz with amplitude 2 + 400Hz with amplitude 2)
Open the provided worksheet file **realtime-fft.wrk**. It contains the schema that needs to be created and the timer script to be executed and registered in the Neo shell. Execute each of these. Once done, all preparations for data input will be complete.
Creation of Table FFT
Adding Retention Period
If data is input at a rate of 3,000 entries per second without limit, a disk full error could occur, causing service issues. To address this, Machbase provides a RETENTION feature. This function specifies a period for which data should be kept in the target Tag table, and deletes data that exceeds this period, maintaining a consistent storage space.
In this demo, we've applied a policy to store one month's worth of data and delete old data once daily, as shown below.
CREATE RETENTION policy_1m_1d DURATION 1 MONTH INTERVAL 1 DAY;
ALTER TABLE fft ADD RETENTION policy_1m_1d;For detailed information about RETENTION, please refer to the relevant manual (link)
4. Creation of Timer
Copy of Timer Script
Executing Timer script in SHELL
After creating a shell window and pasting the content copied from above, the timer will execute as shown below, and a total of 3,000 vibration data points per second will be input into the FFT table.
In the example above, we can see that a total of 30 million data points have been input so far.
A Brief Explanation of the Input TQL
Let's examine the source code of input-auto-fft-x.tql. This code is called once every second by a timer, and at that time, it instantly generates and inputs 1000 data points into Machbase.
FAKE( oscillator( freq(150, 2.0), freq(300, 50), range("now", "1s", "1ms") ))
MAPVALUE(1, value(1) + (random()-0.5) * 0.1 * value(1) )
PUSHVALUE(0, "vib-x", "name")
APPEND(table("fft"))After generating 1000 waveform data points from the current time to one second later using FAKE(), noise is mixed in with random(), the tag name 'vib-x' is specified, and then high-speed data input is performed into the fft tag table using the APPEND command.
5. Checking the Real-time Raw Data Dashboard
Now, let's output the latest one second of data for each sensor that is actually being input to a dashboard. This dashboard has already been created as a file called realtime-fft.dsh, so when you click on it, the real-time visualization screen below will be displayed. If needed, let's try drawing charts with various options.
While actual sensor vibration data is not as elaborate as this (the X-axis is the most similar), note that the data patterns have been designed to be distinguishable from each other for demonstration purposes.
6. Performing Real-time FFT Analysis (realtime-fft.html)
After loading realtime-fft.html as shown below, when you press the arrow at the top of the source code, the corresponding homepage will start being hosted by the internal web server, and the homepage screen will appear in the browser.
The following screen will appear in the web browser.
This screen is composed as follows:
It performs FFT analysis on the previous 1 second of data for X, Y, and Z axis data based on the current time, and visualizes it in 2D in real-time.
It performs FFT analysis on the previous 10 seconds of data for X, Y, and Z axis data based on the current time, and visualizes it in 3D in real-time.
So, a total of 6 types of FFT analyses are being performed in real-time! Isn't it impressive? The update interval is set to 10 seconds for slower systems, but it can be tuned as needed.
However, let's keep in mind that this is just a small beginning!
7. Performing 3D FFT Analysis with Various Options (custom-fft.html)
Similarly, when you load custom-fft.html from the source code, the screen below will appear. This screen is designed to allow users to perform FFT analysis by specifying various options. The options are as follows:
AXIS: Select X, Y, or Z axis
Analysis Time: Specify how far back from the current time to analyze FFT (up to 60 seconds)
Analysis Precision: Finely adjust the analysis unit from 50ms to 1000ms
Minimum/Maximum Frequency: This determines the frequency range of the analyzed data
Theme: Specify various visualization theme options
In this screen, you can't specify the exact time you want, and while this feature could be added, it wasn't implemented separately because Machbase Neo's own tag analysis tool, Tag Analyzer, provides this functionality more powerfully. Let's check it out in the next section.
8.Performing FFT Analysis Based on Tag Analyzer (realtime-fft.taz)
As mentioned earlier, Machbase Neo's Tag Analyzer not only powerfully visualizes the data of desired tags but also provides FFT analysis.
For the demonstration, realtime-fft.taz is provided. When you open this file, it outputs the actual data of each axis for the latest data on the screen as shown below.
* The red box in the upper left corner: When clicked, it redraws the chart with the latest data
* The red box in the upper right corner: When clicked, it switches to data range selection mode for the chart
The screen below shows that when you select the corresponding data range (range of the red horizontal line) and click on the newly appeared icon (chart shape), it visualizes the result after performing the actual FFT analysis.
Below is the popup window that appears when you press the analysis button.
The actual FFT analysis screen appears as shown above, and zooming in and out is possible through the sliding window below. Finally, when you press 2D in the upper right corner, it transforms into 3D. At this point, if you press the play button right next to it, you can see the analysis screen changing to 3D. You can adjust the analysis resolution by specifying the interval value. (The default is 100ms)
As shown above, a 3D analysis screen is displayed. After modifying the interval and pressing the red play button on the right, the FFT analysis screen is updated with the adjusted resolution.
9. Real-time Statistical Analysis (realtime-data.html)
When you open and run realtime-data.html, the screen below will appear. This homepage is a simple example of calculating statistical functions other than FFT when analyzing actual vibration data in real-time.
The top three charts display all the raw data for each X, Y, and Z axis over a one-second period. (The screen refreshes every 3 seconds) The bottom three charts show time series for a total of three types of statistics (Peak, RMS, Trough) at 2-second intervals, based on real-time statistical values for the past 10 seconds of data for each axis.
These real-time statistical values have the following meanings:
Peak: The maximum amplitude real value during the corresponding 2 seconds (maximum value)
Trough: The minimum amplitude real value during the corresponding 2 seconds (minimum value)
RMS: The calculated Root Mean Square value for the corresponding 2 seconds (magnitude of vibration)
If necessary, more statistical values can be added. Let's refer to the TQL that calculates these values in real-time as follows:
realtime-data-stat.tql
SQL_SELECT(
'time', 'value',
from('fft', param('axis') ?? 'vib-x'),
between(strSprintf('now-%s', param('range') ?? '10s') , 'now')
)
GROUP(
by( value(0),
timewindow(time(strSprintf('now-%s', param('range') ?? '10s')), time('now'), period(param('precision') ?? "2s")),
"TIME"
),
max( value(1),
"PEAK"
),
rms( value(1),
"RMS"
),
min( value(1),
"TROUGH"
)
)
MAPVALUE(0, strTime(value(0), 'DEFAULT', tz('Local')))
CHART( ....생략...)10. Performing Statistical Analysis with Various Options (custom-data.html)
Now, here's the last prepared analysis screen, custom-data.html. This screen is designed to allow users to perform statistical analysis on the latest data through various options.
The left side of this screen shows the raw data from the actual sensor, while the right side shows the analysis chart. The following options can be specified:
Tag type: X, Y, Z axis
Analysis time range: From now to XX seconds ago (from 2 seconds to a maximum of 60 seconds)
Statistical analysis interval: 1, 2, 3, 5, 10 seconds
Below is a screen that outputs Peak, RMS, and Trough at 2-second intervals for all data from the last 30 seconds for the X-axis.
Below is a screen that calculates and visualizes various statistics at 5-second intervals for all data from the previous 60 seconds for the Y-axis.
Closing
Through a simple demonstration, we have verified that real-time FFT and statistical analysis are possible even in situations where thousands of data points are being input per second, based on a time series database. While we minimized the amount of data and the scale of analysis considering the optimal hardware specifications of various users, we are confident that when actually utilized, input of tens to hundreds of thousands of data points per second and large-scale data analysis are fully possible with additional hardware support.
Moreover, it would be ideal if it could be widely known that real-time data analysis has become possible even in environments where large amounts of data are generated through Machbase Neo, and furthermore, that there is no need to delete or transform the relevant data.
If you have any further questions or needs related to this, we promise to kindly respond if you contact Machbase through the link below. With this, we will conclude this article.