Machbase CEO Seongjin Kim
Starting with
This article is specifically for organizations that fulfill requests for various products and services through public procurement, or those who supply such products and services in the Korean public market, especially in the IoT and related IT fields that handle large amounts of data. Written.
As the CEO of a company, I cannot rule out a certain degree of bias, but I am writing this article to summarize my hope that Korea's IT market will become more mature and develop, as well as the sadness I feel when looking at the current state of our public procurement market. It is done.
According to Gartner, the era of time series databases has already arrived.
Gartner, a global company that provides information technology research and advisory, publishes a report on a field called Data Management every few years. Through this, we provide valuable information so that you can see at a glance the status of various data processing technology fields and how they are developing through a unique picture called the Hype Cycle.
This Hype Cycle divides the development stages of a specific technology (product) into five parts below.
1.Innovation Trigger (Start of Innovation)
This is where new, amazing technologies first appear.
I don't know how well this technology will be accepted by the market, but at least that's how it appears to customers.
2.Peak of Inflated Expectations
Ah...customers believe that this is truly an amazing product that will change the world, and their expectations are at the highest level.
Early adopters are rushing to use it and expect great results.
3.Trough of Disillusionment
There are points that differ from actual expectations, and this is the point where you are deeply disappointed with the product and technology.
Most technologies and products disappear at this stage.
However, the seeds of innovation take root, and survival technologies begin to emerge.
4.Slope of Enlightenment
It has proven useful in real markets, and many companies have adopted the technology.
This refers to the diffusion stage where the innovation and value provided by the technology are proven in the market.
5.Plateau of Productivity
This is the stage where the value of the technology and product has been fully proven, and a unique market has been formed.
The point is that it no longer approaches the market with the value of innovation, but exists as a commercial good that must be used and utilized as a matter of course, like air and water.
So, let’s take a look at the Hype Cycle of Data Management published in 2019 found on the Internet. (Since it was published in 2019, all data appears to have been collected in 2018)
If you look at the Peak of inflated Expectations, surprisingly, the Time Series DBMS (time series database) is right at the top. At this time, Gartner estimated that 1-5% of the global market was aware of this product or was applying it to related industries. And, since it is marked in light blue, it is predicted that it will become a fully mature market in the next 2 to 5 years.
It may be TMI, but in the picture above, the Trough of disillusionment shows the Key-Value DBMS as “obsolete before plateau,” which also means that this technology and product is no longer viable in the market. Come to think of it, I've hardly ever heard of Key-Value DBMS recently.
And, let’s take a look at the recently published Hype Cycle for 2023.
In the picture above, Time Series DBMS is located on what is called the Slope of Enlightenment. Additionally, it is predicted that the marked white dot will become a full-fledged market within two years at most. According to Gartner, as of 2023, 20-50% of the population in related industries around the world are not only aware of Time Series DBMS, but are also actively using it in their field.
So, what on earth am I trying to say about the relationship between this Time Series DBMS technology development and Korea’s public procurement market?
A new world consumed by sensor data
I am sure that 95% of the people reading this article are probably hearing for the first time what a time series database is and what it is used for. (Gartner estimates that it is around 20% to 50% of those involved globally, but in Korea, where it is slower, a much lower percentage of practitioners will have access to the technology.)
However, it seems worthwhile to take a quick look at what time series databases have already established themselves as a major technology product overseas.
There is a historical context in which time series databases have been rapidly developed and spread since the mid-2010s, and this is closely related to the spread of sensors, which are a major data source in the IoT market, and our intelligent environment called Smart-X. If we look around us at buildings, equipment, cars, roads, and even living rooms, we live in an intelligent environment where IoT sensor data, which barely existed 10 years ago, is collected everywhere through sensing. In other words, sensor data is covering the world, and whether we like it or not, we live in this era, and the picture below clearly shows that reality.
After 2020, the most abundant data type is sensor and object information, and this data is increasing exponentially, and the data needs to be processed immediately and in real time.
Naturally, this enormous increase in sensor data is called “time series data,” and since existing databases such as Oracle could not process it well, people developed a “time series database,” and the global market is rapidly adopting this product. It will. This is the biggest core of the story I want to tell.
A world running with time series databases, Korea seems to have stopped
What on earth is a time series database used for? Even if you simply look at the data, it is being applied in too many places today as shown below.
Smart Building: Collection of sensor data and intelligence of buildings and fire/hazard prevention
Smart Farm: Collection of various plant growth environment data and autonomous control through AI
Smart Ship/Car/Engine: Collection and analysis/decision making of large quantities of mobility data
Smart Factory: Analysis/quality control of real-time manufacturing and vibration, etc.
Smart City: Integrated large-scale data collection and real-time response on climate/traffic/population/dust, etc.
Oil/Energy/Gas Infrastructure: Real-time collection of pressure/valve/temperature/humidity/vibration sensor data/risk prevention
IT infrastructure monitoring: IDC center equipment data collection/monitoring/abnormal situation prediction
Robot and robot control/abnormality detection: motor and drive unit data collection/analysis/risk prevention
Semiconductors and autonomous vehicles: quality improvement and failure prevention through real-time data storage and analysis
Manufacturing and chemical fields: Real-time manufacturing data collection/analysis/quality improvement
Secondary battery/rechargeable battery field: Real-time battery status collection/visualization/analysis/fire prevention and real-time alarm
Finance/Securities: Real-time transaction data collection/analysis/high-speed transaction system/anomaly detection
AI-based abnormality detection and predictive maintenance: learning and inference data infrastructure through fast data extraction
All other cases where large amounts of sensor data are generated.
Even conservatively, if this applies to a market that only 20% of industry insiders are aware of, what will happen in the next two years? Perhaps time series databases are applicable to most social infrastructures?
On the other hand, you can probably easily guess why the title “Korea at a standstill” was used here.
The biggest reason for introducing a time series database - cost-effectiveness
So, what are the advantages of a time series database, and why are countries other than us already using this product by 20-50%?
The biggest reason is that it is the only data storage that stores hundreds to hundreds of billions of sensor data in real time, allows for quick search, and enables data analysis such as AI in real time. In addition to these advantages, the main reason for its adoption is the economical reason of reducing the various resources used in data processing, such as CPU usage and storage space, as well as the power consumption to run it and the cost of hardware infrastructure that must be provided, by several times. no see.
In Korea, such sensor data is generally used simply as “big data,” so in the public procurement market, big data platforms such as “Hadoop” are often used or built through traditional databases. Of course, there are cases where good results can be achieved through a good construction company, but when considering the efficiency, scalability, service performance, and maintenance costs of the entire system, it is very different from introducing a “time series database.”
Below, let's compare a time series database and Hadoop on the total cost of ownership for IoT sensor data processing, citing data officially released by TPC.org, which compares the performance and cost of official database engines around the world.
As you can see above, if you process one unit (1000 units) of data and the time series database costs 1 ($54.85), you will have to pay a whopping 6.01 times more ($329.75) for Hadoop. Moreover, this cost is a combination of the total cost of ownership of hardware and SW, which does not even take into account the future increased storage space costs of the HW and future additional power costs required for operation. The bigger problem is that big data solutions such as Hadoop inherently have structural problems that make it difficult to process IoT sensor data in real time, and it is also quite cumbersome and difficult to find, develop, and manage relevant experts.
To elaborate further, as the trend of edge computing spreads further in the future, there will be more requirements to process and analyze sensor data collected from terminal devices through palm-sized computers. In this case, Hadoop cannot be used at all, so a single packaged software such as a time series database will be the only alternative.
Data services in the era of paradigm-changing energy transition
The biggest reason for collecting various IoT sensor data in the public procurement market is that public/government data services are the main goal going forward. Through this, we are improving the quality of people's lives and using this service as the first button to move towards a better welfare state.
However, now we have reached a point where even the public service process not only has to take into account the global agenda called ESG, but has also been upgraded as a form of the future. (Reference: https://www.lifein.news/news/articleView.html?idxno=14487 )
So what on earth do time series databases have to do with ESG? This is because it is a major axis that enables “achieving carbon neutrality” and eco-friendly services through ESG practices.
In the future, the amount of data, service types, number, and system infrastructure for public/government data services will inevitably and continuously increase. And, it is certain that these services will explode once again under the name of “building public data services,” similar to the explosion of “building public institution homepages” during the old Internet revolution.
Perhaps, an era is coming when building a high-efficiency data service system with the lowest power consumption and hardware construction costs, and being close to carbon neutral and eco-friendly, and being praised as most appropriate for the ESG mission, will be the proudest decoration of the relevant public institution. ?
Public institutions in Korea are starting to run
When introducing the Gartner Hype Cycle, I said that if the world's time series database industry suitability is about 20%, Korea's would be less than 5% at most.
I am very happy to see that our country's public institutions have gradually recognized the problems with data services since last year and are introducing time series databases one by one, both yin and yang, which is very surprising and gives me hope.
The easiest indicator to determine whether a public institution is using a time series database is the purchase of products through “Nara Marketplace.”
Below is a record of Hydro & Nuclear Power Co., Ltd. purchasing a time series database for integrated analysis of related sensor data.
In addition, we were able to find records of related projects being carried out at the “Korea Institute of Civil Engineering and Building Technology” for the storage and analysis of measurement data through IoT sensors in 2023.
Lastly, I was also able to find out that “time series database” is being used as a core database for the “Seoul City Real-time City Data Service,” which services Seoul City’s enormous IoT sensor data as of today.
As everyone knows, in the public procurement market, it is very important for the ordering body to draw a detailed picture of the project and what philosophy and concerns it has. Otherwise, there is no choice but to listen to the construction company's opinion unconditionally. Therefore, there is a task to acquire more external experience and information about what problems the public organization is currently facing in the public procurement market, how to solve these problems in the future, and what kind of service will be transformed in the future. That also seems to be true.
Despite many difficulties, I once again express my respect to the people in charge who always try new things and find new technologies and products through global technological trends and future concerns. In addition, I hope that more consideration and analysis will be given regarding the utility and necessity of the “time series database,” which is the subject of this article, in a way that can help the development and future of public organizations.
Data, data, data and innovation
No matter what anyone says, we are now in the era of data.
In particular, as of today, with the advent of IoT, the largest amount of future data will be time series sensor data. Today, as the world changes, new products come out, and the types of services and user patterns and requirements change day by day, we look back on the current state of “data services.” Although it is insufficient, I will end with the hope that this material will be a piece of brick that will help the development and innovation of various organizations that play a large role in the public-related market.
thank you