Introduction
There are various methods to collect IoT data using sensors.
Traditionally, Single Board Computers (SBCs) like Raspberry Pi were used. Sensors were attached to the SBC, and the data collected from the sensors was directly stored in a database or storage space installed on the SBC. However, this method has limitations due to the restricted number of IO pins on the SBC and the need for wired connections between sensors and the SBC. Installing sensors in various locations requires significant consideration for wiring and connections.
The increasing price of Raspberry Pi due to inflation and high-performance upgrades is also an issue that cannot be ignored.
In this environment, creating a smart sensor by combining a simple microcontroller with sensors and transmitting data using the MQTT protocol over a WIFI network, and then installing this sensor where desired, can save costs when installing many sensors in various locations. The ESP32 microcontroller board and sensor prices covered in this document are less than 1/10 of the price of the latest version of Raspberry Pi.
ESP32 and Sensors
ESP32 is an SBC (Single Board Computer) similar to Arduino, supporting WiFi + Bluetooth connections and featuring a dual-core RISC CPU with built-in memory and ROM. Firmware can be developed through the Arduino IDE and transferred via USB, allowing the board to operate using that firmware.
In this example, we used the ESP-WROOM32 board. It allows for the creation of useful smart sensors at an affordable price. The sensor used in the example is the DHT-11, which collects temperature and humidity data.
In this example, we used the ESP-WROOM32 board. It allows for the creation of useful smart sensors at an affordable price. The sensor used in the example is the DHT-11, which collects temperature and humidity data.
Because it operates on low power, it can function for a considerable time even when connected to an external battery pack for mobile phones. This makes it convenient to install in places where power connections are not available.
Wiring and Configuration
We connected the DHT-11 cables to the power and GPIO pins respectively. With this setup, we can now read the sensor values.
The GPIO pin number connected to the data line must be specified as a constant value when developing the firmware. If wired according to the diagram below, the data line is connected to **D15**.
Installation of Machbase Neo
To receive data via MQTT and record/store it, installation of Machbase Neo is necessary. Install the latest version of Machbase Neo from doc.machbase.com and run it.
When running the Machbase Neo server with default settings, the connection IP is the IP of the installed equipment, and the MQTT port is 5653. This information needs to be configured in the firmware development below.
__ __ _ _
| \/ | __ _ ___ | |__ | |__ __ _ ___ ___
| |\/| | / _` | / __| | '_ \ | '_ \ / _` | / __| / _ \
| | | | | (_| | | (__ | | | | | |_) | | (_| | \__ \ | __/
|_| |_| \__,_| \___| |_| |_| |_.__/ \__,_| |___/ \___|
_ __ ___ ___
| '_ \ / _ \ / _ \ v8.0.17 (fedc1d8f 2024-04-19T11:42:46)
| | | | | __/ | (_) | engine v8.0.18 (df3cccd1e)
|_| |_| \___| \___/ static_standard_windows_amd64 Windows 10.0 19045
2024/05/10 10:10:39.043 INFO scheduler started.
2024/05/10 10:10:39.044 INFO neosvr MACH Listen tcp://0.0.0.0:5656
2024/05/10 10:10:39.068 INFO grpcd gRPC TLS enabled
2024/05/10 10:10:39.069 INFO grpcd gRPC Listen tcp://0.0.0.0:5655
2024/05/10 10:10:39.069 INFO httpd HTTP token authentication disabled
2024/05/10 10:10:39.070 INFO httpd HTTP path /db for machbase api
2024/05/10 10:10:39.070 INFO httpd HTTP path /lakes for lake api
2024/05/10 10:10:39.071 INFO httpd HTTP path /metrics for the line protocol
2024/05/10 10:10:39.071 INFO httpd HTTP path /web for the web ui
2024/05/10 10:10:39.072 INFO httpd HTTP Listen tcp://0.0.0.0:5654
2024/05/10 10:10:39.073 INFO mqttd MQTT token authentication disabled
2024/05/10 10:10:39.073 INFO mqtt-tcp MQTT Listen tcp://[::]:5653If you've installed Machbase Neo on a Windows environment, you need to configure the firewall settings. Allow machbase-neo to make connections by performing app allowance in the Windows Firewall.
To create a table for receiving data, access http://(server IP):5656 using a web browser. Then, create a table to receive the data.
CREATE TAG TABLE tag (name VARCHAR(20) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE SUMMARIZED) WITH ROLLUP (hour)
Now, the preparation for the Machbase Neo server acting as an MQTT broker to receive data and the table is complete. Proceed with firmware development with the following information ready:
* IP address of Machbase Neo (In this example, it's "**192.168.1.131**")
* MQTT port of Machbase Neo (It's **5653** for default installation)
* Name of the sensor tag table for data collection (In this example, it's **TAG**)
Firmware Development
How the Firmware Works
To read sensor values and transmit them to the Machbase Neo server, it operates in the following order: The firmware for this example is publicly available on GitHub (https://github.com/machbase/esp32-mqtt).
Network Connection and Preliminary Preparation
Connect to the WiFi network. (WiFi SSID and password required)
Synchronize the built-in clock using the NTP (Network Time Protocol).
Connect to Machbase Neo's MQTT broker.
Initialize the DHT-11 sensor.
Check the connection status of the MQTT broker (Machbase Neo server). If disconnected, attempt to reconnect.
Read data values from the sensor. Encode the read data into JSON, which is the MQTT data format of Machbase Neo.
Publish the encoded JSON data to the MQTT topic.
Wait for a certain period, then return to step 2.
Let's examine this process through the code.
Development Tools and Module Installation
Using ESP32 allows for quick and easy development using the Arduino IDE, unlike typical firmware development. First, install the Arduino IDE, then add the "esp32 by Espressif" module from Tools->Board->Board Manager.
Installing the module involves a download and compilation process, so it takes a little time.
Next, install the USB driver for the development board. The board used in the test was CP2102, and its driver could be downloaded from Silicon Labs. https://www.silabs.com/developers/usb-to-uart-bridge-vcp-drivers?tab=downloads
After installing the module and driver, connect the ESP32 board via USB and set the connected USB to serial port. It will then be recognized in the Arduino IDE. If you build and run a simple sample program, you can check the execution results in the IDE's serial console emulator window.
Installing Sensor Driver and MQTT Library
Install the DHT-11 driver and MQTT library through the Tools->Library Manager menu. Search for and install the "DHT sensor library by Adafruit".
Search for and install the PubSubClient library.
Once all libraries are installed, the development preparation is complete.
Sensor Configuration
Configure the sensor information by setting the GPIO pin number connected to the sensor and the ESP32 board, along with the sensor model.
#define DHTPIN 15
#define DHTTYPE DHT11
//DHTTYPE = DHT11, but there are also DHT22 and 21
DHT dht(DHTPIN, DHTTYPE); // constructor to declare our sensorThen initialize the sensor in the setup() function.
dht.begin();WiFi Connection and Time Synchronization
First, connect to WiFi and perform time synchronization using the Network Time Protocol. The ESP32 development board has built-in WiFi connectivity, so there's no need to install additional hardware.
// WiFi
const char *ssid = "MACHBASE_AP"; // Enter your Wi-Fi name
const char *password = "xxxxxxxxxx"; // Enter Wi-Fi password
WiFiClient wifi_connection;Set the SSID and password of the Access Point you want to connect to.
WiFi connection is performed through the following code:
WiFi.mode(WIFI_STA);
WiFi.begin(ssid, password);The ESP32 board does not have a battery to store time values. To collect time series data, it must be able to measure the current time accurately. Now that the network connection is established, we use NTP (Network Time Protocol) to set the current time.
//Time
const char* ntpServer = "pool.ntp.org";
const long gmtOffset_sec = 3600*9;
const int daylightOffset_sec = 0;When synchronizing with NTP, it's based on GMT (Greenwich Mean Time), so to set it to Korean time, you need to add 9 hours. Daylight Saving Time is not applied in Korea, so it's set to 0. The NTP server is set to the public pool.ntp.org. After this, calling the following function in the code will perform time synchronization:
configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);Setting up and Connecting to MQTT Broker
We confirmed the server IP and MQTT broker port number when installing Machbase Neo. Set these values as follows:
// MQTT
const char *mqtt_server = "192.168.1.131";
const char *topic = "db/append/TAG";
const int mqtt_port = 5653;
PubSubClient mqtt_client(wifi_connection);After establishing a WiFi connection, proceed with the MQTT connection using the code below. It's implemented to repeatedly attempt reconnection if the connection fails.
In the main loop of the firmware, check the MQTT connection and attempt to reconnect if the connection is lost.
// initialize mqtt client void setup() { mqtt_client.setServer(mqtt_server, mqtt_port); mqtt_client.setCallback(callback); // other code omitted. } void reconnect() { // Loop until we're reconnected while (!mqtt_client.connected()) { Serial.print("Attempting MQTT connection..."); String clientId = "ESP32-"; clientId += WiFi.localIP(); // Attempt to connect if (mqtt_client.connect(clientId.c_str())) { Serial.println("connected"); } else { Serial.print("failed, rc="); Serial.print(mqtt_client.state()); Serial.println(" try again in 5 seconds"); // Wait 5 seconds before retrying delay(5000); } } }
MQTT Data Transmission
MQTT Topic
MQTT uses pub/sub to exchange data. When inputting data for the tag table created in Machbase Neo, you need to publish to the topic "db/API type/table name". In this example, we'll use the append API to input data, so the topic for input will be "db/append/TAG".
When inputting data to Machbase Neo, the input format can be either JSON or CSV. In this example, we'll use JSON for input.
Input Time Value
The default time value in Machbase Neo is an integer value in nanosecond units. For this test, second-level precision is sufficient, so it was set as follows:
int64_t get_timestamp_milis() {
struct timeval tv;
gettimeofday(&tv, NULL);
return tv.tv_sec * 1000LL;
// return (tv.tv_sec * 1000LL + (tv.tv_usec / 1000LL));
}
... //miliseconds to nanoseconds
current_time = get_timestamp_milis() * 1000000LL;
...Data Transmission
When inputting data using MQTT, the data must be structured in JSON or CSV file format. The code below shows how to convert the data read from the sensor into JSON format and send it to the Machbase server.
// The DHT11 returns at most one measurement every 1s
float h = dht.readHumidity();
//Read the moisture content in %.
float t = dht.readTemperature();
//Read the temperature in degrees Celsius
snprintf (msg, MSG_BUFFER_SIZE, "[ [\"ESP32.hum\",%lld,%f],
[\"ESP32.tem\",%lld,%f] ]",
current_time, h, current_time, t);
Serial.print("Publish message: ");
Serial.println(msg);
mqtt_client.publish(topic, msg);After transferring the implemented firmware to the ESP32 board through the Arduino IDE, it will start running automatically. You can check the execution status using the IDE's serial console. You can verify the input status by performing queries in the Machbase Neo web interface.
Data Utilization with Machbase Neo
Data stored in Machbase Neo can be easily visualized and analyzed without installing additional tools. You can simply visualize the collected temperature and humidity data using Machbase Neo's TAG Analyzer.
Machbase TSDB compresses data for storage, so it occupies less storage space even when collecting long-term data. It can also quickly search through large amounts of data using high-speed index-based search capabilities.
In this post, we developed a network-connectable smart sensor using the ESP32 board. We confirmed that IoT implementation is possible in a very simple way by utilizing the various sensor support functions of the Arduino ecosystem and data transmission functions using WiFi and MQTT. We also found that the data collected in Machbase Neo can be conveniently utilized for visualization and other purposes.