How We Do It: Offline IoT

(This is a repost of a blog post made for thingsquare.com, slightly edited to fit this format.)

When the wireless mesh is unavailable, store data locally.

A wireless mesh network reliably connects IoT (Internet of Things) devices, even in places where wireless is tricky.

But even with a wireless mesh network, the IoT devices may become disconnected. We then need to store their data offline – and have them automatically post their data when they come back online.

In a recent customer project, sensors were deployed deep inside freezers to monitor vaccines.

The freezers’ walls sometimes would block the wireless signals. So we needed to store the data as the devices were offline, and automatically send the data when the devices were reachable again.

With the Thingsquare IoT platform, we use the built-in offline buffering mechanisms to do this.

Today, we’ll look in detail how we build solutions that work even if the wireless mesh network is not always available.

Why offline IoT? 🔗

IoT systems are almost always online. Devices can directly post their data over the wireless mesh network as soon as they get it.

But what if the network is not available? This can happen when an access point has a spotty cellular connection to Internet, or if the devices themselves are placed in spots where their wireless connection is spotty.

In our customer’s scenario, wireless sensors were used to monitor vaccines. These vaccines are stored in freezers, deep inside buildings.

The sensors are connected in a wireless mesh network, which helps maintain good connectivity. Because it is a mesh network, relay nodes extend the reach of the network. But even with a mesh network, devices may temporarily be out of range.

So what to do? Enter offline mode.

With offline mode, devices store their data locally. If the device is online, it will immediately send their data to the backend server database.

But if it happens to be offline, the local data storage will keep the data for as long as it needs to - until the next time the device is back online.

Even something as reliable as a wireless IoT mesh network may break. This is when we enable offline mode.

Challenges with offline IoT 🔗

The wireless devices are battery-powered and small.

Inside each device is a microprocessor and a memory chip. The microprocessor (such as the CC1352R) can only hold temporary data. This data will go away if the battery dies.

Offline data needs to go into the memory chip where it will survive a reboot or power outtage.

Memory chips for this class of devices can hold a few megabytes of data. One such example is the MX25U1633FM2I03 NOR-flash chip. It has a 2 megabyte capacity and a pricetag of about 37 US cents.

Parts of the external memory chip is reserved for Over-The-Air (OTA) updates, usually some 400 kB, but the bulk part of the chip is available for offline data.

In principle, this is straightforward, but the devil is in the details. We need to handle:

• Synchronized data timestamps.
• Data serializing and de-serializing.
• Protect against data corruption.
• Data expiration.
• Encryption-at-rest.
• Data pacing to avoid overwhelming the network when many devices go online simultaneously.

Fortunately, the Thingsquare system helps us with this.

How the Thingsquare IoT platform helps 🔗

The Thingsquare system is designed to build complete IoT solutions, and includes tools for offline processing.

Offline data is timestamped, checksummed, encypted and stored in the external flash chip.

When the device is back online in the wireless mesh network, the data is sent to the server piece by piece. This is handled gracefully and reliably so that the network isn’t overwhelmed, and ensures that no data is lost.

Even if the device would be powered off in the middle of a transfer, the system picks up where it left off when it is powered up again.

Since data is timestamped, the server inserts it in the data history at the corresponding place.

The result is a data history without gaps, as if the device had been online the whole time.

To demonstrate offline mode, we build a small custom IoT solution, based on off-the-shelf hardware.

Example: A custom IoT solution 🔗

To illustrate how offline mode works, we build a miniature, bare-bones IoT solution. We use off-the-shelf hardware and the Thingsquare IoT platform.

In the Thingsquare platform, each IoT solution has different hardware and software.

In general, when we build an IoT solution, Thingsquare’s experts will develop the software to suit the needs of the project, and the source code for the solution is made available for the customer.

Devices are programmed in the C programming language. In this article, we show the exact C code statements that a device will run to operate in offline mode.

In this example, we do just that. But in a much smaller scale.

We use the Texas Instruments CC1352R-based LPSTK. It is a sub-GHz wireless, battery-powered demonstration platform with sensors: temperature, relative air humidity, air pressure, and ambient light.

We develop a custom application that periodically samples the sensors. Data is pushed to the server, or stored in the external flash chip if we are offline.

Step 1: Define the data 🔗

In the Thingsquare system and its offline mode, data is stored in queues. Each queue has its own data structure.

We first define the structure of the data. We do this with a C language struct definition.

In this example, the same format can be used for all of the data we want to store, both battery voltage and relative humidity, so the same queue and struct with a simple float is used for all.

More complex compositions are of course possible, and having several queues with their respective format.

struct queue_data {
  float value;
};

Step 2: Data queue definition 🔗

Next, we define the queue and how much space in external flash we will use. Here we allocate 32 kB for this queue.

#define STARTPAGE  0
#define PAGES      4

THSQ_XMEM_QUEUE(data_queue,
                struct queue_data,
                send_queue_callback,
                STARTPAGE, PAGES);

The data_queue is the name for our C variable. The struct queue_data is the data structure we defined above. The send_queue_callback is a C function that we define below. STARTPAGE and PAGES define how much of the flash memory we want to dedicate to this queue.

Step 3: The code that sends the data 🔗

Next up, we define the send_queue_callback function. This is invoked when a datapoint is pushed to the server and allows us to set the format of the data as sent to the server. In the code below, printfloat() prints a float to a buffer.

static enum thsq_err
send_queue_callback(struct thsq_xmem_queue *q, uint32_t timestamp, const char *key, void *data)
{
  /* thsq-queue is ready to send next element from queue */
  struct queue_data *queue_data = (struct queue_data *)data;

  char valuestr[16];
  printfloat(valuestr, sizeof(valuestr), queue_data->value);
  return thsq_sset_printf_important_timestamp(timestamp, key, "%s", valuestr);
}

Step 4: The code that enqueues the data 🔗

To add datapoints to our queue, we use a small helper routine.

static void
enqueue_data(const char *variable, float value)
{
  struct queue_data d;
  d.value = value;
  thsq_xmem_queue_add(&data_queue, variable, &d);
}

This function is called every time we want to push data to the queue.

Step 5: Enqueing the data 🔗

When it’s time to add data to the queue, we do as the example below for the HDC2010 relative humidity and temperature sensor. Since the sensor returns fixed point scaled results, we scale down to the proper SI unit again and store as float.

hdc20x0_data_t hdc_result;
hdc20x0_read(&hdc_result);
enqueue_data("rh", (float)hdc_result.rh / 100.0);
enqueue_data("t", (float)hdc_result.temp / 100.0);

Step 6: Fake being offline 🔗

To test the offline operation, we need to fake being offline.

We use the on-board button on the LPSTK device to turn the device from an online to an offline device.

This is the code:

static void
callback_thsq(enum thsq_reason r, const char *msg, int num)
{
  if(r == THSQ_PERIOD) {
    /* read sensors */
    process_poll(&sensor_read_process);
  }

  if(r == THSQ_BUTTONACTION) {
    if(num == 2) {
      should_be_offline = !should_be_offline;
      process_poll(&offline_scheduler_process);
    }
  }
}

Trying it out 🔗

To try things out, we can now run the setup in offline mode for a few days.

We use the default Thingsquare web frontend to view the data that the LPSTK generates.

To test it out, I simply double-click the button and see the LED verification that it is in offline mode. It is then transported to a place that I want to gain some insight into.

The LPSTK is retrieved a week later. After a second double-click, it goes back online and starts to offload data from the past week to the server. Space in external flash is reclaimed as data point by data point is acknowledged.

After a while, we can see the data history for the past week using the app or web interface. There are no gaps in the data despite being offline.

We’ve thusly enabled data collection in a remote location where no coverage was available. It enables a product to be temporarily deployed without setting up the supporting infrastructure.

Your turn 🔗

Would you like to hear more about how Thingsquare can help your product come alive? Please do contact us, we’re looking forward to talking to you. Should you prefer it, feel free to use the IoT solution planner to get an automated time and cost estimate.