B4R Question In the LITTLEFS library how to save a structure

[Gerardo Tenreiro]

Hello Group,
How can I save a structure to a file using the LITTLEFS library?

I have a series of values that don't fit in RAM. This is the structure I'm attaching.

Type D_Logger1 (P_Encoder As Int, _ ' Posicion del Encoder
P_Encoder_F As Int, _ ' Posicion del Encoder Filtrada
P_Puerta As Int, _ ' Posicion de la Puerta en %
C_Velocidad As Int, _ ' Consigna de Velocidad al Accionamiento
Intensidad_M As Int, _ ' Intensidad Consumida por el Motor
Secuencia As Byte, _ ' Numero de Secuencia
CP_Encoder As Byte, _ ' Calidad de la Comunicacion con el Encoder en el ultimo Segundo
CT_Encoder As Byte, _ ' Calidad de la Comunicacion con el Encoder Total
Modo_M As Byte, _ ' Indica el Modo de maniobra de la Puerta
SG1 As Byte, _ ' Grupo 1 de Variables
SG2 As Byte, _ ' Grupo 2 de Variables
SG3 As Byte, _ ' Grupo 3 de Variables
SG4 As Byte _ ' Grupo 4 de Variables
)
Public Logger1(420) As D_Logger1 ' Datos del Logger

The values are saved in the variable "LOGGER1" at a time interval of 20 to 40 mS.
I need to store about 800 values, but if I expand the variable to more than 500, I'll run out of memory on the ESP32.
Since the data collection process is in two parts: about 2 seconds of activation in one direction, 1 second of pause where I don't need to store data, and 4 seconds in the opposite direction where I need to store data again.

The idea is to use the 420 positions I have available to take two blocks of data, the first and the third. During the second block, store the values from the first block in a file, and when finished, store the data from the third block.

In short, I need to store the 420 positions of the variable in a file as quickly and efficiently as possible.

This would involve opening the file in write mode. The data in the file would be deleted. Write the 420 positions, and then close the file.

I have two problems.
1. The problem is with the length when saving. How do I know the length of the variables?
2. How do I convert the variable LOGGER1(n) to a byte to store it with the LITTLEFS library?

Thank you.

 

[peacemaker]

The variable of D_Logger1 type is just 18 bytes length. I guess, you can use LOTS of such var instances in the ESP32 RAM...
I used such settings for big sketch for HTML operations without problem, try also.

#Region Project Attributes
   #AutoFlushLogs: True
   #CheckArrayBounds: True
   #StackBufferSize: 17000
#End Region

800 х 18 = just 14400

 

[Gerardo Tenreiro]

Thanks, Peacemaker
The ESP32 uses 99% of the available memory; it's a very large application that requires a lot of memory.
In my experience, increasing the stack size doesn't affect global variables, which are the ones I'm using.
If you set "#StackBufferSize: 17000," you're using RAM for the stack, and this memory can't be used for global variables.
The stack size is consumed by local variables and module interleaving. The deeper you are and the more local variables you use, the more stack size you need.
In my application, 600 is sufficient. I know this because I analyze stack usage in each module and never exceed 500. So, 600 is perfect for leaving the maximum RAM size for global variables, which are the ones that consume RAM.

That's why, if I increase the size of the LOGGER1 variable, an out-of-memory error occurs when compiling. So I thought about saving to a file, but that's slow, taking about 40ms at best and sometimes even 100ms.

Any suggestions on how to save the variable to a file in the fastest and most efficient way?

Thank you very much.

 

[peacemaker]

Is file saving slow by C++ arduino code ?

 

[Gerardo Tenreiro]

Yes, the save-to-file speed is slow on Arduino and B4R.
When I say slow, I mean it's much slower than saving directly to RAM.
I can run the save-to-RAM routine every 20ms, and the ESP32 is capable of solving the problem. However, when saving to a file or FLASH memory, the minimum time is about 50ms, and it still sometimes gets stuck.
I have a file-based test routine that I've attached, and when I call it, I take the micros at the beginning and the micros at the end. The difference is the time taken.

'
' Guarda Movimiento de la Puerta al Abrir
' Los Datos estan en el LOGGER1
' El fichero se Borra y se crea uno Nuevo.
' Los datos se situan del registro 1 a la longuitud de LOGGER1
Sub Guarda_Movimiento_Abrir
Dim Nombre_Fichero As String = "/Mov.DAT" ' Nombre del Fichero
Dim Cnt As Int = Logger1.Length ' Cantidad de Datos
Dim C As Int = 0 ' Auxiliar
Dim Ayuda As String


' Si el Fichero Existe lo Borra.
If FS.Exists(Nombre_Fichero) = True Then ' Si el Fichero Existe
FS.Remove(Nombre_Fichero) ' Borra el Nombre del Fichero
End If
' Abre el Fichero para Añadir Datos
If FS.OpenAppend(Nombre_Fichero) = True Then ' Si puede Abrir el Fichero
For C = 0 To Cnt ' Recorre Todos los Datos de Apertura
' Crea el Dato
Ayuda = JoinStrings(Array As String( _
NumberFormat(Logger1(C).P_Encoder,4,0), _ ' Posicion del Encoder 0 a 8192
NumberFormat(Logger1(C).P_Encoder_F,4,0), _ ' Posicion del Encoder Filtrada 0 a 8192
NumberFormat(Logger1(C).P_Puerta,5,0), _ ' Posicion Puerta 2 Decimales 0 a 10000
NumberFormat(Logger1(C).C_Velocidad,5,0), _ ' Consigna Velocidad 2 Decimales 0 a 10000
NumberFormat(Logger1(C).Intensidad_M,3,0), _ ' Intensidad Motor 2 Decimales 0 a 999
NumberFormat(Logger1(C).Secuencia,3,0), _ ' Secuencia en Proceso 0 a 255
NumberFormat(Logger1(C).CP_Encoder,3,0), _ ' Calidad Comunicacion Encoder P 0 a 100
NumberFormat(Logger1(C).CT_Encoder,3,0), _ ' Calidad Comunicacion Encoder T 0 a 100
NumberFormat(Logger1(C).Modo_M,3,0), _ ' Modo de Maniobra 0 a 255
NumberFormat(Logger1(C).SG1,3,0), _ ' Grupo Señales Digitales 1 0 a 255
NumberFormat(Logger1(C).SG2,3,0), _ ' Grupo Señales Digitales 2 0 a 255
NumberFormat(Logger1(C).SG3,3,0), _ ' Grupo Señales Digitales 3 0 a 255
NumberFormat(Logger1(C).SG4,3,0) _ ' Grupo Señales Digitales 4 0 a 255
))
' Guarda el Dato
FS.Stream.WriteBytes(Ayuda.GetBytes,Ayuda.Length * C,Ayuda.Length) ' Guarda el Dato
Next
End If
End Sub

 

[peacemaker]

If so, the solution is just to use MCU module with extra RAM (PSRAM ?).

 

[Gerardo Tenreiro]

I have over 5,000 units manufactured right now; adding more PSRAM isn't feasible.

All units are working normally. This is a modification that requires more data to be stored, so I'll have to squeeze as much out of the software as possible.

Thank you very much for the help.
I'm sure someone else in this great group has had something similar happen to them and found a solution.
Thank you.

 

[peacemaker]

Any fixed resource is to be out, earlier or later. App is to be optimized, if the hardware is locked.

The actual writing speed to an **SD card using LittleFS on an ESP32** depends on several factors, including:

1. **SD Card Class & Quality** (UHS-I, Class 10 is faster than Class 4)
2. **SPI Bus Speed** (default is often 20 MHz, but some cards handle 40 MHz)
3. **File Size & Write Mode** (single large writes are faster than many small ones)
4. **ESP32 Model** (some have faster SPI interfaces)
5. **LittleFS Overhead** (filesystem metadata updates slow things down)

### **Benchmark Results (Typical Scenarios)**
| Scenario | Speed (KB/s) | Notes |
|-----------|-------------|-------|
| **Sequential Write (Large File, 4KB Blocks)** | 500 - 1200 KB/s | Best-case scenario |
| **Random Small Writes (512B - 1KB)** | 50 - 300 KB/s | Slower due to FS overhead |
| **Default Arduino SD Library (LittleFS)** | 200 - 600 KB/s | Depends on settings |
| **Worst-Case (Fragmented FS, Many Small Files)** | < 100 KB/s | High overhead |

### **How to Maximize Speed**
1. **Use a High-Quality SD Card** (SanDisk Extreme, Samsung EVO perform well)
2. **Increase SPI Clock** (try `SPI.begin(SCK, MISO, MOSI, SS);` with 40 MHz)
3. **Write in Larger Chunks** (4KB+ blocks instead of byte-by-byte)
4. **Disable Safe Writes** (if possible, trade speed for risk of corruption)
5. **Use `FILE_WRITE` Mode** (avoid reopening the file repeatedly)

### **Example Speed Test Code**

#include <LittleFS.h>
#include <SD.h>

#define SD_SCK  18
#define SD_MISO 19
#define SD_MOSI 23
#define SD_SS   5

void setup() {
  Serial.begin(115200);
  SPI.begin(SD_SCK, SD_MISO, SD_MOSI, SD_SS);
 
  if (!SD.begin(SD_SS, SPI, 40000000)) { // Try 40 MHz
    Serial.println("SD Card Mount Failed");
    return;
  }

  File file = SD.open("/speed_test.bin", FILE_WRITE);
  if (!file) {
    Serial.println("Failed to open file");
    return;
  }

  byte buffer[4096]; // 4KB buffer
  memset(buffer, 0xAA, sizeof(buffer));

  uint32_t startTime = millis();
  for (int i = 0; i < 100; i++) { // Write 400KB
    file.write(buffer, sizeof(buffer));
  }
  file.close();
 
  uint32_t duration = millis() - startTime;
  float speed = (400.0 / duration) * 1000; // KB/s
  Serial.printf("Write Speed: %.2f KB/s\n", speed);
}

void loop() {}
```
Run this to test your specific setup.

### **Conclusion**
- **Best Case:** ~1.2 MB/s (with good card & optimized settings)
- **Average Case:** 200-600 KB/s (typical Arduino LittleFS usage)
- **Worst Case:** < 100 KB/s (many small writes, slow card)

For **real-time logging**, expect **50-300 KB/s** if writing small chunks frequently. For **bulk storage**, speeds can reach **500+ KB/s**.

Would you like help optimizing your specific use case?

 

[Gerardo Tenreiro]

Any fixed resource is to be out, earlier or later. App is to be optimized, if the hardware is locked.

I don't understand what you mean by this statement.

Manufacturing of these cards began in 2023, with approximately 250 produced each year. This is now the third year, and there have been some changes to the software that the modified cards must support. In fact, they are frequently updated via OTA.

I don't think I can make the decision to modify the design and consider replacing all the boards in just three years.

Times are moving forward, and improvements and more possibilities are emerging, so I have to keep moving forward, as much as possible, to adapt manufacturing to new demands.

Thank you, and please explain this statement to me.

 

[peacemaker]

I mean that if to update the sketch of any MCU adding functions - the out of memory will be earlier or later. And if to keep it - you have to optimize the sketch, or update the MCU board with bigger memory.

 

[Gerardo Tenreiro]

I mean that if to update the sketch of any MCU adding functions - the out of memory will be earlier or later. And if to keep it - you have to optimize the sketch, or update the MCU board with bigger memory.

Now I understand. Of course, every 5 or 6 years the design is modified and improved.
This application was launched around 2000 with a PIC18. It didn't have Wi-Fi or Bluetooth, it was simpler, but I manufactured around 3,000 units that still work today. No TFT, just a 2x16 LCD.

Then I moved on to a PIC32 with an ESP8266. It was a major evolution of the product, a 2.4" TFT, and many improvements.

The ESP32 was born, and manufacturing costs dropped significantly. The TFT went to 4", external SD memory, and many more possibilities, but this doesn't stop and continues to evolve.

Today, information is power, so what used to be a closed board now has to communicate with everything and more, so the ESP32 is becoming too small, but that doesn't make it useless.

So I have to keep looking for solutions that the current ESP32 supports and can handle.

For now, this routine, which isn't perfect, saves the information in about 12ms to the FLASH memory, so I think the next challenge is to work with the FLASH memory partitions to adapt them to my needs.

'
' Guarda Movimiento de la Puerta al Abrir
' Los Datos estan en el LOGGER1
' El fichero se Borra y se crea uno Nuevo.
' Los datos se situan del registro 1 a la longuitud de LOGGER1
Sub Guarda_Movimiento_Abrir
Dim Nombre_Fichero As String = "/Mov.DAT" ' Nombre del Fichero
Dim Cnt As Int = Logger1.Length ' Cantidad de Datos
Dim C As Int = 0 ' Auxiliar
Dim Ayuda As String


' Si el Fichero Existe lo Borra.
' If FS.Exists(Nombre_Fichero) = True Then ' Si el Fichero Existe
' FS.Remove(Nombre_Fichero) ' Borra el Nombre del Fichero
' End If
' Abre el Fichero para Añadir Datos
If FS.OpenAppend(Nombre_Fichero) = True Then ' Si puede Abrir el Fichero
C = 0
'For C = 0 To Cnt ' Recorre Todos los Datos de Apertura
' Crea el Dato
Ayuda = JoinStrings(Array As String( _
NumberFormat(Logger1(C).P_Encoder,4,0), _ ' Posicion del Encoder 0 a 8192
NumberFormat(Logger1(C).P_Encoder_F,4,0), _ ' Posicion del Encoder Filtrada 0 a 8192
NumberFormat(Logger1(C).P_Puerta,5,0), _ ' Posicion Puerta 2 Decimales 0 a 10000
NumberFormat(Logger1(C).C_Velocidad,5,0), _ ' Consigna Velocidad 2 Decimales 0 a 10000
NumberFormat(Logger1(C).Intensidad_M,3,0), _ ' Intensidad Motor 2 Decimales 0 a 999
NumberFormat(Logger1(C).Secuencia,3,0), _ ' Secuencia en Proceso 0 a 255
NumberFormat(Logger1(C).CP_Encoder,3,0), _ ' Calidad Comunicacion Encoder P 0 a 100
NumberFormat(Logger1(C).CT_Encoder,3,0), _ ' Calidad Comunicacion Encoder T 0 a 100
NumberFormat(Logger1(C).Modo_M,3,0), _ ' Modo de Maniobra 0 a 255
NumberFormat(Logger1(C).SG1,3,0), _ ' Grupo Señales Digitales 1 0 a 255
NumberFormat(Logger1(C).SG2,3,0), _ ' Grupo Señales Digitales 2 0 a 255
NumberFormat(Logger1(C).SG3,3,0), _ ' Grupo Señales Digitales 3 0 a 255
NumberFormat(Logger1(C).SG4,3,0) _ ' Grupo Señales Digitales 4 0 a 255
))
' Guarda el Dato
FS.Stream.WriteBytes(Ayuda.GetBytes,Ayuda.Length * C,Ayuda.Length) ' Guarda el Dato
'Next
FS.Close
End If
End Sub

I found this page where there's a partition calculator, but I'm still not sure how to make it work:

ESP32 Partition Builder

thelastoutpostworkshop.github.io

Thank you very much.

 

[candide]

normally we have 520kb SRAM in esp32, it is a lot!

partition change will modify flash partition but nothing more and it is available in B4R,

it should be interesting to check how memory is available in arduino side when it is short in B4R side.

 

[peacemaker]

Much RAM is for RTOS.

### General Program Structure of ESP32 Controller When Programming via Arduino
When programming the ESP32 in the Arduino IDE, a **two-part structure** is used, similar to classic Arduino sketches:

1. **`setup()`** – Runs once at startup.
- Initializes peripherals (UART, GPIO, Wi-Fi, Bluetooth, etc.).
- Configures initial settings.

2. **`loop()`** – Runs indefinitely in a loop.
- Contains the main program logic (reading sensors, controlling outputs, communication, etc.).

#### Additional Elements:
- **Global variables and interrupts** – Declared outside `setup()` and `loop()`.
- **Libraries** – Included via `#include` (e.g., for Wi-Fi or display control).
- **RTOS (FreeRTOS)** – Used *by default*, but the Arduino wrapper hides it from the user.

---

### Is RTOS Always Present?
**Yes**, the ESP32 always runs on **FreeRTOS**, even in the Arduino environment. However:
- The Arduino API *abstracts* RTOS, providing a simple `setup/loop` interface.
- You can use **native FreeRTOS functions** (task creation, queues, semaphores), but this requires deeper knowledge and may conflict with Arduino code.

Example of creating a task in an Arduino sketch:

#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
void myTask(void *pvParam) {
  while (1) {
    digitalWrite(2, !digitalRead(2)); // Blink an LED
    vTaskDelay(500 / portTICK_PERIOD_MS);
  }
}

void setup() {
  pinMode(2, OUTPUT);
  xTaskCreate(myTask, "LED Task", 10000, NULL, 1, NULL);
}

void loop() {}  // Empty if tasks are managed by RTOS

---

### How Many Resources Are Available for an Arduino Program?
The ESP32 has:
- **Dual-core Xtensa LX6** (clock speed 80–240 MHz).
- **RAM**:
- ~320 KB free (out of 520 KB), as some is used by:
- Wi-Fi/Bluetooth stack (~100 KB).
- Drivers and RTOS.
- **PSRAM** (if available, e.g., in ESP32-WROVER) – Additional 4–8 MB (but requires manual management).
- **Flash memory**:
- 4–16 MB (depends on the model).
- ~1.2 MB available for programs and data (e.g., for ESP32-WROOM-32 with 4 MB Flash).

#### Limitations:
1. **Recursion depth/task stack** – Default ~8 KB per task (configurable).
2. **Heap (dynamic memory)** – Fragmentation possible with frequent `malloc/free`.
3. **Interrupts** – Time-critical code should be handled in RTOS tasks.

#### Checking Free Memory:

void setup() {
  Serial.begin(115200);
  Serial.printf("Free heap: %d bytes\n", ESP.getFreeHeap());
  Serial.printf("Flash size: %d bytes\n", ESP.getFlashChipSize());
}

---

### Key Takeaways:
1. **Structure** – Classic Arduino (`setup/loop`), but with RTOS running in the background.
2. **RTOS** – Always active, but Arduino API masks it.
3. **Resources** – Sufficient for most projects, but Wi-Fi/Bluetooth consume significant RAM.
4. **Optimization** – For complex projects, consider switching to ESP-IDF or a hybrid approach (Arduino + FreeRTOS).

For beginners, the Arduino wrapper is convenient, but as projects grow, learning FreeRTOS and ESP-IDF becomes beneficial.

 

[candide]

LOGGER1(n) what it is ? char? string ?

 

[peacemaker]

Public Logger1(420) As D_Logger1

It's a type of those complex data array.

 

[Gerardo Tenreiro]

The ESP32s used are N16, which gives them 16MB of FLASH memory.
The goal is to be able to store the LOGGER1 variable in the FLASH memory in the SPIFFS section, which consists of this structure:

Type D_Logger1 (P_Encoder As Int, _ ' Posicion del Encoder 0 a 8192
P_Encoder_F As Int, _ ' Posicion del Encoder Filtrada 0 a 8192
P_Puerta As Int, _ ' Posicion de la Puerta en % 0 a 10000 -> 0.00% a 100.00
C_Velocidad As Int, _ ' Consigna de Velocidad al Accionamiento 0 a 10000 -> 0.00% a 100.00
Intensidad_M As Int, _ ' Intensidad Consumida por el Motor 0 a 999 -> 0A a 9.99
Secuencia As Byte, _ ' Numero de Secuencia 0 a 255 ->
CP_Encoder As Byte, _ ' Calidad de la Comunicacion con el Encoder en el ultimo Segundo 0 a 100
CT_Encoder As Byte, _ ' Calidad de la Comunicacion con el Encoder Total 0 a 100
Modo_M As Byte, _ ' Indica el Modo de maniobra de la Puerta 0 a 255
SG1 As Byte, _ ' Grupo Señales 1 0 a 255
SG2 As Byte, _ ' Grupo Señales 2 0 a 255
SG3 As Byte, _ ' Grupo Señales 3 0 a 255
SG4 As Byte _ ' Grupo Señales 4 0 a 255
)

Public Logger1(65) As D_Logger1 ' Datos del Logger de movimientos de la puerta

The problem I have is the time it takes to store the information in FLASH memory. The LITTLEFS library takes approximately 40ms to open the file, save the information, and close it again. If I keep the file open permanently, the time is reduced to about 20ms. Either way, the ESP32 doesn't have this amount of time to save the information.

This processor has two RS485 serial communications with four slaves on each channel, three SPI and eight i2C, a 4" TFT, Wi-Fi, Bluetooth, and readings from two TTL encoders, so it's very limited.

Each program cycle executes in 0.2 ms, and it has to respond to many things within this cycle time, but not everything, as that would be impossible. If I spend a lot of time saving to a file, I'm neglecting other elements, and this can cause problems. There are fast-moving elements that have to stop at their exact points.

There are times when I have time to process information; when the elements aren't moving, reaction time isn't important. These are the times when I send information via Bluetooth or Wi-Fi and want to store the information in a file.

This is the problem in a nutshell.

Thank you very much.

 

[peacemaker]

This processor has two RS485 serial communications with four slaves on each channel, three SPI and eight i2C, a 4" TFT, Wi-Fi, Bluetooth, and readings from two TTL encoders, so it's very limited.

And now extra task to start into the space and save all the info from the galaxy into small poor ESP32
All is non-endless.

 

[Gerardo Tenreiro]

It's not the end of peace; the board is this one. I'm attaching a couple of photos of two boards, one on the test bench and the other new.
The ESP32 program is highly fragmented, so depending on the needs, the ESP32 dedicates more time to necessary tasks while others stop executing.
The ESP32 has a very large, but limited, computing capacity, so resources must be optimized according to needs.
It's of little use stopping 1,000 times per second in a temperature that changes every 20 seconds. However, requesting information from the slaves that are in process if it's very important or reading the digital input and output expanders via I2C, which you can do every 50ms, doesn't have to be done 1,000 times per second either.

Following this philosophy, you can carry out large projects with the ESP32 or similar micros. It's also clear that you can opt for a more powerful 8-core micro and dedicate one core to each task, but it seems much more inefficient to me.
Software is cheaper than hardware at these levels.
These improvements pay off over the years, and end customers appreciate it.

Thank you very much for your collaboration and help.

 

[peacemaker]

But now you have to find time for extra saving the data. It means to make some existing tasks more rare. As the hardware cannot be changed.

 

[Gerardo Tenreiro]

As I mentioned, there are times when the process isn't moving. The movement time is approximately 2 seconds in one direction and 3 seconds in the opposite direction, with a time between both movements of no less than 0.5 seconds.
The idea is:
During the first movement, save the information in the variables.
When the first movement is finished, transfer the data from the variables to the file system in FLASH memory, leaving the variables set to ZERO for the next process.
During the second movement, store the data in the variables.
At the end of the second movement, transfer the data from the variables to the file, thus creating the complete movement file.

While I'm not moving, I also have to attend to communications and other elements, but the response time isn't as critical.

I couldn't upload the card images because they indicate they take up a lot of space.

Thanks.