B4R Question ESP32 flash memory - max volume for sensor results

[peacemaker]

HI, All

If we have a sensor based on ESP32 - and we need to make a storage for the measured data - how maximally to use the available ESP32 flash memory to fill it in ?
I see we have good lib https://www.b4x.com/android/forum/threads/rarduinonvs-nvs-library-for-esp32.149307/, but ... how about the max capacity and deleting the oldest records to store the latest ?

 

[hatzisn]

Why would you need the internal esp32 board eeprom or SD card while you could post a Json in an outside server and store there the measured data?

 

[hatzisn]

Why would you need the internal esp32 board eeprom or SD card while you could post a Json in an outside server and store there the measured data?

Or even better store them in a Google sheet...

 

[peacemaker]

Sending results immediately is a normal way, when connection is available.
But also there is a task to store the measurements at offline mode.
Adding a SD-card, IMHO, is overkill...

So, any way to make the storage within all free flash ?

 

[XorAndOr]

Adding a SD-card, IMHO, it overkill...

the advantage however is that you can store a large amount of data
and you don't have to worry about the flash memory if it is full
and... you can remove the micro SD and read it with your PC too

 

[hatzisn]

If I were you I would use global store by @Erel to store in memory a JSON when I am not on-line and post it directly when I am.

 

[peacemaker]

Say, measurements are 10 times per second as RecordIndex, one record is, say, fixed RecordBuffer = 512 bytes like "parameter1=value2,parameter2=value2,parameter3=value3.....", for sure within the buffer.

We have to know the free available memory TotalFree, possible records qty MaxRecordsQty = TotalFree / RecordBuffer.
If RecordIndex +1 > MaxRecordsQty then "delete the oldest and save the latest"...

Do you think the Global store can help here ? For maybe thousands of records... MaxRecordsQty is variable at any MCU reboot ?

 

[hatzisn]

Say, measurements are 10 times per second as RecordIndex, one record is, say, fixed RecordBuffer = 512 bytes like "parameter1=value2,parameter2=value2,parameter3=value3.....", for sure within the buffer.

We have to know the free available memory TotalFree, possible records qty MaxRecordsQty = TotalFree / RecordBuffer.
If RecordIndex +1 > MaxRecordsQty then "delete the oldest and save the latest"...

Do you think the Global store can help here ? For maybe thousands of records... MaxRecordsQty is variable at any MCU reboot ?

Actually no, I do not believe it could be done at least in this way you present it. You could of course make it a little simpler. F.e. store a List of lists in Global store with [[value1-1,value1-2,value1-3.....], [value2-1,value2-2,value2-3.....],.......] and post a JSON with this List of lists to a web server. Then clear globalstore and restart. In a database it would be a piece of cake to delete the unneeded records by filtering with ID or Date. The problem is that I do not believe that the server will respond in less than 1/10 sec. You could try with an interrupt maybe for the measurements but the ball may be lost with all these. I cannot think of a way to do it in eeprom which is not suggested as it is small and will burn out eventually and also not in an SD card because I think FAT32 must be used which supports files up to 2 GB. A workaround in this, would be to write different files for a bunch of data in SD and keep in a file a registry with {length of filename}-{filename}-{deleted 0/1} in each line. Then you just delete the corresponding file and set in registry file the deleted to 1 for this file while for all the available files the deleted will be 0. In restart recreate the registry file with only the not deleted files.

 

[peacemaker]

Just imagine that no any servers at all ?, forget about them, the task is fully offline here.
Yes, fast measurement, and full logging. And later the user will touch the device and download the whole latest storage to view on PC as charts.
No care about the oldest records that were deleted when storage is full, the important just recent, but all and fastly measured.

It seems to me this is a normal situation for, say, video surveillance: all must be saved (i mean all video frames, or all measured sensor records...), but when it's needed - we can see any moment among the latest X hours, days... detailed. Earlier were deleted already, it's normal.

It looks like the file system idea is OK: store 100...1000 records and save them into "file0001.csv" file......
The free space is controllable so the oldest file "file0001.csv" can be deleted when free space is near some AlmostZERO_value.