B4R Question ESP32 flash memory - max volume for sensor results

Sort by date Sort by votes
hatzisn

hatzisn

Expert
Licensed User
Longtime User
Why would you need the internal esp32 board eeprom or SD card while you could post a Json in an outside server and store there the measured data?
 
Upvote 0
hatzisn

hatzisn

Expert
Licensed User
Longtime User
hatzisn said:
Why would you need the internal esp32 board eeprom or SD card while you could post a Json in an outside server and store there the measured data?
Click to expand...

Or even better store them in a Google sheet...
 
Upvote 0
peacemaker

peacemaker

Expert
Licensed User
Longtime User
Sending results immediately is a normal way, when connection is available.
But also there is a task to store the measurements at offline mode.
Adding a SD-card, IMHO, is overkill...

So, any way to make the storage within all free flash ?
 
Last edited:
Upvote 0
hatzisn

hatzisn

Expert
Licensed User
Longtime User
If I were you I would use global store by @Erel to store in memory a JSON when I am not on-line and post it directly when I am.
 
Upvote 0
peacemaker

peacemaker

Expert
Licensed User
Longtime User
Say, measurements are 10 times per second as RecordIndex, one record is, say, fixed RecordBuffer = 512 bytes like "parameter1=value2,parameter2=value2,parameter3=value3.....", for sure within the buffer.

We have to know the free available memory TotalFree, possible records qty MaxRecordsQty = TotalFree / RecordBuffer.
If RecordIndex +1 > MaxRecordsQty then "delete the oldest and save the latest"...

Do you think the Global store can help here ? For maybe thousands of records... MaxRecordsQty is variable at any MCU reboot ?
 
Upvote 0
hatzisn

hatzisn

Expert
Licensed User
Longtime User
peacemaker said:
Say, measurements are 10 times per second as RecordIndex, one record is, say, fixed RecordBuffer = 512 bytes like "parameter1=value2,parameter2=value2,parameter3=value3.....", for sure within the buffer.

We have to know the free available memory TotalFree, possible records qty MaxRecordsQty = TotalFree / RecordBuffer.
If RecordIndex +1 > MaxRecordsQty then "delete the oldest and save the latest"...

Do you think the Global store can help here ? For maybe thousands of records... MaxRecordsQty is variable at any MCU reboot ?
Click to expand...

Actually no, I do not believe it could be done at least in this way you present it. You could of course make it a little simpler. F.e. store a List of lists in Global store with [[value1-1,value1-2,value1-3.....], [value2-1,value2-2,value2-3.....],.......] and post a JSON with this List of lists to a web server. Then clear globalstore and restart. In a database it would be a piece of cake to delete the unneeded records by filtering with ID or Date. The problem is that I do not believe that the server will respond in less than 1/10 sec. You could try with an interrupt maybe for the measurements but the ball may be lost with all these. I cannot think of a way to do it in eeprom which is not suggested as it is small and will burn out eventually and also not in an SD card because I think FAT32 must be used which supports files up to 2 GB. A workaround in this, would be to write different files for a bunch of data in SD and keep in a file a registry with {length of filename}-{filename}-{deleted 0/1} in each line. Then you just delete the corresponding file and set in registry file the deleted to 1 for this file while for all the available files the deleted will be 0. In restart recreate the registry file with only the not deleted files.
 
Upvote 0
peacemaker

peacemaker

Expert
Licensed User
Longtime User
Just imagine that no any servers at all ?, forget about them, the task is fully offline here.
Yes, fast measurement, and full logging. And later the user will touch the device and download the whole latest storage to view on PC as charts.
No care about the oldest records that were deleted when storage is full, the important just recent, but all and fastly measured.

It seems to me this is a normal situation for, say, video surveillance: all must be saved (i mean all video frames, or all measured sensor records...), but when it's needed - we can see any moment among the latest X hours, days... detailed. Earlier were deleted already, it's normal.

It looks like the file system idea is OK: store 100...1000 records and save them into "file0001.csv" file......
The free space is controllable so the oldest file "file0001.csv" can be deleted when free space is near some AlmostZERO_value.
 
Last edited:
Upvote 0
You must log in or register to reply here.
Top