1,000,000 records = 500,000 bits ???
-
Welcome to B4X forum!
B4X is a set of simple and powerful cross platform RAD tools:
- B4A (free) - Android development
- B4J (free) - Desktop and Server development
- B4i - iOS development
- B4R (free) - Arduino, ESP8266 and ESP32 development
B4XDatabase
- Thread starter sfsameer
- Start date
-
- Tags
- b4xcode.com b4xdatabase b4xdb
- Similar Threads Similar Threads
- Status
I said bits replying to your previous post which was (bytes).
Again :
Once we have concluded the first release, we will release more information on how everything works in the background, why it's very small, why it's very fast, and other information that created a confusion here.
I had to look up what "clowning" meant. Seems it means "to bring ridicule upon someone". (ref)
When I hear fascinating claims without any real explanation I tend to get curious. This curiousity sometimes leads to questions, so I ask them. I'm sorry you feel ridiculed when I'm asking honest questions to better understand your claims.
As for the assumptions... I imagine you're talking about me assuming you have test files for your tables? Well, it did just make sense to me. How else would you be able to make those fantastic claims on how much disk space was used? Surely you didn't just pull the numbers out of thin air.
Again :
Once we have concluded the first release, we will release more information on how everything works in the background, why it's very small, why it's very fast, and other information that created a confusion here.
Now, that's just plain wrong - that's not what he said.
You said
Half byte per row. What is each row made of... air?
and sfasameer replied
I think it's obvious that he commented on your "half byte" part. So on average 4 bits per row.
If the data in the tables is very similar (Row 1, Row 2, Row 3, ...) then compression of 1.000.000 records to 500kb or less is definitely possible. However, with 'real life' data, where records are nothing alike, you will not get such a compression. And that is normal as it is how compression algorithms work: they 'look' for similarities in the data so they can replace large data parts that are used many times with a much shorter one, referring to the big one, written once.
I'm pretty happy with just the statement that data is compressed and that is enough for me. Some data compression will go better that others, depending on the type of data and that is normal.
For me this is even just a 'nice-to-have' feature. I'm more excited by the speed part to query the database of the project. I myself did write a 'specialized' database engine about 20 years ago to do fuzzy matching of sentences for a translation tool. It could find 'similar' sentences in the database extremely quick and rank them on similarity and even mark the differences (finding sentences e.g. 80% alike in a multiple millions database took a couple of milliseconds on an average 486 PC).
Such a fuzzy search was for example:
And it could find:
Weight was given to differences and words the machine could replace themself in the translation was also done, but that would lead us to far.
The point is, the database was extremely fast to query due the design of the database engine and because it had a very specialized task. So I'm not surprised sfameer would be able to write a more 'general' DB engine nowadays that can match those old specialized ones, given the hardware we have now at our disposal.
It sounds like a great project and I'm eager to play with it a bit. I will be honest that it will have to go through many months of real life testing before I will replace the existing proven database engines by it, but it certainly looks promising and in the the future it may be a candidate for some of my projects.
So, great job guys!
Alwaysbusy
I'm pretty happy with just the statement that data is compressed and that is enough for me. Some data compression will go better that others, depending on the type of data and that is normal.
For me this is even just a 'nice-to-have' feature. I'm more excited by the speed part to query the database of the project. I myself did write a 'specialized' database engine about 20 years ago to do fuzzy matching of sentences for a translation tool. It could find 'similar' sentences in the database extremely quick and rank them on similarity and even mark the differences (finding sentences e.g. 80% alike in a multiple millions database took a couple of milliseconds on an average 486 PC).
Such a fuzzy search was for example:
B4X:
We have to move the blue box into the back room.And it could find:
B4X:
We have to move the red box into the living room.
We have to move the yellow boxes into the back room.
He has to move the blue box into the bed room now.
Yesterday, she did move the blue boxes into the back room.Weight was given to differences and words the machine could replace themself in the translation was also done, but that would lead us to far.
The point is, the database was extremely fast to query due the design of the database engine and because it had a very specialized task. So I'm not surprised sfameer would be able to write a more 'general' DB engine nowadays that can match those old specialized ones, given the hardware we have now at our disposal.
It sounds like a great project and I'm eager to play with it a bit. I will be honest that it will have to go through many months of real life testing before I will replace the existing proven database engines by it, but it certainly looks promising and in the the future it may be a candidate for some of my projects.
So, great job guys!
Alwaysbusy
Last edited:
Bringing discussion to fruitful point.
Saif,
Can we replace the sqlite database in our B4A with your database, since it is small, we will benefit ?
I may have missed the point if you already mentioned it.
Saif,
Can we replace the sqlite database in our B4A with your database, since it is small, we will benefit ?
I may have missed the point if you already mentioned it.
For the nay sayers, if it works, fantastic. If it doesn't, at least it will pave the way for more such thoughts and attempts.
If we were to be satisfied with what we have, we would still have Hotmail with 2mb (mb not gb), and we would pay for 50mb storage.
Best of luck Saif
If we were to be satisfied with what we have, we would still have Hotmail with 2mb (mb not gb), and we would pay for 50mb storage.
Best of luck Saif
HI - sounds really great 
For me I run MySQL and SQLite on B4J (server side) and SQLite on Android (client side).
1) TYPES: Can you take either directly or serialised maps and lists from B4X? It would be great to be able to simply store and re-access maps and lists without having to unpack them on the fly - even better if you could reference the map keys as part pf a query.
2) Mirroring and Replication: I need to be able to keep a hot standby for my servers databases as part of a security and safety system. Are there tools to manage that process or is that up to the user to code. (cf MySQL Enterprise)
Whatever I will be taking this for a trial run as were are growing the real time database very rapidly now with installations across the whole of Australia.
For me I run MySQL and SQLite on B4J (server side) and SQLite on Android (client side).
1) TYPES: Can you take either directly or serialised maps and lists from B4X? It would be great to be able to simply store and re-access maps and lists without having to unpack them on the fly - even better if you could reference the map keys as part pf a query.
2) Mirroring and Replication: I need to be able to keep a hot standby for my servers databases as part of a security and safety system. Are there tools to manage that process or is that up to the user to code. (cf MySQL Enterprise)
Whatever I will be taking this for a trial run as were are growing the real time database very rapidly now with installations across the whole of Australia.
Referring to the post by "HAH' earlier in this thread and the subsequent flames he received, I think English is not his native language and the meaning of his message was probably misunderstood.
When Saif went dark for many months after he was a victim of a ransomware attack there was many posts in this forum about the fact that several of the apps/code that Saif had previously sold were incomplete or required extra work by the buyers to get the code to work. I think the meaning of HAH's message was probably that Saif should maybe concentrate a bit more on polishing his existing products and offering support rather than embarking on a new project and leaving his previous customers in the dark.
That was my understanding anyway
When Saif went dark for many months after he was a victim of a ransomware attack there was many posts in this forum about the fact that several of the apps/code that Saif had previously sold were incomplete or required extra work by the buyers to get the code to work. I think the meaning of HAH's message was probably that Saif should maybe concentrate a bit more on polishing his existing products and offering support rather than embarking on a new project and leaving his previous customers in the dark.
That was my understanding anyway
There are many features that i can't discuses until we release the project, but :
1- Unlike SQLite, B4XDatabase can be used/modified by multiple users at the same time, where SQLite doesn't support concurrency (no more than one user can be writing to the database at the same time)
2- It can be accessed remotely and locally
3- In MS SQL the database growth is managed by MB and in our case it's managed by KB
4- A lot faster than SQLite
5- We have added 2 new features which is : Manual Commit and Automatic Commit
6- Data monitoring in real-time
7- Because it's designed and developed to be used in B4X products so we are adding a new field to the database which is (Objects) meaning you can store labels, objects, text, maps, lists, etc... and this can be used in many ways for example designing app templates where you draw the whole activity/form from the database.
There are many other features which make this database engine great.
Very cool
Database Speed and Capacity are key to a great database.
My only other wish is to have a database that does not collapse, or slow down dramatically, just because you added or deleted too many records.
Reorganisation of a database periodically is the making or breaking of the database.
How do you get around Reorganisation of a Database where there are always users connected to it. (Either intentionally or unintentionally).
I have an application that adds and removes over 500,000 records a day - I am using a Microsoft Access database which dies, and doubles in size, if I dont reorganise
the database immediately after the process. The catch is, you can't reorganise an Access database if users are attached to it.
Currently I do prefer using MS SQL Server.
My only other wish is to have a database that does not collapse, or slow down dramatically, just because you added or deleted too many records.
Reorganisation of a database periodically is the making or breaking of the database.
How do you get around Reorganisation of a Database where there are always users connected to it. (Either intentionally or unintentionally).
I have an application that adds and removes over 500,000 records a day - I am using a Microsoft Access database which dies, and doubles in size, if I dont reorganise
the database immediately after the process. The catch is, you can't reorganise an Access database if users are attached to it.
Currently I do prefer using MS SQL Server.
Currently we have many ways to prevent data loss, decrease in speed and abnormal size increase, two of them are :
1- We have an automatic data change commits, meaning once an insert, update, delete occurs, the database engine will save the changes in a temp database (happens in the background) then commits those changes to the internal database
The above process will make sure the data has been inserted correctly and prevents any data loss
Q: What happens if let's say 1000 users made transactions within that millisecond?
A: We have implemented a Queue system on a single database, MS SQL has this feature but their problem is a database lock until all the transactions are committed, we solved it by creating the above solution where all the data are being saved into a temp database while in the same time the data are being saved in the internal database, below is a simple graph to show the above statement :
Q: Would that effect the performance?
A: We have tested inserting 1 million record in the database and the automatic commit took exactly 1 second, so it will never effect the performance.
2- As mentioned in the Post #72, our focus was on speed and size, many people discussed in this thread that the database size doesn't matter and we shouldn't focus too much on it, but there are cases like yours where there are millions of records inserted, updated, deleted on a daily basis and that would take a lot of disk space, CPU usage, RAM Usage and useless temp files.
So unlike other database engines like MS SQL where their database growth is managed by MB, B4XDatabase growth is managed is managed by KB, that means less disk space, less CPU usage and less RAM Usage
Thank you Guys, that sounds good here are my comments and suggestions:
- I think most of us will work on PC with Windows Platform thereforw a Windows OS Desktop App is a must have. It Should be good if the App could be started inside B4X GUI form a Menu Item.
- Cloud may be come important in future but is not yet in European area.
Next the new database might have some extra features partly relaized with in B4X Forum like dbutils or dbgrid a.s.o. With this new database there is still the chance to put all this separate libs together under the engine of the datatabase. For developers not only storing of data is an important question but handling the data and presenting the data as well. Here you know that current B4X views are on level early access and far away from those fine functionalty like a database ms access has. Think about date handling, editable grid, reports a.s.o.
At least I think the forum members including me are interesting to be informed what about your release plan.
Stay well
Happy Christmas and Successful New YSear
Guenter
The release date will be announced on 26-12-2021
Marry Christmas and happy new year
It has any sync future with other databases?
Like Mysql or MSSql
Like Mysql or MSSql
Currently no, but we will look into it
- Status