Figuring out an old "dos style" database table

MarrowZero

Member
Licensed User
Hello peoples,

I've got a small application that stores the application data in some files, they seem to be in a group of three (I assume DAT is DATA, NDX some INDEX, and DIA? No idea) This type of format predates my computer experince, so I wonder if there's any older (cough; I mean "experienced") people out there that could help identify the format and most importantly how I might use B4J to read and hopefully write to the file structure. I've attached a sample of the files (zipped).


1623352856229.png



The best idea I have so far is from analsying the program executable with TRID (https://mark0.net/soft-trid-e.html) RESULTS:

83.3% (.EXE) Win32 Executable Borland Delphi 5 (451463/56/28)
7.9% (.EXE) InstallShield setup (43053/19/16)
2.6% (.EXE) Win32 Executable Delphi generic (14182/79/4)
1.9% (.EXE) Win64 Executable (generic) (10523/12/4)
1.8% (.EXE) DOS Borland compiled Executable (generic) (10000/1/2)


Thanks for any help.

(I've been reading the forum and playing with B4J for about a year.... Loving it as a replacment for VB6 and MS Access! Thanks Erel)

--Paul
 

Attachments

  • oldfilessample.zip
    384.6 KB · Views: 162

Peter Simpson

Expert
Licensed User
Longtime User
I doubt very much that you'll be able to read these files. Just because a file extension ends in .DIA or .DAT does not mean that they are created and saved in the same format all the time.

Let's take your .DIA extension. That file could be saved as good old fashioned plain text format or has been compressed (in multiple ways), encrypted (in multiple ways) or dare I say it all of the above in that one file, or none of the above which I doubt very much.

I personally believe that you're going to run into huge difficulties figuring out the file formats above, let alone reading the data out of them, but I also could be wrong about that.

What is the name of the program that these files belong too?
 

Sandman

Expert
Licensed User
Longtime User
I doubt it'll help anything, but here's how the Linux command file sees the files:
B4X:
> file *
oldfilessample.zip: Zip archive data, at least v2.0 to extract
S_RESO.DAT:         DOS executable (block device driverC)
S_RESO.DIA:         data
S_RESO.NDX:         DOS executable (block device driver \307 )
 

agraham

Expert
Licensed User
Longtime User
They are not, unfortunately dBase files which is the first DOS database that comes to mind.

The important file is the .DAT file that looks like it is a database record file with a 0x166 byte header and subsequent 0x143 byte long records that appear to be road repair items (including Dumper trucks!) The item description is a string but the remainder of the record seems to be binary. It you have the application that creates these files, or can read them, perhaps we can get an idea of the fields which are in each record and reverse engineer some of the data.

The .NDX file is probably a structured index for for rapidly accessing the database records. It's contents will be relevant only to the operation of the original application and will contain nothing useful.

I can't guess what the .DIA file is but it will also not be relevant to accessing the database data.

EDIT: Added the hex prefix to the byte contents. There is also an 0x120 byte trailing record that appears to be a duplicate of the preceding full size record which also appears to be a duplicate so the trailing records may be left over garbage.
 
Last edited:

agraham

Expert
Licensed User
Longtime User
Each record seems to start (or the previous record ends) with a byte value of 0x4F (letter O) following by a byte count for the item identification string followed by an ANSII encoded string. It looks to me like each field is probably a fixed length field with string fields having a length indicator prefix. I can't find anything in those files which looks like a field definition table so maybe that knowledge is hard coded into the application. Knowing what the fields are and having some sample record contents should make it possible to recreate the record layout and read the data.
 
Last edited:

agraham

Expert
Licensed User
Longtime User

Magma

Expert
Licensed User
Longtime User
can be also a custom made "database"... ..actually will be custom made "table" (those days every table was different file...)

remember Open "file.dat" For Random As #num Len = Len(TYPE)
that will more difficult to be read...
as you must know the TYPE and also all the possible variables... that may be string,byte,currency,integer,long,double...array with any type..

Also ndx and dia could be then Index files for better and faster search results... you never know how a custom "maker" programmer thinks ?
 

Star-Dust

Expert
Licensed User
Longtime User
can files be damaged and therefore any identification attempt would be useless?
 

Magma

Expert
Licensed User
Longtime User
I have to say that I opened files...

.dat file seems to be the data (table and not damaged)
.dia file seems to be just like the number of latest record and if file .dat is using
.ndx file seems to be index of records...

I think that if you open with notepad ++ the .dat file ... count the bytes (every character) before the ...180 Excavatorer panelne)quipment... (this is a field of a record in string)
or better if you have a screenshot of program or a printed page with known records....

seperate the records when the same string of counted characters...
count the characters (bytes) if for example are 4 ---> what is 4-byte numbers ?? integer,long
...if 8-byte... what can be ?
nulls can be just strings...

You must make tries...

or if you have the old dos app ----> run it into DOSBOX.... somewhere will be export (as ascii) or print the data will be easier to ocr... :)
 
Last edited:

Magma

Expert
Licensed User
Longtime User
Start from seperating the records.... will be a nice division... every record will be the same size :) ... The strings seems to be showed without decryption... so is just numbers the prob...
 
Last edited:

MarrowZero

Member
Licensed User
Thankyou for all your comments, here are some answers below, sorry i could not respond yesterday work was a crazy day.

What is the name of the program that these files belong too?
@Peter Simpson
I don't think I can say about the program or give full access to the data as there is commerically sensitive data.
The purpose of this discovery is to figure out how this program should be replaced, I'm keen to see a new program build in B4J.
I was inititally wondering what data was in those files, what structure, how do these older dos/binary file work and how they could be read with B4J (personal techincal interest)

It you have the application that creates these files, or can read them, perhaps we can get an idea of the fields which are in each record and reverse engineer some of the data.
@agraham Yes I have the application and can make it run. It seems you are excited by the dumper trucks, so I thought you might like this screenshot

1623489918999.png



They are mostly likey DBase files (DOS), I suspect dbase (I-III). / Foxpro
@Jmu5667 I was wondering if using "Borland Delphi 5" then what would be the goto method for storing data? Would someone writing in Delphi use dBase or Foxpro ?

actually will be custom made "table" (those days every table was different file...)
@Magma oh no! This is going to be hard work right?!?! (could be fun though?) Here are some Excavators for you :)

1623490617835.png



Would anybody be kind enough to offer an example of how the DAT file could be read byte for byte with B4J ?

Thanks to all of you!
--Paul
 

MarrowZero

Member
Licensed User
I found two methods (by searching forum) which allow the file to read - Now I just need to figure out the structure


Two methods to read bytes from file:
Private Sub Button1_Click
    
    Dim b() As Byte = Bit.InputStreamToBytes(File.OpenInput(File.DirAssets, "R_RESO.DAT"))
    Log(BytesToString(b, 16508,32, "utf8")) 'Read 15 bytes starting from 10
    ' RESULT     180 Excavatorer panelne)quipment
    
End Sub


Private Sub Button2_Click
    
    Dim b() As Byte = File.ReadBytes(File.DirAssets, "R_RESO.DAT")
    Log(BytesToString(b, 16508,32, "utf8")) 'Read 15 bytes starting from 10
    ' RESULT     180 Excavatorer panelne)quipment
    
End Sub
 

MarrowZero

Member
Licensed User
I've had some fun and hand-picked the locations to recreate the first two columns of the PLANT/EXCAVATORS results.


Hand picked byte locations:
Private Sub Button1_Click
    
    Dim CS As String = "ASCII"
    Dim b() As Byte = Bit.InputStreamToBytes(File.OpenInput(File.DirAssets, "R_RESO.DAT"))
    
    Log("=1=============")
    'Log(BytesToString(b, 16506,1, CS)) '4F         "O"
    Log(BytesToString(b, 16508,13, CS)) '            "180 Excavator"   
    Log(BytesToString(b, 16590,4, CS)) '             "Hour"
    Log("=2============")
    'Log(BytesToString(b, 597260,1, CS)) '4F         "O"
    Log(BytesToString(b, 597262,23, CS)) '            "360 Excavator (8 Tonne)"
    Log(BytesToString(b, 597344,4, CS)) '             "Hour"
    Log("=3============")
    'Log(BytesToString(b, 2941,1, CS)) '4F             "O"
    Log(BytesToString(b, 2942,24, CS)) '            "360 Excavator (13 Tonne)"
    Log(BytesToString(b, 3024,4, CS)) '             "Hour"
    Log("=4============")
    'Log(BytesToString(b, 10692,1, CS)) '4F         "O"
    Log(BytesToString(b, 10694,24, CS)) '            "360 Excavator (20 Tonne)"
    Log(BytesToString(b, 10776,4, CS)) '             "Hour"
    Log("=5============")
    'Log(BytesToString(b, 192541,1, CS)) '4F         "O"
    Log(BytesToString(b, 192543,24, CS)) '            "360 Excavator (40 Tonne)"
    Log(BytesToString(b, 192625,4, CS)) '             "Hour"
    Log("=6============")
    'Log(BytesToString(b, 573681,1, CS)) '4F         "O"
    Log(BytesToString(b, 573683,33, CS)) '            "360 Excavator (45 Tonne) Operated"
    Log(BytesToString(b, 573765,4, CS)) '             "Hour"
    Log("=7============")
    'Log(BytesToString(b, 574004,1, CS)) '4F         "O"
    Log(BytesToString(b, 574006,26, CS)) '            "Transport costs (45 Tonne)"
    Log(BytesToString(b, 574088,4, CS)) '             "Item"
    Log("=8============")
    'Log(BytesToString(b, 442543,1, CS)) '4F         "O"
    Log(BytesToString(b, 442545,24, CS)) '            "Mini Excavator (3 Tonne)"
    Log(BytesToString(b, 442627,4, CS)) '             "Item"
    Log("=9============")
    'Log(BytesToString(b, 54620,1, CS)) '4F         "O"
    Log(BytesToString(b, 54622,8, CS)) '            "Tractair"
    Log(BytesToString(b, 54704,4, CS)) '             "Hour
    Log("=10===========")
    'Log(BytesToString(b, 74969,1, CS)) '4F         "O"
    Log(BytesToString(b, 74971,14, CS)) '            "Loading shovel"
    Log(BytesToString(b, 75053,4, CS)) '             "Hour"
    Log("=11===========")
    'Log(BytesToString(b, 75292,1, CS)) '4F         "O"
    Log(BytesToString(b, 75294,15, CS)) '            "Transport costs"
    Log(BytesToString(b, 75376,4, CS)) '             "Trip"
    Log("=12===========")
    'Log(BytesToString(b, 75615,1, CS)) '4F         "O"
    Log(BytesToString(b, 75617,10, CS)) '            "Dump truck"
    Log(BytesToString(b, 75699,4, CS)) '             "Hour"
    
End Sub
 

josejad

Expert
Licensed User
Longtime User
Obviously you'll have checked the program has not any menu to export data as cvs, text or similar...
 

MarrowZero

Member
Licensed User
Obviously you'll have checked the program has not any menu to export data as cvs, text or similar...
Indeed I have looked for export functions, sadly there is nothing useful. Even printing and OCR might not be good as those printed reports are for the summary of infomation, not the table data.
 
Top