B4J Question Calculating CRC16-CCITT FCS to match C implementation [solved]

[Didier9]

I need to calculate a CRC on a block of data that matches a CRC calculation done in an 8 bit microcontroller (8051 using Keil C51 compiler).
The CRC is calculated on a block of 512 bytes according to the Get_Buf_CRC() function below:

uint Get_Buf_CRC( uchar *ptr, uint numbytes ){
    uint i, CRC;
    CRC = 0x0000;
    // Process each byte in the buffer into the running CRC
    for( i = 0; i < numbytes; i++ ){
        CRC = Update_CRC( CRC, *ptr++ );
    }
    return( CRC );
}

uint Update_CRC( uint crc, uchar newbyte ){
    uchar i; // loop counter
#define POLY 0x8408                 // CRC16-CCITT FCS (X^16+X^12+X^5+1)
    crc = crc ^ newbyte;
    for( i = 0; i < 8; i++ ){
        if( crc & 0x01 ){
            crc = crc >> 1;
            crc ^= POLY;
        }else{
            crc = crc >> 1;
        }
    }
    return( crc );
}

I have created these "matching" functions in B4J:

Sub Get_Buf_CRC( buf_offset As Int, numBytes As Int) As Int ' needs unsigned int16
    Dim crc As Int = 0, i As Int
    For i = 0 To numBytes-1
        crc = UpdateCRC( crc, mHexImage(buf_offset + i))
    Next
    Return crc
End Sub
    
Sub UpdateCRC( crc As Int, newbyte As Byte ) As Int
    Dim CRCpoly As Int = 0x8408        ' CRC16-CCITT FCS (X^16+X^12+X^5+1)
    Dim i As Int

    crc = Bit.Xor( crc, newbyte )
    For i = 0 To 7
        If Bit.And( crc, 0x0001 ) = 1 Then
            crc = crc/2
            crc = Bit.Xor( crc, CRCpoly )
        Else
            crc = crc/2
        End If
    Next
    Return crc
End Sub

but, should I be surprised? the results don't match...

I have been having much pain with the type conversions in Java from byte to unsigned int and such and it's probably another one of those cases...

Any help or suggestion appreciated.

Note that I am not bound to that particular implementation or even that type of CRC, it's just that the C version results in very small code space utilization and I have very little code space available, so I would like to use it.
TIA

 

[OliverA]

Try

crc = Bit.Xor( crc, Bit.And(newbyte, 0xFF) )

 

[Didier9]

It's giving me a different CRC, but it still does not match the C calculation.

I am going to have to peel that one offline for a bit. The automatic type conversions in Java, and the additional layer of indirection afforded by B4J have made these types of operations frustratingly difficult to me.

 

[OliverA]

What is mHexImage? How is it generated? How does it compare to the data pointed to by uchar *ptr?

 

[Didier9]

mHexImage is a byte array. In the C code, *ptr points to what should be the "same" byte array.
mHexImage is read from an Intel Hex file, decoded, mapped into a memory image, broken down into pages and sent to the uC via UART one page at a time. That part seems to work.

This code is part of a bootloader. The B4J app sends a flash image (actually a flash page at a time) along with a CRC to the target (the uC). The uC recalculates the CRC of the received data and checks that the CRCs match before flashing the data.

I did verify that at least the first few bytes of the image were transferred correctly. I have not checked the entire page but it's a test file and it's mostly 0's at the moment.
I also have a logic analyzer watching over the UART data to make sure it is what I think it is. I am much more comfortable with the uC side of things because C pretty much talks to the hardware directly and I have done that for over 30 years. B4J, not so much.

But I did not write the CRC routine in the C code, it came from the chip vendor as part of their bootloader architecture sample code, along with a PC app in exe form with source for an earlier rev. Unfortunately, the whole thing is "work in progress" and the PC app is a C# mess that I have given up on after spending a week and a half trying to make sense of it. The C# source code they provided is a rev down from the exe that sort of works and does not even include the CRC routines. At least the C code was pretty complete and I understand what it does, so I have tried not to change it too much.

There is a control block (a couple dozen bytes, it's an 8 bit uC) being transferred from the target to the PC before the actual image data is transferred to make sure the image was intended for that target and even though it was harder than I was hoping for, that part now works fine and downloading the image also seems to work. The last stumbling block is the CRC.

I am going to work on the CRC separately, and maybe skip the CRC for now in the bootloader project itself to make sure everything else is working. With the perfectly functional source level in-system debugger in the C tools and the logic analyzer on the UART, I should be able to figure it out...

 

[OliverA]

I'm pretty sure your inputs are not the same. I've created a B4J console application and a Visual C Console application, slightly modified the C source to work with Visual C's C++ compiler (mainly type declarations) and created some dummy input data. Both console applications produce the same result (given the same input data). I'm attaching both B4J and Visual C source (just a .cpp file for the C++ portion).

 

[Didier9]

Thank you, that will help a lot.

Rechecking that the image data sent to the target was correct, I did find that some bytes are corrupted. I had only checked the first few bytes and they were correct so I assumed the whole page had been transferred correctly but that's not the case, I do have occasional data replaced with 0x3F values. The byte count is correct, so it looks like I need to troubleshoot that first. Probably another one of those Java type conversions.

Update: It looks like it is a signed/unsigned issue. Bytes that are < 0x80 are transferred correctly, bytes > 0x7F are changed to a default 0x3F...

Anyhow, I should be able to figure that out. I'll update when it's fixed.

 

[OliverA]

0x3F

That's in indicator that you are using ASCII set in B4J instead of ISO-8859-1. In the B4J application that I posted above, if you replace the ISO-8859-1 in the GetBytes method with ASCII, you'll get the same 0x3F issue.

 

[Didier9]

I am using ISO-8859-1. None of it works at all with UTF-8.
I was using UTF-8 on an earlier program this one was derived from, but that previous program was using 7 bit + parity.
This program uses straight 8 bit data and no parity.
I am puzzled as I am getting the headers OK, which are binary too and some fields have values above 0x7F. The program has multiple back and forth exchanges before getting to transferring the flash image and it all works...

Thanks for your help but I feel bad your spending time on this right now on my behalf as I should be able to fix this issue which is not CRC related.

 

[OliverA]

Thanks for your help but I feel bad your spending time on this right now on my behalf as I should be able to fix this issue which is not CRC related.

I'm not wasting my time. I'm currently working on a pretty large C code port to B4X. A lot has to do with "bit-ness" differences between C and B4X (and pointers, and data structures, and pointers, btw did I mention pointers?) and processing of data (such as your CRC function). So anything that seems to cause issues, I like to investigate, since it may impact on how I handle the data portion in my project. So far though, it's always been an issue with garbage in/garbage out instead of program language issues per se.

 

[Didier9]

Well, thank you for your time nonetheless! And I agree, as you can see below...

I think I found the problem
At least I found another problem
I was off by one page when calculating the CRC, I was calculating the CRC on the wrong page, not on the page I was sending...
I have not proven that it now works as there may still be yet other issues but I am cautiously hopeful.
I did make a few program structure changes in the meantime in an attempt at making it easier to debug so it may be tomorrow before I get it back together and I can prove it but that would definitely be a problem...

I have not done any large scale porting of C code to B4J, but I have translated a number of functions, not unlike what I am doing now because a lot of my B4A/B4J projects end up talking to microcontrollers programmed in C, so a number of data interface routines have versions in C and congruent versions in B4J.
And it's not been smooth.
Pointers are like Vinales at the last Moto-GP race, jumping off a perfectly straight up race bike at 140 mph because he wanted to, not because he fell...

 

[Didier9]

OK, I am copying the data properly now (the 0x3F issue). The problem was the conversion from the old 7 bit + parity code into straight 8 bit was not working properly. Since the old code was using strings because the data was mostly printable (I had already changed the decoding to ISO-8859), I still had a number of conversions between strings and bytes and that was just not proper in this application, so I rewrote all the I/O functions to use bytes. At least it is much easier to debug and cleaner and it did fix that particular problem. Now back to the checksum..

 

[Didier9]

I did try the two projects you linked above and as expected they give the same result.

Thanks a lot!

I do not routinely use Visual Studio, only to run software I downloaded that I want to evaluate, or in a case like this. I do like B4J A LOT more as a development environment. VS is just such a pig... I have 16BG of RAM and a fast SSD on this i7 machine and it's slow...

 

[Didier9]

OK, success!
The bootloader is working, thanks again for your help Oliver!
There were a few more issues along the way after resolving the CRC but it is now working.