B4J Question XML/JSON Fields-Values Automatic Matching

[Magma]

Hi there,

the question (not exactly only for B4J - may be a tactic how to manage something) came from Chit chat https://www.b4x.com/android/forum/threads/because-you-can.167192/ when talking with my friend @aeric who did a great job with Web Server API (Pakai Framework).

Well..
In Greece we have about 30 different E-Invoicing Providers having DIFFERENT JSON or XML (MyDATA - the old/first public invoicing for all) - the Json are having different structure and different fields but sometimes if mapping the right you need... less work to do your work...

Sometimes need loops for extracting values from a post, or need just to match..

I am attaching a WORD (docx) file in a zip with 3 different providers for the same type of invoice.. have in mind that invoice types in Greece are about ~30... the providers are also about 30 and there may be some different flow working them..

If having same API for all will be great... but is a dream...

My opinion is to use some AI tech - to magic MATCH them, feed the AI xmls, jsons, examples...
and create me templates to match them...

May be with the power of Pakai (aeric's) and the power of AI (there are many examples here how to use it in b4x) and may be some power of new Erel's features (using python)... can create a web server wrapper or offline wrapper to do my job better...

automatic converting - matching values and elements for xmls-json ...

What do you think ? - Have in mind some code will help... try to create a project like this "MAGIC-MATCH-XML/JSON" >?

 

[aeric]

There is another possibility that the Providers means they are the appointed agents or Authorized e-Invoice Issuers recognized by the government.
Business or Companies who want to generate e-Invoice must go through this Providers to generate the e-Invoices.

 

[Magma]

There is another possibility that the Providers means they are the appointed agents or Authorized e-Invoice Issuers recognized by the government.
Business or Companies who want to generate e-Invoice must go through this Providers to generate the e-Invoices.

Providers ...
Are the end target of e invoices and certified from gov... sending to gov only specific data ...at mydata.(which in 1/7/2025 not usable for apps you can go only with those service providers that can take with gov mydata)

They are the courier for the different type of invoices, receipts, sendinvoices, etc to mydata accounting... you can send direct from 1/7... and all have different api for one reason ... to not lose U from customer
.. because integration needs 10-15 days for every provider.. this is capitalism (political talk for creating needs and source for money for nothing)

Their official name from gov AADE our irs for those Providers is:Certified Electronic Invoicing Service Providers
Or
Licensed Electronic Invoicing Providers

Is a big talk

 

[Magma]

If you can't draw the flow chart, can you listed out the parties from Left to Right indicate Start to End for each steps?


Since you didn't provide answers for these questions and I could not ask your government for the answers, I can only make my own assumptions.

I assume the flow is as following:

Scenario 1
1. End User (Buyer/Customer/usually consumer of the products and services) makes a purchase with --> 2. Providers (Seller/Merchant/usually business owners providing products or services) generate and send e-Invoice in the form of JSON/XML to --> 3. Government e-Invoicing API Server (MYDATA?) and the server validate the e-Invoice then send back the validated e-Invoice back to --> 4. Providers and generate a validated e-Invoice to 5. End User (who may use ERP system, apps or custom software developed by Developers).

Scenario 2
The process continues from #5 where the End User can accept or reject the e-Invoice...
There are many other scenarios but let us focus on Scenario 1 at this point.

I assume Providers are the one who concern about sales profit.
Hence they are also the party that making request or consuming the APIs provided by the government API server (MYDATA?).
Meanwhile End User concern about tax submission or claim the tax refund where the e-Invoice is a proof of purchase.

Here you want to slot in as an intermediate Middleware provider so the flow changed to this:
2. Providers (Seller/Merchant/usually business owners providing products or services) generate and send e-Invoice in the form of JSON/XML to 3. Middleware provider that accepts different types or non standard types of e-invoices then output the correct format required by the API of --> 4. Government e-Invoicing API Server (MYDATA?)

So your actual concern is to make the Providers' life easier for providing an intermediate service to make the API submission smoother.

Am I right so far?

No, I want to make programmers life easier...
Freelancers or small companies not having their own providers...

 

[aeric]

It makes more sense to me now.

So the Providers concept is very similar to what we have here call Service Providers.

Would I scare you that not only 30+ but maybe we have 3,000 or 30,000+ of service providers?

As you said above, actually companies don't really need these providers.

In fact, anyone can also become one of the service providers.
Since I already registered myself as one with my country's inland revenue bureau (IRB), I can do testing in sandbox environment.
When I want, I can provide a service to help companies to submit their e-Invoice.

This also mean some companies can also submit their e-Invoice without engaging any third party service providers if they have an in-house software team.
If I understand, MYDATA used to be the only authorized provider but by time goes by, now more and more providers are available.

So let say I am in your situation, to become a middleware provider, is that mean I need to deal with 30,000+ of service providers?

The different situation between what happen in your country vs what we have here, that I can see is:

In Greece, different service providers have different "pre" e-Invoice format.
I don't know why is this happening.
By right, they (so called the Provider) should "provide" or adhere to the format required by the government API server. You call it AADE server?
They should become the Solution Provider who should "Solve" the Problem.
A Middleware provider is not required.
The government should enforce a standard API format documented in a SDK guideline where a list of mandatory fields need to be provided.
Optional fields are accepted.
If the providers are not providing the correct format then it means they are not compliant or eligible to call providers.

In Malaysia, each service providers should follow the SDK guideline or else end up their e-Invoice getting validation failure.
The role of Middleware providers are different here.
I am not 100% sure what they are doing but I see there is a company who claim its product name associated with AI can help any business or company convert their legacy ERP system or excel format invoices into e-Invoice compliant with the government API standard.

Back to your case, I have an idea to solve your problem. (maybe 2 ideas)

Idea #1
Become a Middleware provider
Either you need to communicate with the providers to get their "pre" formatted e-invoice or you get from the companies that engage services from these providers.
You need to map their fields to AADE format.
Every time there is a change of key names involving the "mandatory" fields, you need to update your middleware system.
This mean you have a database to store each provider "template" and associate with their client id.

Idea #2
Don't need to become a Middleware provider
Because your target are not service providers, so why need to focus or concern about the providers format?
Straight away build one product that can generate the e-Invoice compliant to AADE format
You can sell this product at a very minimum price or have a different business model

Lastly, I can say for both Ideas, you can use B4J to develop the server or even a standalone system.
You may also consider to build using one of my frameworks.

 

[Magma]

It makes more sense to me now.

So the Providers concept is very similar to what we have here call Service Providers.

Would I scare you that not only 30+ but maybe we have 3,000 or 30,000+ of service providers?

As you said above, actually companies don't really need these providers.

In fact, anyone can also become one of the service providers.
Since I already registered myself as one with my country's inland revenue bureau (IRB), I can do testing in sandbox environment.
When I want, I can provide a service to help companies to submit their e-Invoice.

This also mean some companies can also submit their e-Invoice without engaging any third party service providers if they have an in-house software team.
If I understand, MYDATA used to be the only authorized provider but by time goes by, now more and more providers are available.

So let say I am in your situation, to become a middleware provider, is that mean I need to deal with 30,000+ of service providers?

The different situation between what happen in your country vs what we have here, that I can see is:

In Greece, different service providers have different "pre" e-Invoice format.
I don't know why is this happening.
By right, they (so called the Provider) should "provide" or adhere to the format required by the government API server. You call it AADE server?
They should become the Solution Provider who should "Solve" the Problem.
A Middleware provider is not required.
The government should enforce a standard API format documented in a SDK guideline where a list of mandatory fields need to be provided.
Optional fields are accepted.
If the providers are not providing the correct format then it means they are not compliant or eligible to call providers.

In Malaysia, each service providers should follow the SDK guideline or else end up their e-Invoice getting validation failure.
The role of Middleware providers are different here.
I am not 100% sure what they are doing but I see there is a company who claim its product name associated with AI can help any business or company convert their legacy ERP system or excel format invoices into e-Invoice compliant with the government API standard.

Back to your case, I have an idea to solve your problem. (maybe 2 ideas)

Idea #1
Become a Middleware provider
Either you need to communicate with the providers to get their "pre" formatted e-invoice or you get from the companies that engage services from these providers.
You need to map their fields to AADE format.
Every time there is a change of key names involving the "mandatory" fields, you need to update your middleware system.
This mean you have a database to store each provider "template" and associate with their client id.

Idea #2
Don't need to become a Middleware provider
Because your target are not service providers, so why need to focus or concern about the providers format?
Straight away build one product that can generate the e-Invoice compliant to AADE format
You can sell this product at a very minimum price or have a different business model

Lastly, I can say for both Ideas, you can use B4J to develop the server or even a standalone system.
You may also consider to build using one of my frameworks.

To become provider you need at least 100K$ and take responsibility for all software using your services.
Mydata was something like provider ..now that is changing due regulations and laws

API standard is our second name in OUR UNION...and is not fair not have it...as you can understand

 

[aeric]

What I still don't understand is why you need to deal with the non compliant output from the providers.

As I wrote, the providers suppose to come out with the ready-to-submit data and they suppose to submit to the API? No. The small business still need to submit the data themselves?
I don't see a point here.

Let me assume.
For example, you are a provider. When I need to issue e-invoice, I need to make API call to your API server first and you return a new API response embeded with your registered code.
Now with this data, I will make a second round of API call to AADE?

I have to pay you for the number of API calls to your server?

Now come to software developers of ERP, POS, Accounting software or Apps who wants to support all possible API provided by these providers.

Questions now are,
Actually a software can just lias to a single provider, right? Or 2 for backup.

Even 30 providers can be choose, they all should provide the same data to submit to AADE right? The difference is the part of unique code embeded. Like a stamping of certified true copy.

In Malaysia style, we don't do this. We have an option to use Digital Signature and this DS providers are the authorized by IRS. We don't bother the format as we only use the DS private key to encrypt the API to send as v1.1 e-invoice to IRS. The other option is to send without DS as v1.0 e-invoice. As for DS, we need to pay an annual fee for single company use or as service providers to handle multiple companies.

I don't see the sample API you provided involve digital signed data.

The issue is the authorised providers have
1. non standard return data or
2. non standard API

But now I see, it should be possibility #2. The providers indeed output the correct format for AADE.
It just that they have their own input API format.
Now I see the fun part.
The providers are the boss now.

As a union, you may have submit a petition to IRS or AADE to enforce a standard API format and then the providers' role is just to certify the API.

If in-house software developer want to be easy, just support 1 or 2 providers is enough. But in reality they are greedy and want to market their product claiming it support all providers.

 

[Magma]

Now with this data, I will make a second round of API call to AADE? No - The provider will send at AADE/MYDATA (only him will have the Lisence) - the model will use Greece is different from all Countries !!!

I have to pay you for the number of API calls to your server? At provider yes - paying for every call... at my middleware "NO" (is something that will help all - as you know i believe in KARMA) - this will only translate data between providers API's (read the following - will understand)


It just that they have their own input API format.
Now I see the fun part.
The providers are the boss now.
--------------------------------------->Yeap... you ve understand.. well not exactly the boss... i can say that only for the big Providers... have the knife and the Pie

As a union, you may have submit a petition to IRS or AADE to enforce a standard API format and then the providers' role is just to certify the API.
We ve already did that.. but in Greece some things go slow.. I am with you...
Now think different... you wanna sell your POS product (Done i think) in Greece... first you must follow the laws-tax/regulations (understandable), second you must intergrate with one Provider to send your invocies/receipts to AADE... yeap... pay him, learn his api... and ofcourse if he close his services... you must intergrate with another one...

But in reality they are greedy and want to market their product claiming it support all providers. ...Yes must intergrate with 3-5... but there are also other traps... big companies (or the half of 30...15) have their own Software-POS... Invoicing APPS... so think about it?

 

[aeric]

at my middleware "NO"

Can you explain what's your idea at the moment?

 

[Magma]

Can you explain what's your idea at the moment?

Actually to intergrate all the providers and "can" translate between their json/xml (including also old mydata xml's)
May be a class, may be a service (free)...

at first:
For sure can be class
perhaps..

FromMyDataToParochos(xml) as json
or FromParochosToMyData(json) as xml

but first must create templates for matching the fields/elements/values... may be those templates can be text/ini/settingsmap

for example:
FromMyDataToParochos.template
Price|0.00=Amount|0
companyName=Name
VatNo=TaxNo

using delimeter or something... can have different conversions or Format to recognize or translate the field/element..

So this project can start with a simple tool of Matching... and can be better...
I think Matching Json Values will be helpful for other projects too.. may these templates can be a guide to convert other files formats too..

 

[aeric]

Before
I thought the json/xml samples you shared are output
After
now I realize the sample is actually required API payload

The problem now indeed is on the providers part.

Let say I want to build a "translater" server as you may call it a middleware,
I need to

1. Create my own set of fixed fields
2. Study all providers format
3. Create the template mapping my fixed fields to required fields for each provider I want to work with
4. Maintain the template whenever any updates from each provider

 

[Magma]

Before
I thought the json/xml samples you shared are output
After
now I realize the sample is actually required API payload

The problem now indeed is on the providers part.

Let say I want to build a "translater" server as you may call it a middleware,
I need to

1. Create my own set of fixed fields
2. Study all providers format
3. Create the template mapping my fixed fields to required fields for each provider I want to work with
4. Maintain the template whenever any updates from each provider

1. You are thinking to create a new "clear" API for esoteric/internal use
2. may be ...an AI first hand ?... kidding, I feel a hate for it...
3. sure
4. that's right... this is a difficult part too..

 

[aeric]

Actually to intergrate all the providers and "can" translate between their json/xml (including also old mydata xml's)
May be a class, may be a service (free)...

at first:
For sure can be class
perhaps..

FromMyDataToParochos(xml) as json
or FromParochosToMyData(json) as xml

but first must create templates for matching the fields/elements/values... may be those templates can be text/ini/settingsmap

for example:
FromMyDataToParochos.template
Price|0.00=Amount|0
companyName=Name
VatNo=TaxNo

using delimeter or something... can have different conversions or Format to recognize or translate the field/element..

So this project can start with a simple tool of Matching... and can be better...
I think Matching Json Values will be helpful for other projects too.. may these templates can be a guide to convert other files formats too..

For the matching/mapping, I think we can just use maps.
It should be one to one for each provider API.
Using class with custom type for fields also good idea. It's what I am doing.
Since your "User" can have many different type of file, text, CSV, excel, access, direct db, etc
I think you need to deal with case to case basis.
You can do charity but this is really not worth it.
This is too much of work.

 

[aeric]

1. You are thinking to create a new "clear" API for esoteric/internal use
2. may be ...an AI first hand ?... kidding, I feel a hate for it...
3. sure
4. that's right... this is a difficult part too..

Just start with one provider first then add another.

 

[Magma]

For the matching/mapping, I think we can just use maps.
It should be one to one for each provider API.
Using class with custom type for fields also good idea. It's what I am doing.
Since your "User" can have many different type of file, text, CSV, excel, access, direct db, etc
I think you need to deal with case to case basis.
You can do charity but this is really not worth it.
This is too much of work.

Since your "User" can have many different type of file, text, CSV, excel, access, direct db, etc ?
what do you mean ?

charity?
I think that is other thing to create a class translating, other thing to create templates (train the system)
Perhaps the training can be the cheese for the success of a service or an app or a class

Training/Tranlastion can be a service with a cost> For any developer or advance user wanna do that.

 

[aeric]

Since your "User" can have many different type of file, text, CSV, excel, access, direct db, etc ?
what do you mean ?

charity?
I think that is other thing to create a class translating, other thing to create templates (train the system)
Perhaps the training can be the cheese for the success of a service or an app or a class

Training/Tranlastion can be a service with a cost> For any developer or advance user wanna do that.

There are 2 options you can provide to the developers as a Middleware provider:
Option 1
Share your fixed fields or API format to the developer so they can self service to use your server
Option 2
Provide an integration service to help convert whatever data type they are working with

What you mean by training?
1. Feeding mapping information to your system?
Or
2. Training session for users on how to use your middleware ?

 

[Magma]

1st ofcourse

 

[aeric]

1st ofcourse

If you mean use AI or machine learning to help your middleware doing the matching I can understand.

However, using this tool can add up a cost.

I don't see a need or feasibility in early stage.

It is more reliable if manually done by human.
Even with AI, we still want to verify again.

I don't know how advanced the AI agent can find the correct specifications from the providers.

 

[Magma]

If you mean use AI or machine learning to help your middleware doing the matching I can understand.

However, using this tool can add up a cost.

I don't see a need or feasibility in early stage.

It is more reliable if manually done by human.
Even with AI, we still want to verify again.

I don't know how advanced the AI agent can find the correct specifications from the providers.

may be if the developer-of-an-app/user provide the same "values" or formatted "values" with examples of xmls / jsons for the same invoice type but different provider

 

[Magma]

I enjoy the conversation with you... hope both sides learning from swaping ideas

 

[aeric]

may be if the developer-of-an-app/user provide the same "values" or formatted "values" with examples of xmls / jsons for the same invoice type but different provider

I don't get it.

I still believe AI doesn't need to be involved here.
Not every time we need rely on AI.
AI can be unpredictabe and provide inconsistent results.