B4A Library MteEval - B4X Expression Compiler and Eval Library (Open Source)

B4X Expression Compiler and Eval Library

MteEVAL is a library for compiling and evaluating expressions at runtime. Expressions are converted to bytecode and then executed on demand with a simple virtual machine.

There are five editions of the library:
  • Android (B4A)
  • iOS (B4i)
  • Java (B4J)
  • JavaS2 (B4A/B4J)
  • .NET (C#)
JavaS2 is our stage 2 performance edition of the library written in native Java.

Application

Creating expressions at runtime is a powerful tool allowing calculations and program flow to be modified after installation, which otherwise would require a physical update or a custom build of an application. For example, any application designed to manage a sales compensation plan could benefit from runtime expressions, where the end-user may want to customize the plan's formulas by team members, product mixes and sales goals.

Codeblocks

MteEVAL implements a single class named Codeblock. MteEval's codeblock adopts the expression syntax from the venerable 1990's xBase compiler Clipper 5 where the construct began. Codeblocks start with an open brace, followed by an optional parameter list between pipes, then the expression, and end with a closing brace.

{|<parameters>|<expression>}

Operator support

The library supports C/Java style operators along side a growing list of B4X native functions.
  • Math operators: +-*/%
  • Relational: > < >= <= != ==
  • Logical: || && !
  • Bitwise: << >> & ^ |
  • Assignment: =
  • Constants: cPI, cE
  • Functions: abs(), ceil(), floor(), iif(), if(), min(), max(), sqrt(), power(),round()
  • Trig Functions: acos(), acosd(), asin(), asind(), atan(), atand(), cos(), cosd(), sin(), sind(), tan(), tand()
Examples

You only need to compile a Codeblock once. Once compiled you can evaluate it as many times as needed, all while supplying different arguments. All arguments and the return value are type Double.

Example 1: Codeblock without parameters
B4X:
Dim cb as Codeblock
Dim Result as Double
cb.Initialize
cb.Compile( "{||5 + 3}" )
Result = cb.Eval           'Result=8

Example 2: Codeblock with parameters
B4X:
Dim cb as Codeblock
Dim Area as Double
cb.Initialize
cb.Compile( "{|length,width|length*width}" )
Area = cb.Eval2( Array( 3, 17 ) )    'Area=51

When evaluating with parameters, use the Eval2 method.

Example 3: Compile, eval and repeat
B4X:
Dim cb as Codeblock
Dim Commission1, Commission2, Commission3 As Double
cb.Initialize
cb.Compile( "{|sales,r1,r2| r1*sales + iif( sales > 100000, (sales-100000)*r2, 0 ) }" )
Commission1 = cb.Eval2( Array( 152000, .08, .05 ) )    'Commission1=14760
Commission2 = cb.Eval2( Array( 186100, .08, .07 ) )    'Commission2=20915
Commission3 = cb.Eval2( Array( 320000, .08, .05 ) )    'Commission3=36600

Linking to your project
  • To use the Android or Java editions, add the .JAR and .XML files to your Additional Libraries folder and check the MteEVAL library in the Libraries Manager of the IDE.
  • For iOS, copy the modules Codeblock.bas, Codegen.bas, PCODE.bas, and Run.bas to your project folder or place them in the Shared Modules folder. Then add the modules to the project through the IDE.
Demo

Library and demo attached.

Source

Source code is maintained here:
https://github.com/macthomasengineering

Follow this link to track the status of the library at Waffle.io
https://waffle.io/macthomasengineering/mteeval-b4x-library
 

Attachments

  • mteevalS2_b4a_v105.zip
    40.5 KB · Views: 278
  • mteeval_b4a_v106.zip
    46 KB · Views: 287
Last edited:

stanmiller

Active Member
Licensed User
Longtime User
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
No worries. The iOS version is included in the source code bundle at GitHub.

https://github.com/macthomasengineering/mteeval-b4x-library/releases
Hi again,

My new app will ultimately be both Android and iOS so TICK for this.

It is also associated with surveying so needs the full gamut of trigonometric functions.

I have looked through the code in your GitHub project and can see no indication that such are supported - what's the scoop?

Incidentally your code is very professional looking - unlike some - gives me lots of confidence!

Thanks in anticipation...
 

stanmiller

Active Member
Licensed User
Longtime User
Hi again,

It is also associated with surveying so needs the full gamut of trigonometric functions.

Thanks in anticipation...

Adding a function to the source is not too much drama. And not just B4X native functions but your own functions as well. I'll add the trig functions, then you can compare the differences between version 1.03 and 1.04 to see what changed. I also have a draft of an article that I'll post here soon about how to add functions.

In short:

1. You would add a pcode for the function in PCODE.BAS.
2. Add the function string, argument count, and pcode to the case statement in FindInternalFunc() of CODEGEN.BAS
3. Add the pcode to the case statement in RUN.BAS.
4. In the case section for said function, grab the arguments off the stack. aSP[ nSP -1 ], aSP[ nSP ], etc. then call the function. Store the return value in nAX. The compiler will do the rest.
5. Add the pcode to the decompiler case statement in RUN.BAS with some descriptive text.

We currently have version 1.04 in progress which allows assignment of variables that may be used later in the expression. I'll bundle the trig functions with the 1.04 release on Monday.
 
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
Adding a function to the source is not too much drama.
OK, that looks simple enough.

I take it that if this were done for a Android/B4A app I could install via the method you have outlined previously for iOS/B4i, that is:

...copy the modules Codeblock.bas, Codegen.bas, PCODE.bas, and Run.bas to your project folder or place them in the Shared Modules folder. Then add the modules to the project through the IDE.
 
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
Hi again,

Reading through your code at:

https://github.com/macthomasengineering/mteeval-b4x-library/blob/master/iOS/Codegen.bas

trying to understand the differences between the B4A and B4I versions - what can I say - I have no life!

It seems to me that you are using redundant "#if B4I ..." blocks in this area:

B4X:
371 #if B4I 
372
373 '*-------------------------------------------------------- ExtractExpressions
374 '*
375 Private Sub ExtractExpressions( cb As Codeblock )  As Int
376     Private matchParts As Matcher
377     Private sTrimmed As String 
378     Private i As Int 
379     Private nGroupCount As Int 
380     Private nError As Int 
381     Private sDetail As String
382 #if B4I    
383     Private sGroupText As String
384 #end if 
385
386     gEvalExpr = ""
387     gParamExpr = ""
388     
389     ' Strip spaces and change case
390     sTrimmed = cb.Text.Replace(" ", "" ).ToLowerCase
391
392     ' Break expression into component parts
393     matchParts = Regex.Matcher(CODEBLOCK_MATCH, sTrimmed )        
394     
395     ' Apply pattern        
396      matchParts.Find
397
398     ' Save group count
399     nGroupCount = matchParts.GroupCount 
400
401 #if B4I 
402     nGroupCount = nGroupCount - 1 
403 #end if    
404
405     ' No matches?
406     If ( nGroupCount = 0 ) Then 
407         Return ( SetError( gCodeBlock.ERROR_SYNTAX, "" ) )
408     End If 
409     
410     ' Inspect groups
411     For i = 1 To nGroupCount 
412
413         sGroupText = "" 
414         Try
415             sGroupText = matchParts.Group( i )
416                         
417             ' Build detail string
418             If ( sGroupText <> Null ) Then 
419                 sDetail = sDetail & sGroupText
420             End If
421
422             ' Group value missing
423             If ( sGroupText = Null ) Then 
424             
425                 ' Which one is missing?
426                 Select( i ) 
427                 Case GROUP_OPEN_BRACKET
428                     nError = gCodeBlock.ERROR_MISSING_BRACKET                    
429                 Case GROUP_OPEN_PIPE
430                     nError = gCodeBlock.ERROR_MISSING_PIPE                                                        
431                 ' Case GROUP_PARAM_EXPR                               ' Param expr null ok.
432                 '    nError = gCodeBlock.ERROR_MISSING_PARAM                                                        
433                 Case GROUP_CLOSE_PIPE
434                     nError = gCodeBlock.ERROR_MISSING_PIPE                                                        
435                 Case GROUP_EVAL_EXPR
436                     nError = gCodeBlock.ERROR_MISSING_EXPR                                                         
437                 Case GROUP_CLOSE_BRACKET
438                     nError = gCodeBlock.ERROR_MISSING_BRACKET                    
439                 End Select
440             
441             End If 
442         Catch
443             nError = gCodeBlock.ERROR_SYNTAX
444         End Try
445
446
447         ' If error found, complete detail and return here
448         If ( nError <> gCodeBlock.ERROR_NONE)  Then 
449             sDetail = sDetail & " <e" & nError & ">"
450             SetError( nError, sDetail )
451             Return ( nError ) 
452         End If
453                 
454     Next
455
456     ' RegEx should create six groups
457     If ( nGroupCount = 6 ) Then 
458
459          ' Store parameter expression
460          If ( matchParts.Group( GROUP_PARAM_EXPR ) <> Null ) Then 
461             gParamExpr = matchParts.Group( GROUP_PARAM_EXPR )  ' a,b,c
462         End If
463
464         ' Store main expression
465         gEvalExpr  = matchParts.Group( GROUP_EVAL_EXPR )  ' 1 * a + c * 5
466         
467         ' And it's not zero length
468         If ( gEvalExpr.Length <> 0 ) Then
469             Return ( gCodeBlock.ERROR_NONE )
470         End If 
471                             
472     End If 
473
474     ' Set syntax error
475     nError = gCodeBlock.ERROR_SYNTAX
476     sDetail = sDetail & " <e" & nError & ">"
477     
478     Return ( SetError( nError, sDetail ) )
479     
480 End Sub
481
482 #else
 

stanmiller

Active Member
Licensed User
Longtime User
OK, that looks simple enough.

I take it that if this were done for a Android/B4A app I could install via the method you have outlined previously for iOS/B4i, that is:

Certainly. You can use the source files instead of the lib (.JAR). The code is the same for all platforms.
 

stanmiller

Active Member
Licensed User
Longtime User
It seems to me that you are using redundant "#if B4I ..." blocks...

Indeed. The regular expression engine behaves differently in B4I. Those nested #if's show where I was modifying the original ExtractExpressions() for iOS compatibility. In the end I block copied and made a B4I specific version of the routine.

The nested #if's were leftovers. I've pulled them out in the upcoming 1.04 release.
 

JackKirk

Well-Known Member
Licensed User
Longtime User
The code is the same for all platforms.
So why not just package it as a single all platforms class?

Also, why not put the function adding stuff in separate subroutines? - that would make it a lot easier for fiddlers like me to maintain against whatever changes you might make in the future to the core.
 

stanmiller

Active Member
Licensed User
Longtime User
So why not just package it as a single all platforms class?

The code is the same today, but that could change as it grows to support platform specific features. Should it diverge, the code could quickly get muddled with #if/#end if. So we structured it as independent projects then all like parts can be synced with a diff tool like Beyond Compare.

Stuffing everything into a single class would make for a very large source file and greatly increase the memory footprint as globals and method references in CODEGEN.BAS and RUN.BAS would be duplicated with every instance of the class.

Also, why not put the function adding stuff in separate subroutines? - that would make it a lot easier for fiddlers like me to maintain against whatever changes you might make in the future to the core.

We initially designed for simplicity. But the spirit of open source is that tinkering is encouraged.

We released version 1.04. The library is attached to the start of this thread and the source has been updated at Github.

In version 1.04, I replaced the case statement for finding internal functions with a table. The case statement was simple, but adding the 12 trig functions tipped the balance to where a table would be easier to maintain and improves performance. To add a new function, just add an entry to the array (e.g. mysub).

"funcname", pcode, argcount

B4X:
' Offsets into func table array
Private Const FUNC_TABLE_FUNCNAME    = 0 As Int    'ignore
Private Const FUNC_TABLE_PCODE       = 1 As Int    'ignore
Private Const FUNC_TABLE_ARGCOUNT    = 2 As Int    'ignore

'*--------------------------------------------------------- LoadFuncTable
'*
Private Sub LoadFuncTable
    Private nTableIndex As Int

    ' Table entry format
    ' ------------------
    ' Array( "func", pcode, argcount )

    ' Internal func table
    aFuncTable = Array( "abs",    PCODE.FUNC_ABS,    1, _
                        "iif",    PCODE.FUNC_IIF,    3, _
                        "if",     PCODE.FUNC_IIF,    3, _
                        "max",    PCODE.FUNC_MAX,    2, _
                        "min",    PCODE.FUNC_MIN,    2, _
                        "sqrt",   PCODE.FUNC_SQRT,   1, _
                        "power",  PCODE.FUNC_POWER,  2, _
                        "round",  PCODE.FUNC_ROUND,  1, _
                        "floor",  PCODE.FUNC_FLOOR,  1, _
                        "ceil",   PCODE.FUNC_CEIL,   1, _
                        "cos",    PCODE.FUNC_COS,    1, _
                        "cosd",   PCODE.FUNC_COSD,   1, _
                        "sin",    PCODE.FUNC_SIN,    1, _
                        "sind",   PCODE.FUNC_SIND,   1, _
                        "tan",    PCODE.FUNC_TAN,    1, _
                        "tand",   PCODE.FUNC_TAND,   1, _
                        "acos",   PCODE.FUNC_ACOS,   1, _
                        "acosd",  PCODE.FUNC_ACOSD,  1, _
                        "asin",   PCODE.FUNC_ASIN,   1, _
                        "asind",  PCODE.FUNC_ASIND,  1, _
                        "atan",   PCODE.FUNC_ATAN,   1, _
                        "atand",  PCODE.FUNC_ATAND,  1, _
                        "mysub",  PCODE.FUNC_MYSUB,  1 )

    ' Create map for fast lookup
    mapFuncTable.Initialize
    For nTableIndex = 0 To aFuncTable.Length-3 Step 3
        mapFuncTable.Put( aFuncTable( nTableIndex + FUNC_TABLE_FUNCNAME ), nTableIndex )
    Next

    bFuncTableLoaded = True

End Sub

'*--------------------------------------------------------- FindInternalFunc
'*
Private Sub FindInternalFunc( sName As String ) As MTE_FUNC_INFO
    Private tFuncInfo As MTE_FUNC_INFO
    Private nTableIndex As Object

    ' Load lookup table
    If ( bFuncTableLoaded = False ) Then
        LoadFuncTable
    End If

    ' Search table for function
    nTableIndex = mapFuncTable.Get(sName)
    If ( nTableIndex <> Null ) Then
        tFuncInfo.nPcode    = aFuncTable( nTableIndex + FUNC_TABLE_PCODE )
        tFuncInfo.nArgCount = aFuncTable( nTableIndex + FUNC_TABLE_ARGCOUNT )
    Else
        tFuncInfo.nPcode    = -1
        tFuncInfo.nArgCount = 0
    End If

    Return ( tFuncInfo )

End Sub
 
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
Hi again,

I will probably respond to the above post when I finish tinkering a bit more.

My preliminary tinkering has revealed another typo you should be aware of.

I downloaded the android version of your GitHub project and put it in a B4A project.

I compiled and ran it then tapped the [Run test!] button on the resulting app - nothing happened.

It turns out you have a subroutine in Main:
B4X:
Sub btnRunTests_Click

    RunTests

End Sub
which should read:
B4X:
Sub btnRunTest_Click

    RunTests

End Sub

no second s in btnRunTest_Click - to match the button name in the designer layout.

After this change all seems to be working properly.
 
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
I'm scratching my head a bit trying to understand the value of Mtelog
 

stanmiller

Active Member
Licensed User
Longtime User
I'm scratching my head a bit trying to understand the value of Mtelog

Mtelog is application log used for validation testing, diagnostics, and debug reporting. Its function is independent of the library and can be commented out and ignored.

In general, the purpose of an application log is to only report events pert to your application. Usually to a file. It also can collect information after the application is deployed to help with troubleshooting and other unexpected behavior.
 
Last edited:

stanmiller

Active Member
Licensed User
Longtime User
Hi again,

I downloaded the android version of your GitHub project and put it in a B4A project.

I compiled and ran it then tapped the [Run test!] button on the resulting app - nothing happened.

It turns out you have a subroutine in Main:
B4X:
Sub btnRunTests_Click

    RunTests

End Sub
which should read:
B4X:
Sub btnRunTest_Click

    RunTests

End Sub

no second s in btnRunTest_Click - to match the button name in the designer layout.

After this change all seems to be working properly.

Another nice catch! Source updated at GitHub.

This glitch didn't affect the library posted here as it is published with the TestMteEval demo project which had the button named correctly. The library source at GitHub is bundled with MteBuildLib which had the mispelling.

I made a late copy of DEMO.BAL between TestEval and BuildLib projects and didn't catch that the name of the button was different. Normally, this would have shown up on the final diff report before posting.

Thanks again for the heads up!
 

stanmiller

Active Member
Licensed User
Longtime User
Next week we'll publish a B4X compatible Java edition of the MteEval library. Early benchmarks show Eval() and Eval2() gain a
100x improvement in execution speed over the native B4X edition.

In addition to gains from native Java, we've also introduced peephole optimization which reduces the final code set.

Below shows how the code generator optimizes the PUSH instruction combining LOADVAR and LOADCONST with PUSH.

1_mteeval_push_peephole_optimized_zpstornsolp.jpg


Here the optimizer improves the code size by 25%.

codeBlock.compile( "{|ref,inp|5+NumberFormat(avg(ref,inp),0,2)}" )

codeBlock.decompile()

Before
B4X:
-- Header --
Parameters=2
-- Code --
1:     loadc   ax, 5.0
3:     push    ax
4:     loadv   ax, varmem[0]
6:     push    ax
7:     loadv   ax, varmem[1]
9:     push    ax
10:    call    avg
10:    pop     2
11:    push    ax
12:    loadc   ax, 0.0
14:    push    ax
15:    loadc   ax, 2.0
17:    push    ax
18:    call    numberformat
18:    pop     3
19:    add     stack[sp] + ax
19:    pop
20:    end

With peephole optimization
B4X:
-- Header --
Parameters=2
-- Code --
1:     pushc   5.0
3:     pushv   varmem[0]
5:     pushv   varmem[1]
7:     call    avg
7:     pop     2
8:     push    ax
9:     pushc   0.0
11:    pushc   2.0
13:    call    numberformat
13:    pop     3
14:    add     stack[sp] + ax
14:    pop
15:    end
 
Last edited:

stanmiller

Active Member
Licensed User
Longtime User
Stage2 Performance Edition of the library released. Library attached to the first post in this thread.

Make sure to select the S2 version of the library in the IDE.

1_mteevals2_lib_manager_zpsrv7fx0u4.jpg
 

JackKirk

Well-Known Member
Licensed User
Longtime User
Stan,

OK, back chipping away at this.

I have been able to add functions, by following the paper trail of the trigonometry functions.

Also been able to add constants, by following the paper trail of the cPI constant.

All works, without huge brain strain.

I installed the library via downloading the source from

https://github.com/macthomasengineering/mteeval-b4x-library/releases

The zip I got was named

mteeval-b4x-library-1.05.1.zip

Implying 1.05 - the latest and greatest.

However, I was looking at your post #2 in this thread where you talk about the new features of 1.05:

- Optimizer can be disabled by setting Codeblock.DisableOptimizations=True
- Moved constants from inline to dedicated constants table.

And went looking for the "dedicated constants table" - because I didn't see anything like that when I added my constants - and couldn't find it.

Then went looking for Codeblock.DisableOptimizations and couldn't find that either.

Then looked at project attributes in Main module - it is saying 1.04 all over the place - dam!

It would appear you have not uploaded 1.05 source to

https://github.com/macthomasengineering/mteeval-b4x-library/releases

Cheers...
 

stanmiller

Active Member
Licensed User
Longtime User
Stan,
Implying 1.05 - the latest and greatest.

Feature-wise, the B4X editions of the library are still at 1.04. Version 1.05 pertains to the JavaS2 version only. My apologies. I should have stated this as the first line in the "What's New" notes.

I had started updating the B4X editions with the improvements from v1.05 but also wanted to address/workaround B4X's select/case performance.

Typically a "case" construct should be the fastest branching structure outside of a "goto", but B4X's conservative approach causes a lot of overhead. See this wish.

https://www.b4x.com/android/forum/t...ance-by-front-loading-constant-targets.74434/

B4i uses the same strategy reloading the entire pcode table for each instruction in a compiled expression.
B4X:
switch ([self.bi switchObjectToInt:@((_npcode)) :@[@(self._pcode._push),@(self._pcode._neg),
@(self._pcode._add),@(self._pcode._subtra
ct),@(self._pcode._multiply),@(self._pcode._divide),@(self._pcode._modulo),@(self._pcod
e._equal),@(self._pcode._not_equal),@(self._pcode._less_than),@(self._pcode._less_equal
),@(self._pcode._greater_than),@(self._pcode._greater_equal),@(self._pcode._bit_and),@(
self._pcode._bit_or),@(self._pcode._bit_xor),@(self._pcode._bit_not),@(self._pcode._bit_s
hift_left),@(self._pcode._bit_shift_right),@(self._pcode._logical_or),@(self._pcode._logical
_and),@(self._pcode._logical_not),@(self._pcode._jump_always),@(self._pcode._jump_fals
e),@(self._pcode._jump_true),@(self._pcode._loadconst),@(self._pcode._loadvar),@(self._
pcode._storevar),@(self._pcode._func_abs),@(self._pcode._func_max),@(self._pcode._fun
c_min),@(self._pcode._func_sqrt),@(self._pcode._func_power),@(self._pcode._func_round
),@(self._pcode._func_floor),@(self._pcode._func_ceil),@(self._pcode._func_cos),@(self._
pcode._func_cosd),@(self._pcode._func_sin),@(self._pcode._func_sind),@(self._pcode._fun
c_tan),@(self._pcode._func_tand),@(self._pcode._func_acos),@(self._pcode._func_acosd),
@(self._pcode._func_asin),@(self._pcode._func_asind),@(self._pcode._func_atan),@(self._
pcode._func_atand),@(self._pcode._endcode)]]) {

One workaround is to inline the Run module in ObjectiveC but takes away from a primary goal of the project which was offer a library that was modifiable by everyone.

Another approach is to group instructions by frequency with if/then blocks. That should help some as only the instructions in the group will be loaded by the Select/Case handler.

For example, coded like this a PCODE.PUSH would only require 6 pcodes to be loaded vs. 54.
B4X:
    ' Set instruction pointer
    nIP = CODE_STARTS_HERE

    Do While ( bRun )

        ' get op code
        nPcode = Code.get( nIP )

        ' Is this a stack, load or store instruction?
        If ( nPcode <= PCODE.STOREVAR ) Then
            Select ( nPcode )
            Case PCODE.PUSH
                '...
            Case PCODE.PUSHVAR
                '...
            Case PCODE.PUSHCONST
                '...
            Case PCODE.LOADCONST
                '...
            Case PCODE.LOADVAR
                '...
            Case PCODE.STOREVAR
                '...
            End Select
        End If
    '...
    Loop

While the v1.05 changes improve performance, the real bottleneck in the B4X editions is how select/case is currently implemented. I'll look at it again this weekend. Also, I'm past due for providing an easy way for one to add their own functions.
 
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
Typically a "case" construct should be the fastest branching structure outside of a "goto", but B4X's conservative approach causes a lot of overhead. See this wish.
https://www.b4x.com/android/forum/t...ance-by-front-loading-constant-targets.74434/
You're way too far down in the bilge for me:)
One workaround is to inline the Run module in ObjectiveC but takes away from a primary goal of the project which was offer a library that was modifiable by everyone.
I would not be able to use your code if it was not modifiable.
Also, I'm past due for providing an easy way for one to add their own functions.
Actually, once I sat down to do this and just followed the paper trails, as per post #36, this was quite easy.

My application's use of your code is what I guess could be called "interpretive" - each expression is basically just evaluated once - so performance of repeated evaluation is not a major issue - but having the ability to easily implement it all sure is.

I'm a happy puppy...
 
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
Also, I'm past due for providing an easy way for one to add their own functions.
If you are going to do this what about making it equally easy to add constants?

Regards...
 

stanmiller

Active Member
Licensed User
Longtime User
Over the weekend I added the "S2" optimizations to the B4X editions of the library. I'll publish here and GitHub later today.

Peephole optimization in B4X

1_mteeval_peephole_optimizer_b4x_zps41yoqumf.jpg
 
Top