B4J Tutorial [B4X] Faces, Assemblies, Traces, and Points. Standard Classes for creating a story.

Since retiring some time ago, I have been thinking about creating stories that involve
computer operated robots which carry out actions and speech - like actors in a drama.

It would have at least two mobile robotic platforms that can move on a stage - about the size of a dining table.
Each platform would have a Android tablet on-board that receives instructions from a desktop or laptop.
Those instructions would determine movements on the stage.
Each tablet would also show a human-like face on the screen.
This face would display emotions and lipsync to sound tracks.

I have been working intermittently on pieces of this project.
For me, the journey is definitely more fun than getting there.

Here I present some of my experiences. I will concentrate on things one can do with B4X
on the Desktop and Android tablet. Most of the code should also run on B4i.

First a taste of what we can achieve. A very short video.

https://youtube.com/shorts/ag_u-x9Lta8?feature=share

For now, I will discuss the software building blocks of this ambitious project.
There are other discussions I can have about hardware and inter-device communication.
I'll leave that for another time.

Let's create a set of structures - each will be implemented as one or more standard classes.
Story: a set of ordered scenes
Scene: a collection of Faces
Face: a collection of Assemblies
Assembly: a collection of Traces
Trace: a set of ordered Points
Point: a pair of ordered Numbers

This tutorial comes in sections A, B, C, D, E, anf F

_____________________________________________Tutorial A._______________________________

Let's start from the bottom.
One could define a "Point" as a custom type. I have done that in the past, and that works well.
But as a class, it can have time-saving methods such as adding points and finding the distance between points.

One note about terminology. It is very cumbersome to always distinguish between the instances of a class and the class itself.
My convention here is to use lowercase and mostly plural for instances and ProperCase and singular for classes.
For example points are instances of the class Point.

In B4X, an instance of a class can spawn instances of itself and other classes. This is very useful.
If we start with a generator instance, we can use it to generate new instances, without cluttering up modules with extra methods.

Copy the following in a new project or download the attached.zip


B4X:
'B4XMainPage
'_____________________________________
Sub Class_Globals
    Private Root As B4XView
    Private xui As XUI
    Private CV As B4XCanvas
    Private Pnt As Point            'generator instance
End Sub

Public Sub Initialize
    Pnt.Initialize
End Sub

Private Sub B4XPage_Created (Root1 As B4XView)
    Root = Root1
    CV.Initialize(Root)
 
    Dim center As Point = Pnt.New(Root.Width / 2, Root.Height / 2)
    Log(center.X & TAB & center.Y)    '300    300
 
    CV.drawCircle(center.X, center.Y, 50dip, xui.Color_Blue, False, 3)

    Log(2 * center.DistanceTo(Pnt.New(0, 0)))    'length of screen diagonal
'Or
    Log(2 * Pnt.New(0, 0).DistanceTo(center))    'length of screeen diagonal
 
'You could also define center as halfway down the diagonal
    Dim TopLeft As Point = Pnt.New(0, 0)
    Dim BottomRight As Point = Pnt.New(Root.Width, Root.Height)
    center = TopLeft.halfwayTo(BottomRight)
    Log(center.X & TAB & center.Y)    '300    300

'Or as the centroid
    center = TopLeft.Add(BottomRight).MultBy(1 / 2)
    Log(center.X & TAB & center.Y)    '300    300
End Sub

Point
___________________________________
Sub Class_Globals
    Public X, Y As Float
End Sub

Public Sub Initialize
End Sub

'Spawns a new instance of Point
Public Sub New(X_ As Float, Y_ As Float) As Point
    Dim P As Point
    P.Initialize
    P.X = X_
    P.Y = Y_
    Return P
End Sub

'Copies a Point
Public Sub Copy As Point
    Return New(X, Y)
End Sub

'Subtract second Point from first
Public Sub Minus(p As Point) As Point
    Return New(X - p.X, Y - p.Y)
End Sub

'Multiplies a Point by a constant
Public Sub MultBy(factor As Float) As Point
    Return New(factor * X, factor * Y)
End Sub

'Adds two Points
Public Sub Add(p As Point) As Point
    Return New(X + p.X, Y + p.Y)
End Sub

'Finds distance between two Points
Public Sub distanceTo(p As Point) As Float
    Dim dx As Float = p.X - X
    Dim dy As Float = p.Y - Y
    Return Sqrt(dx * dx + dy * dy)
End Sub

'Finds the mid-Point between two Points
Public Sub halfwayTo(p As Point) As Point
    Return New((X + p.X) / 2, (Y + p.Y) / 2)
End Sub

'Returns a point that is partway (fraction) along the line between two points
Public Sub partwayTo(p As Point, fraction As Float) As Point
    Return New(X + fraction * (p.X - X), Y + fraction * (p.Y - Y))
End Sub

ScreenShot_A.gif
 

Attachments

  • Tutor_A.zip
    17.9 KB · Views: 137
Last edited:

William Lancee

Well-Known Member
Licensed User
Longtime User
_____________________________________________Tutorial B._______________________________
A "Trace" is a geometric shape containing the geometric information which remains when location, scale, orientation and reflection are removed.
I am coining this term because traces are more than just regular shapes like a circles, rectangles, polygons, or enclosed spaces.
They can also be dots, lines and un-closed curves. The term refers to tracing a drawing, or tracing a numbered set of dots.

I also did not want to confuse these geometrical structures with the View objects shaped by the excellent B4X Shape library created by @Steve05.

Traces are rendered as a series of line segments, if they are filled they are rendered with B4XPath.
They do not have physical dimensions until they are rendered on a given screen at a specified position with a tilt angle and a scaling factor.
In addition, to make an instance of Trace visible you have to provide a color scheme and line width.

1. A trace's form is an ordered set of Points. This is implemented as a List. A trace can be a set of daisy-chained curves.

2. It should be possible to move and rotate traces. A trace's position itself is a Point. Traces also have a TiltAngle.

3. Traces should be screen-independent. This is implemented by standardizing all dimensional units.
A convenient unit is Min(Screen width, Screen height) / 1000. This makes the drawings fit in any orientation.
and 10 standard units are equivalent to 1% of the maximum sized square that fits the screen.

4. It also should have color and texture information so that it can be properly rendered.

Screenshot_B.gif


The drawing above was made with the following code.
Because of extensive use of Point, the Trace class is compact (about 450 lines) and is included in the attached .zip

B4X:
'In Sub Class_Globals
'    Private Traces(11) As Trace            'they can also be named individually, in this demo there are 11
'    Public SU As Float

Private Sub B4XPage_Created (Root1 As B4XView)
    Root = Root1
    centerX = Root.Width / 2
    centerY = Root.Height / 2
    SU = Min(Root.Width, Root.Height) / 1000
    backgroundClr = Root.Color
   
    CV.Initialize(Root)
    For i = 0 To Traces.Length - 1
        Traces(i).Initialize        'a un-formed Trace at the screen center
    Next

    'When Traces are defined, the specified dimensions are given relative to the Root, they are automatically changed to standard units.
    Traces(0).Line(150)                        'A 150 pixel wide line which when drawn will be at the center of current screen
    Traces(0).MoveTo(centerX - 200, 75)        'Move the line into position, left of center, near the top, you'll see this when Trace is drawn
    Traces(0).Rotate(-135)                    'Rotate the line around its own center, this will not be visible until the Trace is drawn
    Traces(0).Specify(CreateMap("outline": black, "thickness": 3))
   
    Traces(1).Triangle(50, 100, 100)
    Traces(1).MoveTo(centerX, centerY - 200)
    Traces(1).Rotate(90)
    Traces(1).Specify(CreateMap("fill":paleGreen, "outline":green, "thickness":3))

    Traces(2).Rectangle(200, 100)
    Traces(2).Rotate(90)
    Traces(2).Specify(CreateMap("fill":paleBlue, "outline":blue, "thickness":3))
   
    Traces(3).Polygon(50, 5)
    Traces(3).MoveTo(centerX - 100, centerY + 200)
    Traces(3).Rotate(180)
    Traces(3).MoveLeft(5)        'An alternative way to move Traces is as a percentage in a Left, Right, Up, Down direction
    Traces(3).Specify(CreateMap("fill":paleRed, "outline":blue, "thickness":3))

    'Four pairs of x,y co-ordinates representing the points defining a Bezier curve, the curve will be mirrored to form a symmetric Trace
    Traces(4).SymCurve(centerX, centerY - 13, centerX - 100, centerY - 75, centerX - 55, centerY + 53, centerX, centerY + 70, 25)
    Traces(4).MoveRight(30)      
    Traces(4).MoveDown(30)
    Traces(4).Specify(CreateMap("fill":crimson, "thickness":3))
   
    Traces(5) = Traces(4).copy    'Clones the previous Trace (heart) and moves and rotates it, it will be drawn in a different color scheme
    Traces(5).MoveLeft(25)
    Traces(5).Rotate(15)
    Traces(5).Specify(CreateMap("outline":crimson, "thickness":3))

    'Freeform Trace defined by a sequence of points - this is just a amorphous blob
    Traces(6).AddPoint(100,100)
    Traces(6).AddPoint(150,200)
    Traces(6).AddPoint(200,100)
    Traces(6).AddPoint(250,400)
    Traces(6).AddPoint(300,100)
    Traces(6).AddPoint(350,400)
    Traces(6).AddPoint(100,400)
    Traces(6).AddPoint(100,100)
    Traces(6).Shrink(50)            'Scales the blob to half its size (50%) - when drawn it occupied too much space for this demo
    Traces(6).MoveRight(30)
    Traces(6).MoveUp(10)
    Traces(6).Smooth(5, 5)            'A very useful smoothing function
    Traces(6).Rotate(135)
    Traces(6).Specify(CreateMap("fill":paleRed, "outline":crimson, "thickness":3))
   
    Traces(7).Oval(200, 100)        'This is an oval, if the width and height are the same it is a circle
    Traces(7).MoveLeft(30)
    Traces(7).Specify(CreateMap("fill":paleRed, "outline":crimson, "thickness":3))

    Traces(8).Arrow(250, 15, 60, 15)    'The Arrow class is embedded: https://www.b4x.com/android/forum/threads/b4x-a-class-to-draw-on-canvas-many-types-of-arrows-at-any-angle.142539/
    Traces(8).MoveRight(19)
    Traces(8).MoveDown(13)
    Traces(8).Rotate(-45)
    Traces(8).Specify(CreateMap("outline":blue, "thickness":1, "dashed":True))
   
    Traces(9).Dot
    Traces(9).Specify(CreateMap("outline":black, "thickness":3))
   
    'This simple Bezier curve will become the base line for the curved text example - it can be drawn or remain un-rendered
    Traces(10).Curve(centerX + 50, centerY - 253, centerX - 100, centerY - 75, centerX - 55, centerY + 53, centerX, centerY + 70, 1000)
    Traces(10).MoveLeft(10)
    Traces(10).Rotate(90)
    Traces(10).MoveTo(centerX, 50)
    Traces(10).Specify(CreateMap("outline":red, "thickness":1, "closed":False))
   
    renderTraces
End Sub

Private Sub renderTraces
    Dim r As B4XRect
    r.Initialize(0, 0, Root.Width, Root.Height)
    CV.ClearRect(r)
    For Each dude As Trace In Traces
        dude.Render
    Next

'Fonts sizes will be automatically adjusted to screen sizes and resizing - this text is drawn in the center of a Trace (-45 degree arrow)
    Traces(8).addText("Testing A String", xui.CreateDefaultBoldFont(11), blue)

'The text follows the curve that is Trace(10)
    Traces(10).addCurvedText("Standard Units Work Well Here", xui.CreateDefaultBoldFont(15), black)
End Sub

Tutorial C and D will be posted in the next 24 hours.
 

Attachments

  • Tutor_B.zip
    18.1 KB · Views: 90

William Lancee

Well-Known Member
Licensed User
Longtime User
Addendum to #2 Tutorial B:
Download the project, and open it in B4J. You can see the effect of changing screen sizes by resizing the form.

_____________________________________________Tutorial C._______________________________

One of the methods of Trace is drawing smooth curved amorphous forms. How this is done is explained here
https://www.b4x.com/android/forum/t...h-curved-line-to-a-sequence-of-points.143178/

The text on a curve method is explained here
https://www.b4x.com/android/forum/threads/b4x-text-along-any-curve.138639/

The Trace class can render Bezier curves. I will explain now how these types of curves are created.
See a good example of the power of bezier curves https://www.b4x.com/android/forum/t...ppy-hearts-day-with-fun-tasia-flowers.145886/

Tutor_C.gif



For those not familiar with Bezier curves, they are smooth complex curves that are specified by 4 points.
An simple example is shown here: https://www.b4x.com/android/forum/threads/b4x-text-along-any-curve.138639/

There are start and end points that are the terminals of the curve. In between are two control points, conceptualized as pulling-forces on the curve.
Since the whole curve is defined by these 4 points, it can be animated by simply changing the points and re-drawing the curve.
There is no need for bitmap/image processing or movie frames. Just B4XCanvas and B4XPath objects.

Attached is an editor for the curve seen in the image above.
(For this demo, I have made the right side a mirror image of the left.)

Download the project, and open it in B4J. You can see the effect on the curve by dragging the control points.
You might be interested to see the unusual way that touch events are implemented.

Also try re-sizing the form in B4J, you'll see that all of the dragging and editing work seamlessly on the resized screen.
When the mouse is released, the Log will show the new standardized control points for the curve. They will be un-affected by resizing.
These points can be exported to any code that includes the Point and Trace classes, and shown on any sized screen.
 

Attachments

  • Tutor_C.zip
    17.7 KB · Views: 94

William Lancee

Well-Known Member
Licensed User
Longtime User
_____________________________________________Tutorial D._______________________________

Back to my goal of showing expressive human-like faces on flat screens.
A 2D face can be seen as a collection of about 20 traces - most of them are Bezier curves:
head (skull, jaw, neck)
hair (hair top, hair bottom)
nose (top-part, nostrils)
eyebrow
ear (top-part, lobe)
eye (upper/lower lid - eyeball and pupil are separate)
mouth (upper/lower lip, upper/lower teeth, tongue)

Obviously the mouth area has most of them. upper and lower lips, teeth, and tongue.
The mouth area is also linked to a separate section of the jaw as it is in nature.


ScreenShot_D.gif
 

Attachments

  • Tutor_D.zip
    55.1 KB · Views: 92

William Lancee

Well-Known Member
Licensed User
Longtime User
_____________________________________________Tutorial E._______________________________

The Assembly class is an ordered collections of traces. This is also implemented as a List

The traces in a Assembly should move in a co-ordinated way.
Traces already are capable of rendering themselves.
The assembly can use its traces for animation.

Screenshot_E.gif


In the attached project, you'll see that a lot of complexity can be reduced by this abstraction.
A convincing human-like face can be rendered with about 320 lines of code, including all Bezier curve control points.
 

Attachments

  • Tutor_E.zip
    76.7 KB · Views: 94

William Lancee

Well-Known Member
Licensed User
Longtime User
_____________________________________________Tutorial F._______________________________

This is the final session in the tutorial. It contains all the techniques discussed in the earlier sessions.
Some of the classes in the attached .zip file have matured from the earlier versions.
Therefore I pefixed their name with WiL.

It also has:

1. Multiple faces - in B4J (you need a large screen) you can create and edit new faces - all curves and all colors
2. Lip synching using phonetics - there are several web sites that turn text (in many languages) into phonetics.
3. Emotional expression - there are 28 different facial expressions

It is challenging, but the biggest lesson of all is that if you break things down into manageable parts,
many seeminly impossible tasks become possible. For example, an emotion:
Disgusted = Mouth_PulledUp Brows_Frown Eyes_Center Eyes_Squint
(each of these is then broken down into animated changes in facial curves)

Those of you who work for a living probably think that I have too much time on my hands.
But like @klaus I believe that it is critical to keep my mind challenged as I am getting older.
In any case, I hope that this creation puts a smile on your face.

Take your pick from any of the following:

1. In the spirit of the movie "2001: A Space Odyssey" a 1968 film produced and directed by Stanley Kubrick.
The computer in the movie is named "HAL" <= Chr(Asc("I") - 1) & Chr(Asc("B") - 1) & Chr(Asc("M") - 1). => "IBM"

Dim n() As Int = Array As Int(154 - Asc("N"), 155 - Asc("D"))
Chr(n(1)) & Chr(n(0))) => WL which are my initials. Therefore: ND

2. I was looking for a gender-neutral name and Willie seemed too conceited. (Although I did toy with bilikin - bill(i)/manekin).
My favorite movie actress is Andie MacDowell ('Groundhog Day' and 'Sex, Lies, and Videotape'). Therefore: Andie or ND
Admittedly this actress is not gender-neutral.

3. The characters generated by the program are animated like puppets or like a ventriloquist's dummy.
Instead of traditional wood and cloth, they are made up of more modern materials, bits and bytes. Therefore: Neo-Dummy or ND

4. The Specification string for any face consists of 26 short lines. Think of these lines as the DNA of our characters.
DNA => a ND.

The roman number suffixes are references to royal/papal names, such as Charles-III and Benedict XVI.
I started with two NDs side by side and the first was called ND and second was called ND2 (Andie too).
This seemed appropriate, so I 'formalized' it with roman numerals. The code contains an algorithm
for converting between roman and regular numbers - take a look at the code.

This final tutorial is not something you will be able to understand right away. Here is the code (in B4XMainPage) with the instructions to
"tell a story" and if you then click on the screen, it cycles through the 28 emotions.

B4X:
Private Sub B4XPage_Created (Root1 As B4XView)
    Root = Root1
    GetFaceSpecs

'If you want to edit a face
    Ed.Initialize(allFaceSpecs)
    FCEd.Initialize
    B4XPages.AddPageAndCreate("Ed", Ed)            'only useful in B4J
    B4XPages.AddPageAndCreate("FCEd", FCEd)        'only useful in B4J

'Faces, or Andys (ND) are distinguished by roman numerals ND_I, ND_II, etc.   
    Dim faces() As String = Array As String("ND_I", "ND_II", "ND_IV", "ND_III")
    For i = 0 To 1
        For j = 0 To 1
            panels(i, j) = xui.CreatePanel("")
            panels(i, j).SetColorAndBorder(xui.Color_Transparent, 1dip, xui.color_Black, 5dip)
            Root.AddView(panels(i, j), i * Root.Width / 2, j * Root.Height / 2, Root.Width / 2 - 2dip, Root.Height / 2 - 2dip)
            NDx(i, j).Initialize(panels(i, j), allFaceSpecs.get(faces(2* i + j)))
        Next
    Next

    touchPanel = xui.CreatePanel("Touch")
    Root.AddView(touchPanel, 0, 0, Root.Width, Root.Height)
    groupNumber = 0
    RenderAssemblies

'    changeEmotions
    Sleep(0)
'    Wait For (NDx(0,0).Say(words.Get(1))) Complete (Result As Boolean)

    NDx(1, 0).AnimateTo("Neutral", 500, 15)

'    Wait For (NDx(0,0).DiscussWith(NDx(0,1), Stories.Get(2), phStories)) Complete (Result As Boolean)

    Wait For (NDx(1,0).TellStory(Stories.Get(0), phStories)) Complete (Result As Boolean)
    Wait For (NDx(1,0).TellStory(Stories.Get(1), phStories)) Complete (Result As Boolean)

#If B4J
'    B4XPages.ShowPage("Ed")
'    Ed.modify("ND_I")
#End IF
End Sub

#if B4J
Private Sub Touch_MouseClicked(Ev As MouseEvent)
    groupNumber = groupNumber + 1
    changeEmotions
End Sub
#Else If B4A
Private Sub Touch_Click
    groupNumber = groupNumber + 1
    changeEmotions
End Sub
#End IF
 

William Lancee

Well-Known Member
Licensed User
Longtime User
Tutorial F - Addendum and Attachments

The faces will animate from neutral to the emotional expression.

The project has taken years, and has evolved and has been recreated many times.
I am still not fully satisfied with lip-synching (which is hard to synchronize with sound if the phrases are long).
[It would be nice to have a callback at the end of each word!]

Also some of the robotic elements are pesky.
I have two working robots, but the present mobile platform has only three wheels (2 driving) and the movements are not precise enough.
I have two cheap 6" tablets that can display the faces and have sound. But the sound is tinny, and the CPU speed is not fast enough for lip sysnching.
And while speech synthesis has come a long way in the last few years, off-line synthesis is not good enough.

I have had some success with making a stage that can give feedback to the robot about the robots location on the stage.
I use a matrix of 192 (12 x 16) RFID tags. As the robot passes over each, it detects its id, sends it to the stage manager
over WiFi. The manager can match the id with the location and voila.

As I said at the start, the jouney has been very entertaining, and much more productive than a game.
Having a toolkit like B4X available at your fingure tips is a joy.

It also helps that I have easy access to the B4X community that shares its wisdom so freely.

Screenshot_F.gif
 

Attachments

  • Tutor_F.zip
    159.1 KB · Views: 106

William Lancee

Well-Known Member
Licensed User
Longtime User
Note: I am upgrading my internet access tomorrow, Monday Feb 6.
If I don't respond to posts then, I will as soon as I am back online.
 

kimstudio

Active Member
Licensed User
Longtime User
Hi @William Lancee very nice, thanks for sharing!

By looking at your code I am curious on how the "phonetics" works to generate voices. The B4J app doesn't have sound as I can see maybe the mp3 files are too large to be uploaded. I'd like to know whether the voice is generated from text like speech synth or pre-recorded mp3 files for long sentences. I guess you have many vowels and consonants recorded in mp3 then combined them to become a voice? If works like this way then you know each word ends for lip syncing? It is interesting.
 

William Lancee

Well-Known Member
Licensed User
Longtime User
Hello @kimstudio
The sound is pre-recorded mp3 files.

The steps are:
1. compose the script, break down into manageable chunks - short phrases
2. online to Google for TTS and save files for offline use
3. online to https://tophonetics.com/ for text to IPA phonetics
4. phonetics to mouth movement, using facts about how human sounds are made
5. the size of .mp3 files is closely related to sound duration which is used to control speed of mouth movements

I will post the missing sound files. Indeed they are too large to upload, but I'll use Google drive.
It made take a day or two. I have some temporary technical difficulties.

In meantime, post #1 has the link to a video which has sound.
 
Last edited:

William Lancee

Well-Known Member
Licensed User
Longtime User
The .mp3 files are here, unzip and copy to Object folder of Tutor_F
They turned out to be small enough, but I did not have them in Files.
 

Attachments

  • ND_Sounds.zip
    72.9 KB · Views: 117
Top