Android Tutorial Remote Database Connector (RDC) - Connect to any remote DB

Status
Not open for further replies.
jRDC version 2 is available here: https://www.b4x.com/android/forum/t...ation-of-rdc-remote-database-connector.61801/

This tutorial covers a new framework named Remote Database Connector (RDC). The purpose of RDC is to make it simple to develop Android applications that interact with remote database servers.

There are two components in this framework: a lightweight Java web server (the middleware) and a B4A class named DBRequestManager that interacts with this web server.

The Java web server can connect to any database platform that provides a JDBC driver.

This includes: MySQL, SQL Server, Oracle, Sybase, DB2, postgreSQL, Firebird and many others.

You can also use the web server without a database server by connecting to a local SQLite database file.

The Java web-server is a simple server that connects to the database server and to the Android clients.
As this is a Java app you can run it on Linux or Windows computers. It requires Java JRE 6 or 7.

This solution is much more powerful than the PHP (MySQL) and ASP.Net (MS SQL) solutions that are already available.

Main advantages over previous solutions:
  • Support for many types of database platforms.
  • Significantly better performance.
  • SQL statements are configured in the server (safer).
  • Support for all types of statements, including batch statements.
  • Support for BLOBs.

Server Configuration

JDBC is a Java API that provides a standard method to access any database. A database driver (jar file) is required for each type of database. You will need to download the driver for your database and put it in the jdbc_driver folder.

The Java server configuration is saved in a file named config.properties.
This file includes two sections: general configuration and a list of SQL commands.

For example:

SS-2013-08-04_16.10.20.png


Note that the configuration fields are case sensitive.

DriverClass / JdbcUrl - The values of these two fields are specific to each database platform. You will usually find the specification together with the required driver jar file.

User / Password - Database user and password.

ServerPort - The Java server will listen to this provided port.

Debug - If Debug is set to true then the SQL commands list will be loaded on every request. This is useful during development as it allows you to modify the commands without restarting the server process.

SQL Commands - A list of commands. Each command starts with 'sql.'. The commands can include question marks (parameterised queries). Question marks will be replaced with the values provided by the Android clients. Note that the command name is case sensitive and it doesn't include the 'sql.' prefix.

Client Code
The client sends requests to the server. There are two types of requests: query requests (usually SELECT statements) and batch requests (any other statement).

Note that both the client and the server can manage multiple requests in parallel.
Usually you will only need a single DBRequestManager object.

Each request is made of a command (or a list of commands) and a tag. The tag can be any object you like. You can use it later when the result is ready. The tag is not sent to the server.

A command is an object of type DBCommand:
B4X:
Type DBCommand (Name As String, Parameters() As Object)
Name - The case sensitive command name as configured in the server configuration (without sql.).
Parameters - An array of objects that will be sent to the server and will replace the question marks in the command.

For example to send a SELECT request:
B4X:
Dim cmd As DBCommand
cmd.Initialize
cmd.Name = "select_animal"
cmd.Parameters = Array As Object("cat 1")
reqManager.ExecuteQuery(cmd, 0, "cat 1")
ExecuteQuery expects three parameters: the command, maximum number of rows to return or 0 if there is no limit and the tag value.

Under the hood DBRequestManager creates a HttpJob for each request. The job name is always "DBRequest".

You should handle the JobDone event in your activity or service:
B4X:
Sub JobDone(Job As HttpJob)
   If Job.Success = False Then
     Log("Error: " & Job.ErrorMessage)
   Else
     If Job.JobName = "DBRequest" Then
       Dim result As DBResult = reqManager.HandleJob(Job)
       If result.Tag = "cat 1" Then 'query tag
         For Each records() As Object In result.Rows
           Dim name As String = records(0) 'or records(result.Columns.Get("name"))
           Log(name)
         Next
       End If
     End If
   End If
   Job.Release
End Sub

As you can see in the above code, DBRequestManager.HandleJob method takes the Job and returns a DBResult object:
B4X:
Type DBResult (Tag As Object, Columns As Map, Rows As List)
Tag - The request tag.
Columns - A Map with the columns names as keys and the columns ordinals as values.
Rows - A list that holds an array of objects for each row in the result set.

Non-select commands are sent with ExecuteCommand (single command) or ExecuteBatch (any number of commands). It is significantly faster to send one request with multiple commands over multiple requests. Batch statements are executed in a single transaction. If one of the commands fail all the batch commands will be cancelled (roll-backed).

Note that for non-select requests, Rows field in the results will hold an array with a single int for each command. The int value is the number of affected rows (for each command separately).

Initializing DBRequestManager:
B4X:
Sub Activity_Create(FirstTime As Boolean)
   If FirstTime Then
     reqManager.Initialize(Me, "http://192.168.0.100:17178")
   End If
End Sub
The first parameter is the module that will handle the JobDone event.
The second parameter is the link to the Java server (with the port number).

DBRequestManager provides the following helper methods for common tasks related to BLOB fields: FileToBytes, ImageToBytes and BytesToImage.It also includes a method named PrintTable that prints DBTable objects to the logs.

Framework Setup
  1. Unpack the server zip file.
  2. You will need to download the driver for your database platform and copy the jar file to jdbc_driver folder.
  3. Edit config.properties as discussed above.
  4. Edit RunRLC.bat and set the path to java.exe. If you are running it on Linux then you need to create a similar script with the path to java. On Linux you should change the ';' separator to ':'.
  5. Run the server :)
    You should see something like:

    SS-2013-08-04_17.05.16.png


    Note that the path to config.properties is printed in the second line.
  6. Add an exception for the server port in your firewall.
  7. Try to access the server from the browser:

    SS-2013-08-04_17.06.17.png


    You will probably not see the above message on the first run :(. Instead you will need to check the server output and read the error message.
    If you are able to call the server from the local computer and not from other devices then it is a firewall issue.

Tips

  • MySQL driver is available here: http://dev.mysql.com/downloads/connector/j/
  • Google for <your database> JDBC Driver to find the required driver. For SQL Server you can use this open source project: http://jtds.sourceforge.net/
  • The server uses an open source project named c3p0 to manage the database connection pool. It can be configured if needed by modifying the c3p0.properties file.
  • Use a text editor such as Notepad++ to edit the properties file.
  • If you are running the server on Linux then you can run it with nohup to prevent the OS from killing the process when you log out.
  • Java doesn't need to be installed. You can just unpack the files.
The server is based on the two following open source projects:
Jetty - http://www.eclipse.org/jetty/
c3p0 - http://www.mchange.com/projects/c3p0/index.html
http://www.mchange.com/projects/c3p0/index.html
 
Last edited:

TomDuncan

Active Member
Licensed User
Longtime User
Or maybe use sqlite3 instead.
Big problem is file size. With the number of books we have the size might go to 5gig.

This does work well with postgres but with the hassles so far with bytea will have to think about what to do.

Tom
 

TomDuncan

Active Member
Licensed User
Longtime User
Will do some more testing in the morning with the bytea.
Changing the way it stores to the older escape type. (Beyond my knowledge of postgresql but will test)

Tom
 

patrick14384

Member
Licensed User
Longtime User
Ok, I am doing something simple wrong. Trying to get the server running, and I get

C:\Users\gallagher>"C:\Program Files (x86)\Java\jdk1.7.0_25\bin\java" -Xmx256m -
cp .;libs\*;jdbc_driver\* anywheresoftware.b4a.remotedatabase.RemoteServer
Error: Could not find or load main class anywheresoftware.b4a.remotedatabase.RemoteServer

Everything is local to this computer in my effort to start simple.

Not that I think this is the problem, but config.properties is
B4X:
DriverClass=org.postgresql.Driver
JdbcUrl=jdbc:mysql://localhost/test?characterEncoding=utf8

User=postgres
Password=mypassword
ServerPort=17178

Pretty sure I need to put something somewhere on the hard drive, but not sure what or where. Thanks for any help!

Patrick
 

alienhunter

Active Member
Licensed User
Longtime User
Or maybe use sqlite3 instead.
Big problem is file size. With the number of books we have the size might go to 5gig.

This does work well with postgres but with the hassles so far with bytea will have to think about what to do.

Tom

the way that i solved some problems is try to gzip the file and then upload it as blob in mysql /sqlite then you database will be smaller and quicker
when you download it for the database unzip it ....

save it as binary type
http://www.postgresql.org/docs/8.4/static/datatype-binary.html

The SQL standard defines a different binary string type, called BLOB or BINARY LARGE OBJECT. The input format is different from bytea, but the provided functions and operators are mostly the same.

looks like you have to store the binary different like in sql
 
Last edited:

alienhunter

Active Member
Licensed User
Longtime User
batch command

Hi Erel , i could not figure out how to send let say 1000 commands at the same time
example

update row 1 col 1
update row 2 col 1
....
update row 1000 col1
and so on

how should this list be created /needed , i cannot find any example so far

B4X:
for i = 0  to 1000
parm1=(i)
parm2=(i)
ccd=cmd.Name = "update_touda"
list.add (ccd,parm1,parm2)
next
reqManager.ExecuteBatch(list,1)

?

thanks AH
 

opal

Member
Licensed User
Longtime User
Hello, I am new here.
So, I run server. In browser writed: http://127.0.0.1:17178/?method=test. Response OK: RemoteServer is running (Sun Nov 10 13:56:23 CET 2013)
Connection successful.

Next, part of my confige file - I use Pervasive SQL:
DriverClass=com.pervasive.jdbc.v2.Driver
JdbcUrl=jdbc:pervasive://localhost:1583/DEMODATA?transport=tcp

#commands
sql.create_table=CREATE TABLE animals (\
id CHAR(10) NOT NULL,\
name CHAR(30) NOT NULL,\
PRIMARY KEY (id))
sql.insert_animal=INSERT INTO animals VALUES (null, ?,?)
sql.select_animal=SELECT name, image FROM animals WHERE name = ?
sql.select_person=SELECT First_Name, Last_Name FROM Person

Then I run demo Remote Database Connector and execute Sub:
B4X:
Sub GetAnimal(Name As String)

    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "select_animal"
    cmd.Parameters = Array As Object(Name)
    reqManager.ExecuteQuery(cmd, 0, Null)
End Sub

and I get error: java.sql.SQLException: [LNA][Pervasive][ODBC Engine Interface][Data Record Manager]No such table or object.

correct, I haven't table animals. Then I execute Sub:
B4X:
Sub GetAnimal(Name As String)
    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "select_person"
    cmd.Parameters = Array As Object(Name)
    reqManager.ExecuteQuery(cmd, 0, Null)
End Sub

Table Person I have in database, but I get error:
java.sql.SQLException: Function not implemented
Error: Server Error

Same error I get if I run Sub:
B4X:
Sub btnCreateAnimals_Click
    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "create_table"
'    cmd.Parameters = Array As Object(Name)
    reqManager.ExecuteQuery(cmd, 0, Null)
End Sub

Can anybody help me ?
 

opal

Member
Licensed User
Longtime User
Where, or how I can get server log file ? I copy log from B4A and attached.
 

Attachments

  • something.zip
    37.1 KB · Views: 407

opal

Member
Licensed User
Longtime User
Server log (I hope):


c:\RemoteDatabaseConnector>"c:\Program Files (x86)\Java\jre7\bin\java" -Xmx256m
-cp .;libs\*;jdbc_driver\* anywheresoftware.b4a.remotedatabase.RemoteServer
B4A Remote Database Connecter (version 0.9)
loading: c:\RemoteDatabaseConnector\config.properties
2013-11-11 16:21:37.692:INFO::jetty-7.4.2.v20110526
2013-11-11 16:21:37.715:INFO::started o.e.j.s.ServletContextHandler{/,null}
nov 11, 2013 4:21:37 PM com.mchange.v2.log.MLog <clinit>
INFO: MLog clients using java 1.4+ standard logging.
nov 11, 2013 4:21:37 PM com.mchange.v2.c3p0.C3P0Registry banner
INFO: Initializing c3p0-0.9.2.1 [built 20-March-2013 11:16:28 +0000; debug? true
; trace: 10]
2013-11-11 16:21:37.930:INFO::Started [email protected]:17178 START
ING

I use Windows 8.
 

opal

Member
Licensed User
Longtime User
Sorry, I did not. You are the best. Here is new post. I executed first Sub:
B4X:
Sub GetAnimal(Name As String)
    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "select_animal"
    cmd.Parameters = Array As Object(Name)
    reqManager.ExecuteQuery(cmd, 0, Null)
End Sub

Table animal doesn't exist in database. Then I executed Sub:
B4X:
Sub btnCreateAnimals_Click
    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "create_table"
    reqManager.ExecuteQuery(cmd, 0, Null)
End Sub

Then I executed Sub:
B4X:
Sub GetAnimal(Name As String)
    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "select_person"
    cmd.Parameters = Array As Object(Name)
    reqManager.ExecuteQuery(cmd, 0, Null)
End Sub

Table Person exist in database.

Server log (see message: Function not implemented):

c:\RemoteDatabaseConnector>"c:\Program Files (x86)\Java\jre7\bin\java" -Xmx256m
-cp .;libs\*;jdbc_driver\* anywheresoftware.b4a.remotedatabase.RemoteServer
B4A Remote Database Connecter (version 0.9)
loading: c:\RemoteDatabaseConnector\config.properties
2013-11-11 19:39:15.966:INFO::jetty-7.4.2.v20110526
2013-11-11 19:39:15.990:INFO::started o.e.j.s.ServletContextHandler{/,null}
nov 11, 2013 7:39:16 PM com.mchange.v2.log.MLog <clinit>
INFO: MLog clients using java 1.4+ standard logging.
nov 11, 2013 7:39:16 PM com.mchange.v2.c3p0.C3P0Registry banner
INFO: Initializing c3p0-0.9.2.1 [built 20-March-2013 11:16:28 +0000; debug? true
; trace: 10]
2013-11-11 19:39:16.203:INFO::Started [email protected]:17178 START
ING
nov 11, 2013 7:40:49 PM com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource ge
tPoolManager
INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acqu
ireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCo
mmitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> fa
lse, checkoutTimeout -> 20000, connectionCustomizerClassName -> null, connection
TesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceN
ame -> 1hge0z88yk3sdwag4zill|18806f7, debugUnreturnedConnectionStackTraces -> fa
lse, description -> null, driverClass -> com.pervasive.jdbc.v2.Driver, factoryCl
assLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -
> 1hge0z88yk3sdwag4zill|18806f7, idleConnectionTestPeriod -> 600, initialPoolSiz
e -> 3, jdbcUrl -> jdbc:pervasive://localhost:1583/DEMODATA?transport=tcp, maxAd
ministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 1800, maxIdleTi
meExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 150, maxStatements
PerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, preferredTestQuery
-> null, properties -> {user=******, password=******}, propertyCycle -> 0, state
mentCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testCon
nectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, userOverrides -> {
}, usesTraditionalReflectiveProxies -> false ]
java.sql.SQLException: [LNA][Pervasive][ODBC Engine Interface][Data Record Manag
er]No such table or object.
at com.pervasive.jdbc.lna.LNAObject.getErrors(LNAObject.java:213)
at com.pervasive.jdbc.lna.LNAStatement.getErrors(LNAStatement.java:937)
at com.pervasive.jdbc.lna.LNAObject.checkError(LNAObject.java:256)
at com.pervasive.jdbc.lna.LNAStatement.prepare(LNAStatement.java:273)
at com.pervasive.jdbc.v2.PreparedStatement.<init>(PreparedStatement.java
:54)
at com.pervasive.jdbc.v2.Connection.prepareStatement(Connection.java:539
)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask.run(Go
oGooStatementCache.java:546)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(Thre
adPoolAsynchronousRunner.java:648)
java.sql.SQLException: Function not implemented
at com.pervasive.jdbc.v2.PreparedStatement.getParameterMetaData(Prepared
Statement.java:653)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.getParameterMetaDa
ta(NewProxyPreparedStatement.java:303)
at anywheresoftware.b4a.remotedatabase.Servlet.createStatement(Servlet.j
ava:178)
at anywheresoftware.b4a.remotedatabase.Servlet.executeQuery(Servlet.java
:140)
at anywheresoftware.b4a.remotedatabase.Servlet.doGet(Servlet.java:78)
at anywheresoftware.b4a.remotedatabase.Servlet.doPost(Servlet.java:52)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:538
)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java
:478)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl
er.java:937)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:
406)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle
r.java:871)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j
ava:117)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper
.java:110)
at org.eclipse.jetty.server.Server.handle(Server.java:346)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.
java:589)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpCo
nnection.java:1065)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:823)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220)

at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:41
1)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEn
dPoint.java:535)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEnd
Point.java:40)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool
.java:529)
at java.lang.Thread.run(Unknown Source)
java.sql.SQLException: Function not implemented
at com.pervasive.jdbc.v2.PreparedStatement.getParameterMetaData(Prepared
Statement.java:653)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.getParameterMetaDa
ta(NewProxyPreparedStatement.java:303)
at anywheresoftware.b4a.remotedatabase.Servlet.createStatement(Servlet.j
ava:178)
at anywheresoftware.b4a.remotedatabase.Servlet.executeQuery(Servlet.java
:140)
at anywheresoftware.b4a.remotedatabase.Servlet.doGet(Servlet.java:78)
at anywheresoftware.b4a.remotedatabase.Servlet.doPost(Servlet.java:52)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:538
)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java
:478)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl
er.java:937)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:
406)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle
r.java:871)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j
ava:117)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper
.java:110)
at org.eclipse.jetty.server.Server.handle(Server.java:346)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.
java:589)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpCo
nnection.java:1065)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:823)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220)

at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:41
1)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEn
dPoint.java:535)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEnd
Point.java:40)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool
.java:529)
at java.lang.Thread.run(Unknown Source)
 

opal

Member
Licensed User
Longtime User
Dear Erel, yes table animal missing, I sad that in post above. This message sent Pervasive SQL and because of that I know that java code called (comunicate with) Pervasive SQL, but table Person exist and server message is: java.sql.SQLException: Function not implemented. Log above include call of 3 different Sub, as I explained in post before. Execute first sub result: java.sql.SQLException: [LNA][Pervasive][ODBC Engine Interface][Data Record Manager]No such table or object. Execute second sub result: java.sql.SQLException: Function not implemented and same message I get when I execute last Sub.
 

alienhunter

Active Member
Licensed User
Longtime User
B4X:
For i = 1 To 1000
   Dim cmd As DBCommand
   cmd.Initialize
   cmd.Name = "i" & i
   cmd.Parameters = Array As Object("a", i)
   commands.Add(cmd)
Next
reqManager.ExecuteBatch(commands, Null)

works perfect i enjoy this RDC more and more
thanks
 
Last edited:
Status
Not open for further replies.
Top