OpenBaseMovil - J2ME Framework, J2ME database and more

Hi folks,

We've been making some tests of the (Open)BaseMovil applications on the BlackBerry emulator (soon in a real device) and also with some Windows Mobile based devices like Qtek S200 and specially an HTC Touch Dual.

Here are some screenshoots of bmSales, our commercial SFA application running there (click to enlarge, on new window):

On the BlackBerry:

On the HTC Touch:

We have some new features in stock that will come with the 3.1 version, but in the meanwhile we've decided to roll out a bug-fix release 3.0.2 which is now available for download at SourceForge and also from the download page in this site.

This release contains some minor modifications to the developer guides (database and quick start) and two bugfixes spotted by Stefan Haacker (thanks Stefan).

People have started to comment on the forums, they are getting alive and start being useful. We are very happy with that and encourage you to collaborate and help us giving life to the forums.

The major changes for the upcoming  release 3.1 are the new storage engine and the new ui engine.

The storage engine will overcome the limitations imposed in the RecordStore size by some devices, such as BlackBerries, Nokia Series40 and Motorola.

The ui engine will provide better support for Nokia Series40 devices and also for touch screen devices. Depending on the schedule, the ui engine might have to wait until 3.2 but we expect to roll it out by 3.1.

Hi, I'm really tired today, yesterday was a very demanding day with very little sleep, a flight (with some delays for the way back), and again not enough sleep.

But it was a very productive day at the fair, though I still think that it's not a fair for us to have an stand.

A friend from Unkasoft who is promoting their very promising mobile game advertising engine was so kind to invite us, they have an stand at the Spanish Pavilion. If you're around there it's worth to talk to them and see what they have to offer.

During the fair we had some meetings with interesting people, some arranged previously, some not. But probably the one of the most interesting and, probably, promising was meeting Robert Virkus, CEO of Enough Software, the company behind J2MEPolish. I think that there's a way for collaboration between both parties that might lead to good opportunities and also to useful free applications. I'll talk to Robert after the show so we can know more about each other's platform.

I won't talk about the thousands of new phones, android testing platforms, and so on. I think there's enough people talking about that now

By the way, we're a bit late with the development of the upcoming release which will fix the reported issues, and will include a new storage engine to override the current limits of many devices (S40, BlackBerry, Motorola, and more) and also a new version of the ui engine. As soon as we have a working beta version we will release it, but in the mean time we will probably publish a 3.0.2 release to just fix the current issues.

Tomorrow we will be spending the day at the 3GSM Mobile World Congress at Barcelona.

If you're around and want to talk a while, drop me an email at narciso ATTTT elondra DOTTT com.

I'll check email even during the day (with my phone, of course)

We've received many questions about different matters around OpenBaseMovil, and many times we ended up writing a brief history of the project to clarify concepts.

So we've thought that it's better to write here the full history of the birth, growth and future of OpenBaseMovil. We're sure that it will help many of you to understand the possibilities of the framework, what to expect now and in the near future, how to collaborate and many other matters (beware, this is a long story ):

The starting point and the problem

About three years ago we were trying to sell a SaaS application for the healthcare sector, and we found that we missed some important issues and the market was not ready for that. It will be, probably it's now starting to be ready. But we needed another path to keep going, or say goodbay.

It happens that one of our partners is also co-owner of a large distribution firm, they produce ice cream and also distribute frozen food and other alimentary goods. They have been struggling with mobility solutions for years, and though they implied benefits that finally paid the TCO the ROI was poor in terms of the time it took and they faced many problems:

1. The terminals are very expensive (around 2000€)
2. They are very large and heavy
3. They need a tech guy to install the software, and also to process the collected data each day and load it with fresh data for the next day.
4. The tech guy also acts as the first line support for the sellers, but has no real knowledge of the inner workings of the system and needs to talk to the manufacturer, which in turn does not have access to the current customer data in order to diagnose. So support is kind of a nightmare.
5. When the terminal brokes, and that happens more and more as they age, they are expensive to fix.The need to have spare terminals to have a fast replacement, usualy up to 50% of the salesperson base.
6. The salesperson needs training, and when something happens is absolutely dependent on the tech guy at the company.
7. They need to be in contact telephonic touch with the company to check for stock breaches or news.

We explored the market as it was by then, and new solutions were appearing to solve some of the above issues:

• Lowend PDAs with Windows CE (now Windows Mobile)
  1. The solve partially point 1, since they're not as expensive
  2. High end ones might have a GPRS module, but are more expensive, and need to interchange data through FTP directly to the corporate office.
  3. Low end ones might have Bluetooth, and be capable of transmitting through a paired cellular phone.

  But they introduce new issues:

  1. Connecting the PDA to a phone over bluetooth can be a nightmare to a non-tech guy such as salespersons.
  2. The require the company (for remote connection) to have a DSL or similar connection with a fixed IP (more costs and problems) or even dedicated modems on dedicated lines to handle the communications (even worse).
• Ruggedized PDAs also with Windows Mobile
  1. They are as expensive as the old terminals
  2. They share the same issues introduced with the low end PDAs.
• BlackBerry specific on-line solutions and WAP or Java based on-line solutions for Java ME
  1. Being on-line does not work for people that need their data everywhere and with fast access
  2. Even with 3.5G, the network latency is too big to give good response time for a good user experience and usability.
  3. They require the company to have a DSL or similar connection with fixed IP, a server for the communcations that it is at the same time connected to the backend server or database creating a dangerous and demanding link on the later.
  4. You don't have data signal everywhere, even in highly populated areas.
     1. Sometimes because of the building you're into
     2. Sometimes because the antennas give precedence to voice signal
     3. Sometimes because other are using data also, and the antennas have limited channels so they delay or drop packets when they cannot handle all the connections (we've tested this with as little as 7 phones connecting at the same time).
     4. Sometimes because the antennas have just voice channels or cover very big areas (in rural and coastal zones the antennas might be voice-only and have a range of up to 70km)

Then the inevitable question came to our heads: WHY are there no off-line (in fact some-times-connected) solutions for cellular phones?

They have many advantages out of the box:

1. The user already has one
2. He cares for it
3. They are cheaper, harder and more easy to use than any of the above.
4. They have increasing abilities that are "eating" other devices day by day (mp3, cameras, gps, etc.)
5. They have more than enough storage space
6. The user perceives the phone as a consumer good, a simple and easy to use tool, and not a computer (though they are now like small computers)

So we tried to do it and started to search for solutions to build our Sales Force Automation solution on Java powered cellular phones.

The beginning, BaseMovil 1.0

We created a prototype using the guidelines for RecordStore usage, with an object-oriented database approach, serializing and deserializing the data and storing and retrieving it with RecordStores.

Too soon we found that the RMS system is very, very primitive, well suited for storing game hi-scores but not for real databases with many tables and thousands of rows in each table. To begin with, just enumerating the records took a big deal of time, and then you needed to iterate over them all to perform a simple search. The search times were simply unusable.

And the object-oriented approach made it very difficult to synchronize the data with the backend server, since no reflection is present in Java ME and you need an specific approach on the server side to process each new object.

So we changed the approach to a table one (so serialization could be done with a single method for any object using a description of the table, the metadata), and we added a Btree index mechanism bound to each table so we could search efficiently for the data we needed (such as customer or product names and /or codes).

At the same time we started with the manufacturer-model-firmware nightmare, since every phone places buttons were it wants, every one has it's own bugs, every one has it's own limitations (number of open simultaneous RecordStores, size limits, memory leaks, and so on).

Finally, we had BaseMovil 1.0 has a Sales Force Automation solution with a custom database and synchronization engine that was connected to a backend server (ERP).

We did all of this always with the SaaS approach in mind, so the sync server was hosted and controlled by us, and the ERP was connected to it using an interchange protocol.

It worked!

BaseMovil 2.0, a platform was born

Given the amount of work done, we realized that 80% of the code was not really part of the SFA solution, but could be easily reused for other applications.

So we refactored the code to split it in several libraries with well-defined functions, and modified the server to support multiple databases and applications.

Then we started to build the ui engine using XML that would in turn create the LCDUI forms and list to create menus, data forms, and data lists.

We also did some big changes to the sync engine to make it more network-proof (we have found errors that were not supposed to happen, errors that the TCP/IP stack was supposed to control), and to be able to handle more incoming requests.

BaseMovil 3.0, the platform gets mature

By now we had a real-world experience of two years, and that meant a lot. We had time to fix almost every bug in the database, to be able to repair it from many situations, and to improve it with new features.

But three main issues remained a problem: network failure tolerance, server load, and user interface usability.

We now had all the knowledge needed to address all of this problems, so we:

• created a really fault-tolerant database initialization and synchronization process, that it is able to recover from network outages and other issues such as the application being closed or battery being drained. And all of it imposing a very moderate pressure on the server, so it is now able to handle an almost unlimited number of requests. It will of course increase the response time of the server (that is when we just add new hardware), but the devices are ready to tolerate it and to recover to an improbable outage.
• we created new menus and data-list browsers using direct drawing (Canvas), so the user interface got much more pretty, usable and responsive. And it also addressed some of the differences between devices. This was a particular gain for SonyEricsson phones, whose native list's are not very well suited to showing data.

Of course, no software is defect-free, and there're still issues around for sure. But the truth is that now we receive little support calls, and most of them are in the first weeks of use and due to network problems (not software ones). And there's something really important about support here: the user is very comfortable and confident with the phone, and is able to repair or reinstall the application without further assistance in the case that a problem arises (phone get's broken, for example).

So we have finally addressed all the issues, and reached a 3.5 stage in our SaaS architecture:

1. Phones are much affordable than most comparable PDAs or Ruggedized PDAs.
2. They are small and light.
3. The user can install the application just sending an SMS, with zero configuration. He can also send and receive just updates at anytime, and knows that the data is updated and introduced into the backend system. He does not need to go the the office anymore.
4. There's no need for a tech guy, or at least there's no need for him to be devoted to the terminals. We can provide the user support, since we know the system and we have always the status of each phone.
5. When the terminal brokes, it is far more easy and less expensive to replace. The end user can simply buy a new phone, send the SMS and get back to work in 15 minutes with the same info he had when he last synchronized.
6. The salesperson needs no training, he knows how to do his job, he knows how to handle a phone and the application seems a natural extension of the phone's capabilities.
7. They have a frequently updated local database, that allows out-of-signal, fast operation, but at the same time has recent information. And he can always use the on-line capabilities to check out information.

OpenBaseMovil, BaseMovil goes Open Source

Almost since we released BaseMovil 2.0 we started to think about going open source. We believe in open source, we use it, and wanted to return something to the community and society.

But we were afraid that other players in our local markets could benefit of our work (we had some buying propositions by some of them), and Alas! we like what we do, but we and our families have the bad habit of eating each day.

Finally, we decided that we could open source a big deal of the platform so it could be used for free under a dual license (GPL and commercial), and at the same time keep the part of it that is the core of our business (the sync engine).

And thus, OpenBaseMovil was born.

The near future plans

1. Reach other platforms
2. Real multi-platform applications
3. The consumer market
4. Others

Reach other platforms

Why keep the platform just to Java ME devices? We were sure that it has to be the first, since it is by far the most spreaded one. But why not support other platforms?

Of course, there's a first target for this plan: Google's Android. It's Java based, so we just need to change some things in the sync engine and ui engine and applications could be written once and deployed both on Java ME and Android.

But there's Windows Mobile also, and we could even benefit by creating a Symbian specific port of the platform (but that is a low priority one since Symbian phones have all support of Java ME). But this last two are very tied to the next point:

Real multi-platform applications

One of the latest additions to the OpenBaseMovil platform was the scripting engine. We created it to build certain parts of the applications, but we're sure that it can be used to build full applications. Applications that would be really platform independent and dynamically downloadable.

Since the scripting engine could be easily written with C# for .NET or C++ for Symbian, we could deploy script-based applications to almost any platform.

Of course, this would need the sync engine to be also ported to .NET, for example, but that is not a challenge since it would be pretty the same that the one we are planning for Android.

The consumer market

The scripting engine is the core of the OpenMidsets project, a host application for script-based applications. Very much like Nokia Widsets or Yahoo! Go 3, but with a database, connectivity to bluetooth devices, and absolutely open (it is Open Source and you can host and connect your application to wherever you want), you are not captive, not locked.

We will develop some applications for OpenMidsets once it's available, and we hope that you take the challenge and write your own applications.

If you have read this far I'm really glad, I hope that the story is an interesting one and that it has helped you to better understand the project and the different alternatives.

Please, submit it to you preferred social site if you liked it, and feel free to comment. We're always open to opinions and fruitful dialog.

Hi folks,

All the feedback that you are giving us is great, and we've thought that a better place for discussion is a forum, because the post comments tend to create off-topic discussions that are difficult to track and less useful to everyone.

So we have created a forum. It's now empty but for the few categories that we have created, but we hope that you find it useful and participate, suggest new categories, etc.

For support request, bug reports and feature requests the best place is  the SourceForge tracker. It helps to keep open issues controlled and centralized.  But for development discussions the best place is a forum, and we are looking forward for your collaboration and input.

You might know from the site comments or the SourceForge bug tracker, that the current ui library has some issues with Nokia S40 3rd edition phones since this models use 3 commands instead of the normal 2.

We were planning some major changes in the user interface library, and we are now finishing the first part of it that addresses this issue.

The new ui library will use the concepts of Desktop, Window and Decorator, managing the screen in full screen mode and handling all the painting.

The Desktop is like the LCDUI Display class, it handles the painting of the window title and the commands.

All the classes that were previously Canvas subclasses such as the dialogs, menus and data lists are now Windows that get painted in the Desktop.

The desktop is divided in four regions: title, content, first command and second command. The commands are optional, you can have none, one, two or more.

Each region has an assigned Decorator, which is responsible of drawing the background and managing the overall aspect of the region.

The library will provide a default decorator that simply paints the background with a solid color, and also an experimental one that will draw an image as the background using any mode ( mosaic or stretch).

Thanks to the way the views are defined (with XML), you don't need to change anything in your application (if you use the default decorator) just compile it against the new ui library.

We will publish soon a beta version of the library and post it here for download.

This is a minor, though important, change.

Nik reported a bug that said that resource-packer would not run on Windows platforms, that in fact was affecting the viewcompiler also.

It is now fixed and tested on Windows XP Pro. We did not remember that there still people (developers) using Windows

We have not changed the release on sourceforge, it remains 3.0, but we have removed the binary and source files named 3.0 and uploaded new ones with version number 3.0.01:

openbasemovil-3.0.01-bin.zip

openbasemovil-3.0.01-src.zip

We have also added a new binary file called openbasemovil-3.0.01-buildtools.zip that contains only the jar files for the build tools, so to fix the bug in the binary distribution you can download only this one and override the libraries in the 3.0 files.

Sorry for the inconvenience, and thanks for your help.

PS: Try Ubuntu! you wont be disappointed.

Thanks to everyone, and thanks to the SourceForge staff.

It is ephemeral, but it's great!

The project news for the OpenBaseMovil 3.0 final release is now in the SourceForge home page.

Just for the posterity a snapshot:

We were thinking about which application seamed the best choice for creating the Quick Start Guide and the sample application.

It should be not too big, but useful, and make use of a good deal of the platform.

It should also be helpful to compare the effort needed to create an application from scratch and using our framework.

So we finally choose PasswordKeeper, a KeePass like application that has J2ME version at source forge (KeePass for J2ME).

Here are a few quick facts about the size of the project, that I think that speak aloud:

UPDATE: the table above represents some statistics about the SOURCE code of both applications, not the final .jar size of the application itself. We are trying to picture the development effort.

The number of 807 between parenthesis is the approximate number of lines of code that are actually part of the application and not of the template, so the ones that you really create.

Have some comments?