TPersistent.com

In my last post I talked about how unfortunately Spring4D does not provide change notifications so that developers can easily determine if an object graph has been modified and needs to be persisted.  It was primarily for this reason I decided to change the persistence layer in my latest project to hcOPF.  Obviously, I’m more familiar with this ORM, and I know it’s viable because systems I have authored 10 years ago are still being used without issue.  Even though I am using Delphi Rio 10.3.2, with lots of new language features, I don’t have time to thoroughly investigate and learn other alternatives. Change notifications will allow me to implement data auditing, and the validation & binding framework functionality will ensure any data editing is easily crafted.

Unfortunately, IBX no longer works well with Firebird 3.04 even if you can actually find a version for the latest Delphi editions.  I already ran into issues with UIB supporting larger varchar columns so that was not an option.  Instead, I decided to implement support for FireDAC, in part because it had been requested and because my Spring4D implementation with FireDAC proved to be performant.  The implementation was basically a copy/paste/modify of another DAL layer.  It currently does not support StoredProcs, but queries work. More testing and additional unit tests are needed, but it should allow developers to get started using FireDAC.

I also added support for quoted column names which is a Firebird requirement if you use mixed case names.  I’ve been dogfooding it for about a week now, and ironing out a few issues encountered.  I’m pleased with the performance, as hcOPF seems to be faster than my Spring4D ORM implementation.  If I get time I will try to put together some benchmark comparisons.  My only explanation for this would be Spring’s use of RTTI.

FireDAC was added for Rio and then Tokyo. Projects should be easy to back port to other recent Delphi versions. There is one demo specific to FireDAC in the Demos folder with projects for Rio. That should show you how to get started, and it’s essentially the same for all DALs;  create a datamodule, drop a connection and all the hcOPF components on it, link them, and configure your database connection.

FireDAC DataModule Components for a Firebird Database

Spring4D’s ORM uses an instance of an IEntityMap to perform change tracking. The default implementation is essentially a threadsafe wrapper around a Dictionary that uses an entitykey of the classname + ‘$’ + the object Id and the Value portion is an array of member values. If an object instance is present in the EntityMap, IsMapped() returns True.

As objects are mapped from ResultSets, they are added to the the EntityMap capturing their values as read from the database. If you want to see what changes were made to the objects you must call GetChangedMembers() which returns a list of attributes for columns that have changed. In effect Spring4D keeps two copies of the data for a given object: the data in the object instance itself and the EntityMap, which is a reasonably compact version of the data as it was originally read from the database result set.

hcOPF takes a different approach, in that each object instance essentially has 2 identical field variables for each attribute and a boolean field to track Nulls. The advantage of this approach is that you can easily test each attribute to see if it’s NULL or has been changed, in user code without testing all of them. Since the implementation does not use RTTI it is likely faster, and hcOPF Attributes surface change notifications. Of course the trade off is larger object instances.

Change Notifications are a very powerful mechanism for both the framework and developer code. They can be used to trigger attribute level validations, provide GUI feedback of modification status and provide for very granular efficient calculations on attributes. In fact these change notifications are the basis for hcOPF’s ability to save an object graph to the database by simply calling object.Save on the root object.  Some frameworks like mORMot have even implemented database auditing tracking.

Sadly, Change Notifications are something missing from the Spring4D ORM. If you want to know if a an object has been Modified you can of course add a FModified boolean field and set it in each property setter after checking if the value being set differs from the current value. It’s a roll your own state tracking that you would expect the ORM to do for you, and it doesn’t take into account situations where the user modifies an object property and then changes it back either by Undoing the change, or changing it back to the original value.

In developer code, change notifications can dramatically simplify logic to save data when the application allows users to modify numerous root objects with the ability to undo changes. It can become quite difficult to determine what method calls are required to save all changes with Spring4D when you don’t know what has been changed in the first place.

Build of course, like most tools, only compiles files that have changed since the last time a compilation was performed.

Rebuild of course, like most tools, re-compiles all source files regardless of whether they have changed since the last successful compile or not.

Simple right?  Not exactly….there are some subtle undocumented differences.  If a configuration file such as an appsettings.json file exists in your project, it is normally meant to be copied to your output folder for use by the application when you run it.  If you update this file, and Build or run the project VS.NET does not update the file in the output folder.  This has bewildered me on more than one occasion.  Only when you perform a Rebuild on that project (or solution) does the newer file get copied to the output folder.

Some intellisense errors in the IDE code editor involving code from other assemblies are also only resolvable by performing a rebuild.  I don’t recall the specific circumstances, but suffice to say, when in doubt perform a Rebuild.

This post is not about DoorDash or SkipTheDishes, it’s about using the Where IN() clause in SQL. Yesterday I thought I had discovered a way to order the inclusion of child records since this StackOverflow answer led me to believe that ordering values in the IN() clause will in fact order the resultset in Firebird. Unfortunately, in my testing with Firebird 3.04 this is not the case.

So beware, StackOverflow can be a great source of information, but it’s not always correct. It’s also of course better to explicitly order results rather than rely on the behaviour of a query processing engine so I will be re-writing the Marshmellow code used to generate the query.

In case you have not been watching RSP-17195 Marco just closed the ticket yesterday. Apparently EMBT has decided that the Linux compiler will remain an Enterprise edition feature and will not appear in the Pro edition as requested.

I have written before as to why I think this a major marketing mistake, so just briefly, Linux is used by more hobbyist programmers and IoT/SBC developers than most other OSes, and there are many FOSS solutions available. Few people interested in targeting Linux will pay for the Enterprise Edition as they can target a Windows machine and pay the M$ license cheaper than buying an Enterprise edition. It also makes no sense to allow the targeting of Windows and MacOS in the Pro edition, but not the other desktop target. I think this is the same mistake that was made when originally selling the Mobile AddOn, which EMBT later decided to include in Pro in order to limit development costs and increase the user base. Licensing FMXLinux will likely not amount to much of an incentive as my understanding is most developers who have currently licensed the Enterprise edition are writing server apps. Increasing the cost of writing a desktop app will not encourage the uptake of hobbyist or professional developers looking to monetize apps on a platform that users typically prefer FOSS solutions.

Other vendors like REMObjects do not charge additional money for a product edition to target a supported platform. You get all platforms for the same subscription price, and they have now written an RTL and WASM support which makes their platform that much more attractive to Delphi developers. REMObjects also has 64 bit ARM target support today.

.NET cannot be ignored now that .NET Native and Core have been released and are rapidly maturing. Delphi’s “native” compilation is no longer an advantage and the C# code generated tends to be more efficient than Delphi’s various compilers by all accounts.

You can have the greatest product in the world, but if you cannot sell it to a sufficiently large user base, it will not be self sustaining. Marketing decisions like this IMHO will negatively impact the Delphi user base.

Just announced, FMXLinux has been licensed from KSDev for inclusion in the Enterprise and Architect editions of RAD Studio and related products (Delphi & C++ Builder).  The product is available via GetIt for all current subscribers.

Since I have been following FMXLinux’s development for some time, I immediately installed the licensed version.  Unfortunately, the IDE experience is the same as the trial I previously installed directly from KSDev.

I created a new Multi-device application with tabbed navigation from the wizard.  Looking at the list of target platforms (Android, Mac OS, iOS, Windows) I didn’t see Linux.  Fair enough, the templates on which the application is based do not contain the new UI platform.  Perhaps in a subsequent release since as per Marco’s introduction the product has not been fully integrated into the IDE.

I went to add the new platform to the solution, by Rt. clicking The Target Platforms node in the project manager and it is greyed out.  If I delete a target it is re-enabled, but the list of targets I can add only contains the one I just deleted.  Rt. Clicking on the Application node in Projects window, there is an option “Add Linux Platform”.  Now I get a Linux 64 bit target platform node.

This is covered in Marco’s introduction, but he doesn’t mention that attempting to add a component to my UI doesn’t work.  I can’t add a button to my form until I switch to another target platform, and then switch back so I can actually deploy and test on Linux.  This has been a known issue with the trial, and I was hopeful that such a poor user experience would be fixed now that EMBT has licensed the product.  For now, it’s an issue you will have to work around, and I’m sure it will be corrected in a future release.

While only major Linux distros are currently supported (Ubuntu, RedHat and possibly a couple others) the team at KSDEV are working to support others including the more obscure ones like Elementary OS.  I for one am very pleased that Delphi finally can target Linux with GUI applications making it a complete XPlatform desktop solution for all 3 major targets.

I’m relatively new to the Android development platform, coming from the Windows desktop. One of the great things about recent editions of Delphi is it’s ability target numerous other platforms. One of the worst things about recent versions of Delphi is it’s ability to target platforms you are not as familiar with. If your use case is atypical, you can quickly find yourself swimming in shark infested waters. You never know what is going to byte you, a bug in the platform, a bug in Delphi RTL or FMX code, a bug in your code, or just your lack of knowledge about the platform.

Today I was trying to figure out why files I was saving to the Documents folder on my Android LG5 were visible from the Android File Manager, but not present when I connected the phone to a Windows, or Ubuntu desktop. Turns out there is a long outstanding bug in Android that google has not even acknowledged. In order to see new files when connecting to a device using the MTP protocol, you may need to re-boot the Android device before the files appear. Indeed once I did so, my files appeared and I could copy them off to my PC. You may want to comment on this motion to get the issue re-opened and addressed.

Hopefully this helps someone else avoid spending a lot of time scratching their head…

Just a note to EMBT customers, JIRA has been down for several hours now.  It is currently giving the following message

HTTP Status 500 - org.ofbiz.core.util.GeneralRuntimeException: Could not determine database type. (Network error IOException: Connection refused: connect)

With a Java callstack if you like that kind of thing.  EMBT has been advised and I’ve been told EDN may also be affected. Hopefully it will be back up shortly.

Today I decided to ensure the project I am working on is in fact database independent. The best way I have found is to make sure you develop and test on multiple databases as you go. Since the ORM portion of Spring4D often called Marshmellow is supposed to isolate your application from the underlying data store, I figured it would be a piece of cake.

The Spring4D ORM supplies numerous database adapters including UIB (Unified Interbase). There is also another I found for IBX. I decided to try UIB since I want minimal deployment dependencies and am using Firebird, but was a little concerned since the last commit was from Jun 13, 2016.

After pulling down the source code from the repo I discovered that there are no projects for Delphi Rio, so I copied the projects for the last version (21) and renamed them to create Rio projects. Getting the components installed proved quite easy, but once I had the basic data ORM setup code for an alternate database incorporated into my project, that’s when the fun started.

Initially UIB would not load the GDS32.DLL from c:\Windows\System32 that was generated from my Firebird 3.0 Win64 install. Providing the full path did not resolve the issue, and GetLastError() wasn’t helpful so I changed the client library to fbclient, and the app crashed a little further down the line.

The ORM uses a TDatabaseManager class to build the database structures from the registered entity metadata when you call the BuildDatabase method.  It worked for SQLLite, but did not for Firebird as the database file did not exist. Unfortunately the TDatabaseManager was never designed to be extended. The class contains only private fields with no protected accessors, and the constructor as well as the BuildDatabase method, are not virtual. As a result I wrote the following decorator class to provide the extended functionality:

type
  TUIBDatabaseManager = class(TObject)
  private
    FConnection :IDBConnection;
    FUIBConnectionAdapter :TUIBConnectionAdapter;
    FDatabaseManager :TDatabaseManager;
    procedure CreateTheDatabase;
  public
    constructor Create(const connection: IDBConnection);
    procedure BuildDatabase;
  end;

implementation

uses
  Spring.Persistence.SQL.Interfaces, {for TQueryLanguage}
  uiblib;

procedure TUIBDatabaseManager.BuildDatabase;
begin
  //if the database does not exist then create it
  if not FileExists(FUIBConnectionAdapter.Connection.DatabaseName) then
    CreateTheDatabase;

  FDatabaseManager.BuildDatabase;
end;

constructor TUIBDatabaseManager.Create(const connection: IDBConnection);
begin
  FConnection := connection;
  FConnection.QueryLanguage := qlFirebird;
  FDatabaseManager := TDatabaseManager.Create(FConnection);
  FUIBConnectionAdapter := connection as TUIBConnectionAdapter;
end;

procedure TUIBDatabaseManager.CreateTheDatabase;
begin
  FUIBConnectionAdapter.Connection.CreateDatabase(TCharacterSet.csWIN1252);
end;

After I got the database created, the application bombed because it was attempting to create boolean object fields as BIT database columns. Tracing through the code I found the where the datatype mapping was performed and added a snippet for FireBird 3 support after descovering that UIB already supported the Boolean datatype for Interbase and Firebird. All I had to do was compile UIB with the FB30 directive. Here is the amended version of the GetSQLDataTypeName method:

function TFirebirdSQLGenerator.GetSQLDataTypeName(
  const field: TSQLCreateField): string;
begin
  Result := inherited GetSQLDataTypeName(field);
  {add support for Firebird 3.0/Interbase 7 new boolean datatype}
  {$ifdef FB30}
  if (field.TypeInfo.Kind = tkEnumeration) and
     (field.typeInfo = System.TypeInfo(Boolean)) then
    Result := 'BOOLEAN'
  else
  {$endif}
  if StartsText('NCHAR', Result) then
    Result := Copy(Result, 2, Length(Result)) + ' CHARACTER SET UNICODE_FSS'
  else if StartsText('NVARCHAR', Result) then
    Result := Copy(Result, 2, Length(Result)) + ' CHARACTER SET UNICODE_FSS';
end;

Once I got that issue resolved the app crashed on a SQL snippet I had in a call to the ExecuteSQL method because Firebird likes quoted identifiers when you use mixed case column names. Once I had worked around that I found that Firebird threw many more errors importing data due to my Column attributes than SQLLite had. It also complained because I had used a reserved word in Firebird as a table name. After a some more fixups I had a running application capable of using either database back end. In the process I re-discovered how much I like Firebird, even if some of it’s SQL error messages are rather cryptic. It is also much faster importing data that SQLLite.

My blog tagline is certainly no accident.  I have been interested in persistence frameworks for a long time, and thought I would use Spring4D’s Marshmellow ORM for a project.  Spring4D has been around for quite some time and just had it’s first conference in Italy so I figured the framework was mature and warranted a closer look.  I had previously used the Spring4D DI container, and decided this time to use Spring4D collections as well to avoid as much code bloat as possible.

The first thing I discovered is that the Spring persistence layer aka Marshmellow has a very unfortunate name.  Trying to google it with “delphi” leads to all sorts of hits related that Android version.  The next thing I learned is that development has been put on hold as of September 2018 due to a lack of resources.  This is unfortunate, as there are not that many open source ORMs that use the newer language features of Delphi when compared C# or Java.

The third thing I quickly learned is that other than the Tests, there is not a whole lot of documentation available.  There is the reference help which is really not that helpful.  It doesn’t contain descriptions of the class interactions or architecture and has no examples of usage.  Really it’s not much more informative than drilling through the code.  The best source of “getting started” help that I could find is the previous bitbucket repo. There is of course also the google groups if you need to get clarification of something, and reading previous posts can help you from running into common issues.

As a newbie, it’s not clear how the [InheritenceAttribute()] works so I will have to investigate it further. I assume if the last descendant in a class hierarchy is the only one marked as an entity, it will by default have all the fields marked with the Column() attribute in all ancestor classes.

Another thing that is not particularly clear from any documentation is how the underlying datatype employed by the database is determined.  There are really only 3 pertinent parameters supplied to the Column() attribute, namely length, precision and scale.  Length only applies to string types and precision, scale to numeric.  It is unclear how precision and scale effectively change the numeric datatype and it’s corresponding precision or scale, other than for Integer datatypes you specify 0,0 for precision,scale.

I’m sure all this will become clear as I use the framework more, but some more thorough ‘Getting Started’ documentation would have been really nice.