ARM Yourself

August 25th, 2021

These days as the Pandemic continues to disrupt society, reducing individual rights, many people are buying guns and preparing for the end of days.  Fewer in Canada than the US as the Trudeau government has seen to it that more guns have been banned, even if it makes no sense.  Certainly the future looks darker than ever in terms of the environment, politics and the economy, but this article is not about any of that or about guns.  It’s about ARMs that you can and should buy.

I bought an Apple M1 Mac mini and the thing is fast! I would highly recommend it as a dev box.  Apple’s M1 has made it perfectly clear that the detractors of the RISC CPU architecture were mistaken. The Power PC processor may have been anemic, but the M1 is not, and after reading Arnaud’s article it’s evident that Apple is not the only player raising the bar in the ARM market.

CISC is not superior, because the future is in parallel processing with lower power consumption.  RISC dominates mobile and is now poised to take over the server side aka cloud market.  With Apple switching their desktop CPUs to ARM chips and seeing companies like Ampere producing CPUs targeting the server market, I think it might be time to sell Intel and AMD stock.  I think it’s only a matter of time before ARM gains a significant share of the desktop and server markets.  Unfortunately, Ampere does not seem to be public.  Other than Apple, has anyone got some suggestions for good ARM stock?

hcPluginFramework Released

June 21st, 2021

I just open sourced a dynamic application framework based on code from Mark Miller’s Borcon 2000 presentation (with his permission).  If anyone remembers CodeRush, the original product developed in Delphi for Delphi, it used the same plugin architecture.

Eagle Software’s Coderush was a revolutionary product that enabled developers to write code quickly using it’s own templating system, bookmarking, navigation and many other advanced capabilities.  I first saw it when a friend of mine, Kevin Meier, introduced me to the version he was using with Delphi 3.  Sadly, Mark sold the IP to, and went to work for, DevExpress who discontinued the product, rolling the core ideas into Coderush for .NET.  Delphi 7 was the last version of Delphi that Coderush supported.  I always wished that DevExpress had open sourced CodeRush.  Not only did it provide a boatload of functionality, but it enabled developers to easily create their own plugins to extend Delphi or Coderush functionality.  This was all back in the day when the OpenToolsAPI was undocumented Voodoo.

The class based interfaces of the plugin framework are defined in the PluginInterface package in the form of abstract classes.  In the ApplicationCore package, descendants of these abstract classes implement the actual functionality.

One of the cool features about this architecture, is that it allows all the plugins to be updated while the application is running.  This is achieved by the ThcCorePlugInLoader found in the dclhcPlugins design-time package.  The ThcCorePlugInLoader component loads the ApplicationCore package and can unload it at run-time.  This means you can update the ApplicationCore as well as any other package it loads, making it possible to replace all the functionality of the application while the shell is still running.

Included in the framework is a an About package that enables users to add/remove and unload packages at run-time.  If you have the need to supply customer tailored solutions for your product, hcPluginFramework will help make that more manageable.

Doing it with Style

April 29th, 2021

I’ve been on many teams over the years and found that Coding style is a subjective and often heated discussion.  Different language backgrounds often result in different styles that can make code reviews a contentious choke point.  Usually the team has to come up with their own style guide with compromises to accomodate it’s members, but no one wants to create a comprehensive guide from the ground up.

Usually the old Borland style guide was used as the basis for establishing the code style and code formatter settings used by the team.  In recent years it has been more challenging to find that guide, and let’s face it, like everything else it probably needed an update.  The good news is that there is now a style guide section on the EMBT DocWiki which will likely be evolved over time and is of course much easier to find.

Object Pascal has survived for over 25 years, in a large part, because it is a very readable language.  Consistent code formatting helps.

hcUpdateFramework - The Server Side

February 14th, 2021

As promised, in this article I will be describing the server side components of the hcUpdateFramework, namely:

  1. ClientUpdateService
  2. Launcher
  3. Register
  4. AutoUpdateServer
  5. Update Manager

4. AutoUpdateServer

The AutoUpdateServer is a Windows application that exposes the IUpdateService SOAP service on port 8080 by default.  It can be compiled as an ISAPI DLL, a Win32/64 VCL Application or a Win32 NT Service running under the local system account.  Most of my implementations have simply used the VCL server application so YMMV.

The server process sends an update to the ClientUpdateService when it checks using the IUpdateServer.GetUpdate method. The service creates a default INI file the first time it is started, if one does not exist. The INI file contains the following settings:

[Config]
MaxConcurrentWebRequests=32
ListeningPort=8080
DatabaseFileName=C:\Data\Deployment.fdb
UserName=<username>
Password=<password>

This service can be compiled as a Win64 NT Service, but currently throws an I/O related exception in the utility methods used for binary file streaming.

Updates are created in subfolders under the Updates folder located in the same directory as the update server.  All updates are static, meaning the contents of the update are determined when the update is created and not on a per request basis from the client (although that is technically possible). In order to reduce the bandwidth and server utilization, binary patch files are supported using Microsoft’s delta compression technology.

Currently, there are two databases supported; SQL Server (via ADO) and Firebird 3.X (via FireDAC).   The SQL Server schema needs to be updated to mirror the changes made when migrating the server to Firebird.  Once that is complete, the FireDAC datamodule could be modified to support both databases.  Scripts for all supported databases can be found under the SQL folder, including scripts to create new deployments in the database without using the Update Manager.

5. Update Manager

The UpdateManager is an application I wrote to automate the creation of updates for a given application.  The framework defines an application as a primary EXE which is entered into the list of registered applications in the Deployment database and whose version drives the update version (they usually match).  An application deployment may contain other EXEs, and supporting files which are detailed in the XML Manifest file which contains:

  • UpdateVersion - version of the Update (often matches primary EXE)
  • WhatsNew - RTF description of what has changed in this update
  • IsMandatory - update must be applied (user is not given a choice)
  • IsSilent - update is applied without presenting a UI (ClientUpdateService)
  • IsImmediate - update is applied immediately after download (ClientUpdateService)

1..N of the following where N is the number of deployment items aka application components:

  • FileName - target filename
  • Version - version of the file
  • TargetPath - location to copy the file into in case it differs from the application folder
  • IsAPatch - this file is a patch file to be applied using MS Patching
  • IsAZip - this file is compressed using the Zip archive format
  • Launch - launch the EXE when applying the update

An example manifest is:

<Manifest LocationGUID=”2bc5cc04-2307-41ab-9f3c-b3f8c0578be9 ApplicationGUID=”8BF6627D-CB05-4601-996E-072DC6785F81 UpdateVersion=”1.0.0.3 WhatsNew=”Incorporated Auto Update IsMandatory=”false IsSilent=”false IsImmediate=”false>
<Item FileName=”MyApp.exe Version=”1.0.0.3 TargetPath=”{app} IsAPatch=”false IsAZip=”false Launch=”false/>
<Item FileName=”App Manual.pdf Version=”1.0.0.0 TargetPath=”{app} IsAPatch=”false IsAZip=”false Launch=”false/>
<Item FileName=”Manifest.xml Version=”1.0.0.3 TargetPath=”{app} IsAPatch=”false IsAZip=”false Launch=”false/>
</Manifest>

An Update on the server side consists of a root folder called Updates under which subfolders are created for each update version using the same naming convention used on the client as shown:

Updates Folder Structure Server Side

Updates Folder Structure Server Side

Within each update folder, there must be a manifest and all files appearing within that manifest.  Once the update folder is created with all the necessary content, a new deployment record must be created and new InstallationDeployment records need to be created in the database and then enabled to distribute the new deployment/update to existing installations/users.  The SQL to do so is present for Firebird so updates can be manually crafted, but it is much easier to use an UpdateManager.  Unfortunately, the current implementation requires the DevExpress ExpressQuantumGrid and Editors.  With the database changes made during migration to Firebird I will also need to update the hcOPF domain objects to reflect the current structures.

The hcUpdateFramework needs some TLC and can certainly benefit from some new ideas, but it is a functional, proven update framework that can be extended in many ways to facilitate your deployment needs.  I am open sourcing it, in the hope that others will contribute and make it a more complete, modern solution.

hcUpdateFramework - The Client Side

February 10th, 2021

The hcUpdateFramework consists of 5 parts:

  1. ClientUpdateService
  2. Launcher
  3. Win/Register
  4. AutoUpdateServer
  5. Update Manager
The first 3 of which reside on the client machine.  In this article I will discuss this half of the framework.

1. ClientUpdateService

The ClientUpdateService is a Windows NT Service that runs under the local system account and is responsible for downloading updates from the AutoUpdateServer to the client machine. All update processing occurs in subfolders under the application installation folder which is typically C:\Program Files\{CompanyName}\{ProgramName}. The update folder structure consists of a subfolder called oddly enough; Updates which has 3 subfolders; Pending, Applied and Backup.  Please note that the initial implementation assumed Administrator rights to the installation folder.  For that reason, it may be necessary to install the application into a user folder (ie: C:\Users\<UserName>\AppData\Local\) if the user is running with Standard User permissions.

The ClientUpdateService polls the AutoUpdateServer at a predetermined interval (defaults to 15 minutes) which is specified in AutoUpdate.ini. A default INI file is created the first time the service is started, if one does not exist.

The INI file specifies the URI of the AutoUpdateServer, in addition to the polling interval.  The same INI file is used for all client side applications, and has the following settings:

[Shared]
UpdateRootDir=C:\Users\<UserName>\AppData\Local\MyCompany\MyApp
AppDir=C:\Users\<UserName>\AppData\Local\MyCompany\MyApp
UpdateServiceURI=http://www.myserver.com:8080/soap/IUpdateService
InstallationGUID=3B72F928-4B46-4098-942C-5309CC38ACF4

[Launcher]
AppToLaunch=MyApp.EXE

[Service]
LogAllMessages=1
SleepTimeInMinutes=15

The ClientUpdateService sends the current application manifest to the AutoUpdateServer as part of the IUpdateService.GetUpdate request. If an update is available, the AutoUpdateServer provides the update components with a new manifest as part of the response. The ClientUpdateService saves the new manifest, and all files in the response into a new Updates\Pending subdirectory named for the UpdateVersion. The service then tells the AutoUpdateServer that it has successfully received the update by calling the IUpdateService.UpdateReceived method.

This method passes the ApplicationGUID, LocationGUID, and UpdateVersion to the server. Afterwards the service sleeps for the specified interval before checking for another update. The ClientUpdateService will never download the same update more than once, because the AutoUpdateServer knows it has already received the update.

The ClientUpdateService has the ability to apply the updates, but normally user interaction is desired.

2. Launcher

The Launcher is a Win32 client application that is responsible for actually applying any downloaded updates, and providing the user with appropriate feedback. The Launcher can be renamed so it will replace the current application if it has already been deployed, and act as an intermediate EXE ensuring all existing shortcuts still function. It creates a default INI file the first time the program is started, if one does not exist. The INI file specifies the program it is to launch if no update is present, or after the update(s) have been applied, the application install directory, where to look for updates, and the URI of the AutoUpdateServer.

Although the functionality has not been tested recently, the Launcher is capable of applying multiple updates at once. It should be verified that it applies multiple updates in the correct order before relying on this functionality. On startup, the Launcher reads all subdirectories that appear under Updates\Pending. It then applies the update by loading the manifest, backing up each file into a subdirectory it creates under Updates\Backup\X.X.X.X (where the Xs represent the UpdateVersion), and copying each file in the manifest into the application install directory. If the manifest indicates that the file is a patch file, the patch file will first be used to create a file with the extension .NEW. The patched file will then be used to replace the version in the application install directory.

Errors are trapped along the way, and if any error occurs, the user is informed that the update failed and application may be unusable. No attempt is made to rollback the update. It is left for manual intervention in part to ensure tech support discovers why the update process failed, and because any automated rollback attempt is just as likely to fail. A complete copy of the previous version of all updated files can be found in Updates\Backup\X.X.X.X. Restoring users to a running version of application should be as simple as copying these files back into the installation folder, provided no database updates have occurred that would prevent the backup version from running.

If an update is successfully applied, the Launcher informs the AutoUpdateServer by way of the IUpdateService.UpdateApplied method. This method passes the ApplicationGUID, LocationGUID, UpdateVersion, UpdateResult, and UpdateLog to the server which records the information for viewing in the Deployment Manager.

3. Win/Register

There is actually a Console and VCL Windows utility which are a relatively recent addition the framework.  This utility is meant to be bundled and called as part of an initial install. It calls the IUpdateServer.RegisterInstall method with the ApplicationGUID.  Each application to be deployed is identified by a unique GUID.  This method returns an InstallationGUID which is saved in the application manifest for identifying the installation to the UpdateServer for future updates.

Since this article is getting a little long, in my next post I will cover the server side of the framework.

hcUpdateFramework is Released

February 4th, 2021

A long time ago, in a place not so far away, I wrote a framework which allowed a corporation to update an application it had deployed to 100+ remote locations with slow internet connections. Back then there were a few alternatives to rolling your own, but nothing seemed to have the feature set and development commitment from a Delphi component vendor. As a result, it was decided that I should implement one.

The entire AutoUpdate Framework was initially written in Delphi XE2. Although BITS was going to be used to transfer the files since it offers the ability to throttle bandwidth, in the end, for simplicity sake, web services were employed instead. Rather than a simple call to download an MSI, the framework provides a large degree of flexibility:

  • Do you simply want to generate and apply a binary patch?
  • Do you need to be able to apply a series of updates to bring the installation up to the current version?
  • Do you want to compress/zip the files to be delivered?
  • What about deploying additional files and removing obsolete ones?
  • Do you want to display a list of changes so the user can choose whether they update?
  • Do you want to notify the user that updates are available while they are using the application?
  • Do you want to make the update mandatory so bug fixes that are causing data issues are squashed ASAP?
  • Do you want to schedule the application of an update silently in the background at a given time in the middle of the night?

All these scenarios are supported and with a custom deployment application, you can see what version each user is running, any errors in the update process and when they last launched the application. It would be easy to extend the Server to accept bug reports and create an integration to your bug tracking system and/or CRM.  Usage Telemetry could also be added, and if user’s don’t pay, an update can render your software inoperable.

Recently I resurrected the code to provide updates to a client during COVID, so I added Firebird support and changed the data model to track registrations (new application users). It would be very easy to extend the system with your own custom licensing code and track where your new users are located and how many “conversions” from trialware or lite to full versions you’re getting.

That’s the power of open source;  no waiting for a vendor to implement the features you want for your use cases.

In subsequent posts, I will describe the moving parts of the system.

Improving Your Form - a Decimal cannot fit into a Decimal

February 1st, 2021

If you have ever gotten an error message like “DATABASE ERROR: [FireDAC][Phys][ODBC][Microsoft][SQL Server Native Client 11.0][SQL Server] Error converting data type decimal to decimal” you might be thinking it’s a bug in FireDAC or whatever database layer you’re using.  The two types are the same so no “conversion” should be necessary and how can assigning the same type to itself fail?

In fact, in the cases I have encountered it was because the value being written to the database exceeds the range (positive or negative) of the column as defined in the database.  For example, attempting to write 100,000 into a column defined as decimal(9,4) would fail.

Ideally the database layer would provide a more precise error message, something that perhaps included the value being written with the name, type and size of the database column.  I’m sure there are some technical reasons for such obscure error messages.  No doubt multi-vendor database support is likely the major one.  If the database doesn’t provide the information no DAL can pass it on.  Crafting meaningful error messages also requires a level of skill, forethought and empathy.

This is exactly why applications really need some sort of framework for UI/Domain level validation for at least minimum and maximum values so monkey testing cannot break the app.  If every form in your application needs to implement validation logic, keeping it consistent over time and not forgetting some control(s) is problematic. So the first step to improving your form is to put all the validation and business logic elsewhere so it’s re-usable. That doesn’t mean you have to go full MVVM with domain objects. A good start is to use an ORM.

Thread Safe ID Allocation Across Multiple RDBMSs

January 13th, 2021

Believe it or Not, I have seen enterprise applications that simply perform a ‘select max(ID)+1 as NextID from <TableName>‘ in SQL against their RDBMS to get each and every ID value.  While this works, in a multi-threaded application, or on a busy SQL server this causes a lot of network traffic.

However, the biggest problem with this approach is that it doesn’t prevent a ‘primary key violation’, since two applications may query the database at the same time and be provided with the same max ID value.  When they both go to write their records, the first writer wins, and the second one gets an exception.  Unless your application traps the error and gracefully handles such a situation (retries everywhere?), the application fails.

Now ID Allocation is such a fundamental building block you would think applications built as part of a suite would use the same code for ID allocations, and that such potential problems would not be present in successful commercial apps that support both Oracle and Microsoft SQL Server.  Guess again!

Oracle has sequences so often ID values are generated on insert, which complicates establishing master/detail relationships.  It is easier to know what ID values are going to be generated for the records especially when the client applications use dynamic SQL.  Microsoft SQL Server has identity values which have their own drawbacks, so an earlier decision had been made to avoid them.  SQL Server has only recently adopted sequences, so supporting databases on older versions still necessitates a different approach to an RDBMS agnostic method of allocating IDs.

What we decided to do is provide the application with permissions to enable it to modify the next value of a sequence on Oracle, and create a table in SQL Server to manage the ID values used there with the following structure:

CREATE TABLE [dbo].[IDAllocation](
    [TableName] [varchar](30) NOT NULL,
    [LastUsedIDValue] [int] NOT NULL,
    CONSTRAINT [PK_IDAllocation] PRIMARY KEY CLUSTERED ([TableName] ASC)
)

Then I created a new interfaced class to implement the ID allocation client code. Since we needed to prove this approach would work in production, I implemented it for a subset of tables that were populated with multi-threaded code I developed, which exposed the flaws in the previous approach to ID allocation. This code creates a server side ADO query that updates the ID value and returns the next one:

unit IDAllocation;

interface

type
  IIDAllocation = interface['{53F7BA50-0D72-4AC1-AB9B-6E5210DD4543}']
    function GetNextId(const TableName: string; IncStep: integer = 1): integer;
  end;

  TIDAllocation = class(TInterfacedObject, IIDAllocation)
  private
    //an interface that provides the database connection and queries using that connection
    FDataSession: IDataSession;
    FQuery: IQuery;
    FTablesSupported: TStringList;
  public
    /// <Summary>
    /// Get the Next ID value for the table specified.
    /// </Summary>
    ///
    /// <param name="TableName">The table for which new ID(s) are needed.</param>
    /// <param name="IncStep">The number to add to the LastUsedIDValue.
    /// In order to reserve a range of IDs just set this increment to
    /// the largest value desired (ie: dataset recordcount) and then
    /// decrement the value for subsequent ID values.
    /// </param>
    ///
    function GetNextId(const TableName: string; IncStep: integer = 1): integer;
    constructor Create(DataSession: IDataSession); reintroduce;
    destructor Destroy; override;
  end;

implementation

uses
  System.SysUtils, Data.Win.ADODB;

constructor TIDAllocation.Create(DataSession: IDataSession);
begin
  inherited Create;
  FDataSession := DataSession;
  FQuery := FDataSession.CreateServerQuery(utResults, True);
  FTablesSupported := TStringList.Create;
  FTablesSupported.Sorted := True;

  // currently only some tables are supported for use with IDAllocation
  FTablesSupported.Add('Table1');
  FTablesSupported.Add('Table2');
end;

destructor TIDAllocation.Destroy;
begin
  FTablesSupported.Free;
  inherited;
end;

function TIDAllocation.GetNextId(const TableName: string; IncStep: integer = 1): integer;
const
  MSSQL_UpdateSQL
    : string = 'set nocount on; update IDAllocation set LastUsedIDValue = ' +
    '(select IsNull(IDA.LastUsedIDValue,0) + %0:d as ID from IDAllocation IDA where IDA.TableName = ''%1:s'' )'
    + 'where TableName = ''%1:s'' ' +
    ';select LastUsedIDValue from IDAllocation where TableName = ''%1:s'' ';
var
  sOwner: WideString;
  Command: TADOCommand;
  Recordset: _Recordset;
begin
  Result := -1; // compiler warning
  // since this is only available for some tables we must fallback to the "classic" generation
  if (FTablesSupported.IndexOf(TableName) = -1) then
  begin
    FQuery.SQL.Text := Format('select max(ID) as ID from %s', [TableName]);
    FQuery.Open;
    if FQuery.Fields[0].IsNull then
      Result := IncStep
    else
      Result := FQuery.Fields[0].AsInteger + IncStep;
    FQuery.Close;
  end
  else
  if FDataSession.IsOracle then
  begin
    if IncStep <> 1 then
    begin
      FQuery.SQL.Text :=
        Format('select TABLE_OWNER from user_synonyms where Table_name = upper(''%s'') ',['s_' + TableName]);
      FQuery.Open;
      assert(not FQuery.EOF, 'The owner of the sequence could not be found');
      sOwner := FQuery.Fields[0].AsString;
      FQuery.Close;

      FQuery.SQL.Text := Format('alter sequence %s.s_%s increment by %d',[sOwner, TableName, IncStep]);
      FQuery.ExecSQL;
    end;

    FQuery.SQL.Text := Format('select s_%s.NextVal from dual', [TableName]);
    FQuery.Open;
    Result := FQuery.Fields[0].AsInteger;
    FQuery.Close;
  end
  else
  if FDataSession.IsMSSQL then
  begin
    Command := TADOCommand.Create(nil);
    try
      Command.ConnectionString := FDataSession.ConnectionString;
      Command.CommandText := Format(MSSQL_UpdateSQL, [IncStep, TableName]);
      Command.CommandType := TCommandType.cmdText;
      Command.ExecuteOptions := [];
      Recordset := Command.Execute;
      if Recordset.Fields.Count > 0 then
        Result := Recordset.Fields[0].Value
      else
        raise Exception.Create('No Recordset Returned!');
    finally
      Command.Free;
    end;
  end;
end;


Why I Hate WebApps

December 31st, 2020

I have avoided deep diving web application development for a long time because the segment undergoes a lot of technological churn for really nothing.  IHMO web applications are better now than 10 years ago but they still do not rival the stability, performance, and usability of desktop applications.  In short Web 2.0 is a flop if you base your conclusion on widely available web applications. It’s hard to write a web application and get it right.  Right today, may not be right tomorrow since you have no control over the vehicle on which you’re hitching a ride; namely the browser. 

I was reminded of this again today when NoFrills redirected me to their flyer page in an email.  The page asked me to enter a postal code even though I was automatically logged into my profile and my preferred store was shown in the upper right.  I might have dismissed that, except the postal code entry control flashed on the page and disappeared.  You might think it was my browser, but I am using the latest FireFox and it did the same thing under Chrome.

I bank with TD Canada and their website often displays an error message after I log in.  If I login again, it is always successful, just the first time I quite often get the following error:

I needed to install CodeSite 5.2.2 so I could compile hcOPF under D7 and the EMBT portal provided the binary installer, but there was no corresponding serial under my registered products (who needs data integrity anyway).  So I decided to contact support, and went to their web page and chose the product and got an animated response. You may notice the disclaimer in white with a red background saying use another browser if you have issues with FireFox. I guess fixing it so it works in the remaining two browsers is too hard.

So if a basic entry form is too difficult to get right in 2020, I don’t think the web has progressed all that much since the 90s. Mind you the 90s rocked! Hopefully the 2020s will be better on all fronts despite the horrific beginnings.

Testing Options for Delphi

December 3rd, 2020

Depending on the version of Delphi in use, unit/integrate testing tools include:

DUnit
-the original framework with the latest version of 9.4.  Embarcadero stills ships this with Delphi.

DUnitX
-released by VSoft Technologies (makers of FinalBuilder) and requires D2010 or higher
-uses attributes to mark test related methods

DUnit2
-a fork of DUnit started by Peter McNab
-introduces SetupOnce/TearDownOnce and other enhancements
-has been extended by tiOPF project

SmokeTest
-newer framework for Joylon Smith, recently released that uses Interfaces rather than abstract classes

DUnitLite
-an extension to DUnit offering more check methods with additional datatypes, and a fluid interface

OpenCTF
- an extension to DUnit to provide testing for components

I assume most people using newer versions of Delphi are using DUnitX, and that most people also use TestInsight to ensure their tests are executed when code changes are made.  What framework do you use and why?

If I missed any, please let me know.