Archive for February, 2021

hcUpdateFramework - The Server Side

Sunday, February 14th, 2021

As promised, in this article I will be describing the server side components of the hcUpdateFramework, namely:

  1. ClientUpdateService
  2. Launcher
  3. Register
  4. AutoUpdateServer
  5. Update Manager

4. AutoUpdateServer

The AutoUpdateServer is a Windows application that exposes the IUpdateService SOAP service on port 8080 by default.  It can be compiled as an ISAPI DLL, a Win32/64 VCL Application or a Win32 NT Service running under the local system account.  Most of my implementations have simply used the VCL server application so YMMV.

The server process sends an update to the ClientUpdateService when it checks using the IUpdateServer.GetUpdate method. The service creates a default INI file the first time it is started, if one does not exist. The INI file contains the following settings:


This service can be compiled as a Win64 NT Service, but currently throws an I/O related exception in the utility methods used for binary file streaming.

Updates are created in subfolders under the Updates folder located in the same directory as the update server.  All updates are static, meaning the contents of the update are determined when the update is created and not on a per request basis from the client (although that is technically possible). In order to reduce the bandwidth and server utilization, binary patch files are supported using Microsoft’s delta compression technology.

Currently, there are two databases supported; SQL Server (via ADO) and Firebird 3.X (via FireDAC).   The SQL Server schema needs to be updated to mirror the changes made when migrating the server to Firebird.  Once that is complete, the FireDAC datamodule could be modified to support both databases.  Scripts for all supported databases can be found under the SQL folder, including scripts to create new deployments in the database without using the Update Manager.

5. Update Manager

The UpdateManager is an application I wrote to automate the creation of updates for a given application.  The framework defines an application as a primary EXE which is entered into the list of registered applications in the Deployment database and whose version drives the update version (they usually match).  An application deployment may contain other EXEs, and supporting files which are detailed in the XML Manifest file which contains:

  • UpdateVersion - version of the Update (often matches primary EXE)
  • WhatsNew - RTF description of what has changed in this update
  • IsMandatory - update must be applied (user is not given a choice)
  • IsSilent - update is applied without presenting a UI (ClientUpdateService)
  • IsImmediate - update is applied immediately after download (ClientUpdateService)

1..N of the following where N is the number of deployment items aka application components:

  • FileName - target filename
  • Version - version of the file
  • TargetPath - location to copy the file into in case it differs from the application folder
  • IsAPatch - this file is a patch file to be applied using MS Patching
  • IsAZip - this file is compressed using the Zip archive format
  • Launch - launch the EXE when applying the update

An example manifest is:

<Manifest LocationGUID=”2bc5cc04-2307-41ab-9f3c-b3f8c0578be9 ApplicationGUID=”8BF6627D-CB05-4601-996E-072DC6785F81 UpdateVersion=” WhatsNew=”Incorporated Auto Update IsMandatory=”false IsSilent=”false IsImmediate=”false>
<Item FileName=”MyApp.exe Version=” TargetPath=”{app} IsAPatch=”false IsAZip=”false Launch=”false/>
<Item FileName=”App Manual.pdf Version=” TargetPath=”{app} IsAPatch=”false IsAZip=”false Launch=”false/>
<Item FileName=”Manifest.xml Version=” TargetPath=”{app} IsAPatch=”false IsAZip=”false Launch=”false/>

An Update on the server side consists of a root folder called Updates under which subfolders are created for each update version using the same naming convention used on the client as shown:

Updates Folder Structure Server Side

Updates Folder Structure Server Side

Within each update folder, there must be a manifest and all files appearing within that manifest.  Once the update folder is created with all the necessary content, a new deployment record must be created and new InstallationDeployment records need to be created in the database and then enabled to distribute the new deployment/update to existing installations/users.  The SQL to do so is present for Firebird so updates can be manually crafted, but it is much easier to use an UpdateManager.  Unfortunately, the current implementation requires the DevExpress ExpressQuantumGrid and Editors.  With the database changes made during migration to Firebird I will also need to update the hcOPF domain objects to reflect the current structures.

The hcUpdateFramework needs some TLC and can certainly benefit from some new ideas, but it is a functional, proven update framework that can be extended in many ways to facilitate your deployment needs.  I am open sourcing it, in the hope that others will contribute and make it a more complete, modern solution.

hcUpdateFramework - The Client Side

Wednesday, February 10th, 2021

The hcUpdateFramework consists of 5 parts:

  1. ClientUpdateService
  2. Launcher
  3. Win/Register
  4. AutoUpdateServer
  5. Update Manager
The first 3 of which reside on the client machine.  In this article I will discuss this half of the framework.

1. ClientUpdateService

The ClientUpdateService is a Windows NT Service that runs under the local system account and is responsible for downloading updates from the AutoUpdateServer to the client machine. All update processing occurs in subfolders under the application installation folder which is typically C:\Program Files\{CompanyName}\{ProgramName}. The update folder structure consists of a subfolder called oddly enough; Updates which has 3 subfolders; Pending, Applied and Backup.  Please note that the initial implementation assumed Administrator rights to the installation folder.  For that reason, it may be necessary to install the application into a user folder (ie: C:\Users\<UserName>\AppData\Local\) if the user is running with Standard User permissions.

The ClientUpdateService polls the AutoUpdateServer at a predetermined interval (defaults to 15 minutes) which is specified in AutoUpdate.ini. A default INI file is created the first time the service is started, if one does not exist.

The INI file specifies the URI of the AutoUpdateServer, in addition to the polling interval.  The same INI file is used for all client side applications, and has the following settings:




The ClientUpdateService sends the current application manifest to the AutoUpdateServer as part of the IUpdateService.GetUpdate request. If an update is available, the AutoUpdateServer provides the update components with a new manifest as part of the response. The ClientUpdateService saves the new manifest, and all files in the response into a new Updates\Pending subdirectory named for the UpdateVersion. The service then tells the AutoUpdateServer that it has successfully received the update by calling the IUpdateService.UpdateReceived method.

This method passes the ApplicationGUID, LocationGUID, and UpdateVersion to the server. Afterwards the service sleeps for the specified interval before checking for another update. The ClientUpdateService will never download the same update more than once, because the AutoUpdateServer knows it has already received the update.

The ClientUpdateService has the ability to apply the updates, but normally user interaction is desired.

2. Launcher

The Launcher is a Win32 client application that is responsible for actually applying any downloaded updates, and providing the user with appropriate feedback. The Launcher can be renamed so it will replace the current application if it has already been deployed, and act as an intermediate EXE ensuring all existing shortcuts still function. It creates a default INI file the first time the program is started, if one does not exist. The INI file specifies the program it is to launch if no update is present, or after the update(s) have been applied, the application install directory, where to look for updates, and the URI of the AutoUpdateServer.

Although the functionality has not been tested recently, the Launcher is capable of applying multiple updates at once. It should be verified that it applies multiple updates in the correct order before relying on this functionality. On startup, the Launcher reads all subdirectories that appear under Updates\Pending. It then applies the update by loading the manifest, backing up each file into a subdirectory it creates under Updates\Backup\X.X.X.X (where the Xs represent the UpdateVersion), and copying each file in the manifest into the application install directory. If the manifest indicates that the file is a patch file, the patch file will first be used to create a file with the extension .NEW. The patched file will then be used to replace the version in the application install directory.

Errors are trapped along the way, and if any error occurs, the user is informed that the update failed and application may be unusable. No attempt is made to rollback the update. It is left for manual intervention in part to ensure tech support discovers why the update process failed, and because any automated rollback attempt is just as likely to fail. A complete copy of the previous version of all updated files can be found in Updates\Backup\X.X.X.X. Restoring users to a running version of application should be as simple as copying these files back into the installation folder, provided no database updates have occurred that would prevent the backup version from running.

If an update is successfully applied, the Launcher informs the AutoUpdateServer by way of the IUpdateService.UpdateApplied method. This method passes the ApplicationGUID, LocationGUID, UpdateVersion, UpdateResult, and UpdateLog to the server which records the information for viewing in the Deployment Manager.

3. Win/Register

There is actually a Console and VCL Windows utility which are a relatively recent addition the framework.  This utility is meant to be bundled and called as part of an initial install. It calls the IUpdateServer.RegisterInstall method with the ApplicationGUID.  Each application to be deployed is identified by a unique GUID.  This method returns an InstallationGUID which is saved in the application manifest for identifying the installation to the UpdateServer for future updates.

Since this article is getting a little long, in my next post I will cover the server side of the framework.

hcUpdateFramework is Released

Thursday, February 4th, 2021

A long time ago, in a place not so far away, I wrote a framework which allowed a corporation to update an application it had deployed to 100+ remote locations with slow internet connections. Back then there were a few alternatives to rolling your own, but nothing seemed to have the feature set and development commitment from a Delphi component vendor. As a result, it was decided that I should implement one.

The entire AutoUpdate Framework was initially written in Delphi XE2. Although BITS was going to be used to transfer the files since it offers the ability to throttle bandwidth, in the end, for simplicity sake, web services were employed instead. Rather than a simple call to download an MSI, the framework provides a large degree of flexibility:

  • Do you simply want to generate and apply a binary patch?
  • Do you need to be able to apply a series of updates to bring the installation up to the current version?
  • Do you want to compress/zip the files to be delivered?
  • What about deploying additional files and removing obsolete ones?
  • Do you want to display a list of changes so the user can choose whether they update?
  • Do you want to notify the user that updates are available while they are using the application?
  • Do you want to make the update mandatory so bug fixes that are causing data issues are squashed ASAP?
  • Do you want to schedule the application of an update silently in the background at a given time in the middle of the night?

All these scenarios are supported and with a custom deployment application, you can see what version each user is running, any errors in the update process and when they last launched the application. It would be easy to extend the Server to accept bug reports and create an integration to your bug tracking system and/or CRM.  Usage Telemetry could also be added, and if user’s don’t pay, an update can render your software inoperable.

Recently I resurrected the code to provide updates to a client during COVID, so I added Firebird support and changed the data model to track registrations (new application users). It would be very easy to extend the system with your own custom licensing code and track where your new users are located and how many “conversions” from trialware or lite to full versions you’re getting.

That’s the power of open source;  no waiting for a vendor to implement the features you want for your use cases.

In subsequent posts, I will describe the moving parts of the system.

Improving Your Form - a Decimal cannot fit into a Decimal

Monday, February 1st, 2021

If you have ever gotten an error message like “DATABASE ERROR: [FireDAC][Phys][ODBC][Microsoft][SQL Server Native Client 11.0][SQL Server] Error converting data type decimal to decimal” you might be thinking it’s a bug in FireDAC or whatever database layer you’re using.  The two types are the same so no “conversion” should be necessary and how can assigning the same type to itself fail?

In fact, in the cases I have encountered it was because the value being written to the database exceeds the range (positive or negative) of the column as defined in the database.  For example, attempting to write 100,000 into a column defined as decimal(9,4) would fail.

Ideally the database layer would provide a more precise error message, something that perhaps included the value being written with the name, type and size of the database column.  I’m sure there are some technical reasons for such obscure error messages.  No doubt multi-vendor database support is likely the major one.  If the database doesn’t provide the information no DAL can pass it on.  Crafting meaningful error messages also requires a level of skill, forethought and empathy.

This is exactly why applications really need some sort of framework for UI/Domain level validation for at least minimum and maximum values so monkey testing cannot break the app.  If every form in your application needs to implement validation logic, keeping it consistent over time and not forgetting some control(s) is problematic. So the first step to improving your form is to put all the validation and business logic elsewhere so it’s re-usable. That doesn’t mean you have to go full MVVM with domain objects. A good start is to use an ORM.