2017-11-02

Do Not Buy Avi-On Light Switches

I bought a GE Avi-On light switch for my daughter's room. This replaces the regular wall switch with one that can be remotely controlled from her phone. The idea was to allow her to turn off the overhead light from the bed. It worked for about 10 minutes until it started downloading a firmware update to the switch, at which point the update failed. The switch no longer works. Lots of people are reporting the same issue, which means there are thousands of these switches throughout the country with this problem.

The Avi-On switches have these problems:

  1. The phone software requires an internet connection and a login to the Avi-On site. Why? This is a bluetooth app. There is no reason whatsoever that it needs a connection to a remote network It only needs a connection between the phone and the switch.
  2. It requires location services turned on and access to the file system on the phone. Why? This just smacks of them just wanting to harvest information from the phone, because there is no need for these permissions. The only thing that would require these accesses is for a firmware update to be downloaded from the network to be pushed to the switch. Which leads to...
  3. Firmware updates fail. You cannot push down a firmware update to remote switches if the firmware destroys the switch! And if your testing is abysmal and you somehow screw up and release one, you must pull the firmware update off your site the moment you realize it. The people have been reporting that the Avi-On current firmware update is causing problems for the last two months, but they are still pushing out the update.
  4. These switches supposedly make a bluetooth mesh with other switches in the area to allow spanning past the normal range of bluetooth. Bluetooth isn't the right technology for home control. The range is too short, which means you either need a lot of these, or you need repeaters in various places. That's an expensive solution to home control.

In summary, do not buy Avi-On GE Bluetooth light switches, or anything else made by Avi-On Labs. I will be returning this switch to where I bought it.

2017-09-02

Outlook VBA to Move Spam from Top Level Domains to Junk

There is a spammer who has been active for the last couple of months. The majority of my spam email has been coming from the top level domains .trade, .bid, .club, .stream, and .date. I get no legitimate mail from any of those top level domains.

I wrote VBA routine to go through my inbox and move all email from those domains to my junk folder. It appears below:


Public Sub JunkSpamDomains()
    On Error GoTo ErrorHandler
    Dim mailItem As Outlook.mailItem
    Dim folderInbox As Outlook.folder
    Dim folderJunk As Outlook.folder
    Dim i As Long
    Dim accessor As Outlook.PropertyAccessor
    Dim strHeaders As String
    Dim lngOffset1 As Long
    Dim lngOffset2 As Long
    Dim lngDomainOffset As Long
    Dim strFrom As String
    Dim strDomain As String

    Const PR_TRANSPORT_MESSAGE_HEADERS = "http://schemas.microsoft.com/mapi/proptag/0x007D001E"    Const strBanned As String = ".trade|.bid|.club|.stream|.date"

    Set folderInbox = Application.Session.GetDefaultFolder(olFolderInbox)
    Set folderJunk = Application.Session.GetDefaultFolder(olFolderJunk)
    For Each mailItem In folderInbox.Items
        If mailItem.Class = OlObjectClass.olMail Then
            Set accessor = mailItem.PropertyAccessor
            strHeaders = accessor.GetProperty(PR_TRANSPORT_MESSAGE_HEADERS)
            lngOffset1 = InStr(1, strHeaders, vbCrLf & "From: ") + 8
            lngOffset2 = InStr(lngOffset1, strHeaders, ">")
            strFrom = Mid(strHeaders, lngOffset1, lngOffset2 - lngOffset1)
            lngDomainOffset = InStrRev(strFrom, ".")
            strDomain = Mid(strFrom, lngDomainOffset)
            If InStr(1, strBanned, strDomain) > 0 Then
                mailItem.Move folderJunk
            End If
            Set accessor = Nothing
            Set mailItem = Nothing
        End If
    Next mailItem
    Set folderInbox = Nothing
    Set folderJunk = Nothing
Exit Sub
ErrorHandler:
    Select Case Err.Number
    Case Else
        MsgBox "Unexpected Error #" & Err.Number & " " & Err.Description
        Resume Next
    End Select
End Sub

Press Alt-F11 to open the VBA Editor. Select Insert Module from the menu to insert a new module. Paste this code into the window. You can bind a button on the Quick Access Toolbar (QAT) to this macro. Click the drop-down button on the right of the QAT, and select "More Commands...". Drop the "Choose Commands from:" list and select "Macros". Select "JunkSpamDomains" and click the "Add>>" button. Click the OK button.

After moving message, review the Junk folder to make sure that only spam got moved.

2017-06-03

VBA Runtime Error Codes

I was teaching a VBA (Visual Basic for Applications) class for Excel today, and the question came up, "is there a list of all of the runtime error code numbers and what they mean?" You might be able to find one, but the easiest thing to do is to generate the list. Here is a small piece of VBA code that shows all of the runtime error code numbers and their descriptions.

Public Sub DisplayErrors()
    Dim i As Long
    
    For i = 1 To 65535
        If Error(i) <> "Application-defined or object-defined error" Then
            Debug.Print i & " " & Error(i)
        End If
    Next i
End Sub

When you run the code, it will print the list to the Immediate Window in the VBA Editor. Press Ctrl+G to make the Window visible.

It is possible to get other runtime errors, but only from some component that is called by VBA, not from VBA itself. When I run the code, this is the list that I get:

3 Return without GoSub
5 Invalid procedure call or argument
6 Overflow
7 Out of memory
9 Subscript out of range
10 This array is fixed or temporarily locked
11 Division by zero
13 Type mismatch
14 Out of string space
16 Expression too complex
17 Can't perform requested operation
18 User interrupt occurred
20 Resume without error
28 Out of stack space
35 Sub or Function not defined
47 Too many DLL application clients
48 Error in loading DLL
49 Bad DLL calling convention
51 Internal error
52 Bad file name or number
53 File not found
54 Bad file mode
55 File already open
57 Device I/O error
58 File already exists
59 Bad record length
61 Disk full
62 Input past end of file
63 Bad record number
67 Too many files
68 Device unavailable
70 Permission denied
71 Disk not ready
74 Can't rename with different drive
75 Path/File access error
76 Path not found
91 Object variable or With block variable not set
92 For loop not initialized
93 Invalid pattern string
94 Invalid use of Null
96 Unable to sink events of object because the object is already firing events to the maximum number of event receivers that it supports
97 Can not call friend function on object which is not an instance of defining class
98 A property or method call cannot include a reference to a private object, either as an argument or as a return value
321 Invalid file format
322 Can't create necessary temporary file
325 Invalid format in resource file
380 Invalid property value
381 Invalid property array index
382 Set not supported at runtime
383 Set not supported (read-only property)
385 Need property array index
387 Set not permitted
393 Get not supported at runtime
394 Get not supported (write-only property)
422 Property not found
423 Property or method not found
424 Object required
429 ActiveX component can't create object
430 Class does not support Automation or does not support expected interface
432 File name or class name not found during Automation operation
438 Object doesn't support this property or method
440 Automation error
442 Connection to type library or object library for remote process has been lost. Press OK for dialog to remove reference.
443 Automation object does not have a default value
445 Object doesn't support this action
446 Object doesn't support named arguments
447 Object doesn't support current locale setting
448 Named argument not found
449 Argument not optional
450 Wrong number of arguments or invalid property assignment
451 Property let procedure not defined and property get procedure did not return an object
452 Invalid ordinal
453 Specified DLL function not found
454 Code resource not found
455 Code resource lock error
457 This key is already associated with an element of this collection
458 Variable uses an Automation type not supported in Visual Basic
459 Object or class does not support the set of events
460 Invalid clipboard format
461 Method or data member not found
462 The remote server machine does not exist or is unavailable
463 Class not registered on local machine
481 Invalid picture
482 Printer error
735 Can't save file to TEMP
744 Search text not found
746 Replacements too long

2017-06-01

C# Optimization of Switch Statement with Strings

C# does some interesting things when you have a switch statement comparing a lot of strings: Suppose you have this:

   switch (input)
    {
        case "AAAA":
            Console.WriteLine("AAAA branch");
            break;

        case "BBBB":
            Console.WriteLine("BBBB branch");
            break;

        default:
            Console.WriteLine("default branch");
            break;
    }

    Console.WriteLine("Complete");


When you look at the IL (intermediate language) that it compiles into, it is essentially the same as a bunch of  if and else if statements. Converted back into C# code, it is as if you wrote this:

    if (input == "AAAA")
    {
        Console.WriteLine("AAAA branch");
    }
    else if (input == "BBBB")
    {
        Console.WriteLine("BBBB branch");
    }
    else
    {
        Console.WriteLine("default branch");
    }

    Console.WriteLine("Complete");

 However, if you continue to add case statements, this becomes inefficient. There are a lot of string comparisons that are really expensive. At a certain point, as you add cases, the compiler uses an entirely different technique to handle the cases. It creates a hash table of the strings. The IL looks like this, if it were converted back into C# code (assume there are more case statements):

    string s = input;

    switch (ComputeStringHash(s))
    {
        case 0x25bfaac5:
            if (s == "BBBB")
            {
                Console.WriteLine("BBBB branch");
                goto Label_0186;
            }

            break;

        case 0xff323f9:
            if (s == "AAAA")
            {
                Console.WriteLine("AAAA branch");
                goto Label_0186;
            }

            break;
    }
    Console.WriteLine("default branch");
Label_0186:
    Console.WriteLine("Complete");

The ComputeStringHash method is a pretty simple hash function that looks like this:

    internal static uint ComputeStringHash(string s)
    {
        uint num = 0;

        if (s != null)
        {
            num = 0x811c9dc5;
            for (int i = 0; i < s.Length; i++)
            {
                num = unchecked((s[i] ^ num) * 0x1000193);
            }
        }

        return num;
    }

This is a version of the FNV-1a hashing algorithm.

The change to using hashing seems to occur at about eight string case statements. The advantage is that there will be, on average, just one string comparison, the other comparisons are all comparing uint values. There is some overhead in performing the computing of the hash, which is why it doesn't use it for small number of case statements.

This actually becomes important when you are trying to write unit tests for the code. If you are trying to cover all of the branches in the unit tests, you will need to write code that hashes to 0xff323f9 but is not "AAAA" to get the goto Label_0186 branches to get covered. Your chances of finding something that hashes to the same value as your legitimate "AAAA" string without being "AAAA" is unlikely unless you are specifically trying to get a hash collision. This means that your code coverage will show branches as not being covered, even though you test every case statement in the switch statement. This will show a failure in your code coverage branch statistics (usually around only 60% covered), even though your unit test are actually adequate.

I have been working with the AxoCover and OpenCover programmers to try to get the coverage statistics for branches to be meaningful, but there may be no way to handle this correctly.

Addendum: The logic of the switch statements when it optimizes is slightly more complicated that what is presented above. The C# compiler actually performs a binary search on the hash index rather than just linearly searching through them, before getting to the comparison of the string. Performing hash collisions will raise your coverage to more than 90%, but will not go through all of the code for the binary search.

2017-05-30

How to Convert Project from MSTest (V1) to MSTestV2

To convert a MSTest version 1 project to version 2, you need to perform several steps:

  1. In the project references, remove the reference to Microsoft.VisualStudio.QualityTools.UnitTestFramework.
  2. Edit the project file and remove the line that contains <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
  3. Add the NuGet package for MSTest.TestAdapter.
  4. Add the NuGet package for MSTest.TestFramework
That should be all that is necessary to make the transition.

2017-05-29

Don't Indicate Status with Just Color


There is a flaw in the AxoCover code coverage tool: The colors used for coverage, red and green, are hard coded. If someone were red/green color blind, this tool would be difficult to use. That creates difficulty for 8% of men and 0.5% of women. Showing status with hard coded colors with no other visual indicator is a mistake in user interface design, not just in AxoCover but in all programs. Either colors must be configurable, or an additional way of differentiating the colors (such as hatching one of them) should be used.
Status indicated with red and green bars
You'll notice that a traffic stop light indicates color with both red and green, but also with position of the colors; it does not have one light that changes color.
Traffic light uses both color and position to indicate status
Indicating status with just color, particularly red and green, is a very common UI mistake among programmers who are not color blind. While I am not color blind, I try to be aware of places where color is used wrong. This is not just an inconvenience to some people. In some cases, particularly in the United States which has the Americans with Disabilities Act (ADA), using just color to indicate status might be a violation of the law, and could have legal repercussions.

Code Coverage with AxoCover

When performing unit tests on code, how do you know if your unit tests are covering all of the code? The answer is a code coverage tool. I first used a tool like this when I was on the Microsoft Access 1.0 programming team, where we got weekly reports on how well the unit tests were covering the code. Microsoft has a code coverage tool in Visual Studio 2017, but only in the Enterprise edition. Unit tests and code coverage go together, and for Microsoft to have unit tests in the Community and Professional releases but not code coverage is kind of dumb.

Fortunately, there is a pretty competent, free, third party choice called AxoCover. You can download it from the Extensions and Updates menu item on the Tools menu in Visual Studio. It places an AxoCover menu item on the Tools menu that brings up a window to control its use. Build your solution once for AxoCover to figure out what is in there to cover. Then on the AxoCover window, click the Run button at the top. This is what it will look like:

AxoCover window in Visual Studio 2017

The critical information is on the Report tab on the left. After running the tests, AxoCover shows how much of the code got covered by percent. There are two numbers: what percent of lines and what percent of branches got covered. If there is an "if" statement, a test that hits the line counts will result in the line counting 100%, but only if both the true and false conditions are tested will the branches show 100%.
AxoCover report tab
The goal here is to get the branches to over 90%. Why not 100%? In production code, there are sometimes error handling code that can happen in only the rarest of conditions. For example: there may be code to handle if the fixed hard drive that you are writing to fails while the program is running. Sometimes trying to create a unit test for these conditions is difficult or impossible. So in general, greater than 90% is considered covered. Obviously, higher is better, but if all of your code is at 90%, you are doing pretty well. So in the example above, the coverage is 85.7%, which is good, but not good enough.

After running the tests, you can right click on a procedure in the report window and select "Show source". The source will show, with some information on the left border on what was covered and what wasn't.
Code window after running AxoCover
The green bar show lines that were covered by the unit tests. The red bar shows lines that were uncovered. The little circles show if branches were covered. In an "if" statement, you want both circles filled in. In the example above, there needs to be a unit test that covers the case "a" branch of the switch statement. After adding a unit test that covers that statement, go back to the Tests tab, build and run the tests. The report tab, then looks like this:

AxoCover report after adding new test case
Here, the coverage is perfect.

If there is one thing lacking on AxoCover, it's the documentation. The Settings tab has pretty much all of the documentation that exists.

AxoCover is really just a UI that integrate into Visual Studio. The engine that drives the code coverage is a different open source project called OpenCover, that has a command-line only interface. OpenCover is installed automatically when you install AxoCover.

2017-05-02

How to Handle Decimal Windows Error Numbers

Occasionally, I receive error numbers from some program that are in decimal rather than the normal way that Windows discusses them, which is in hexadecimal. For example, I ran code contracts on a solution and it reported an error of -1073741571. To look up this error, several steps need to happen.

  1. Run the Windows calculator program. You can just type calc in the search on the task bar.
  2. Click on the menu icon in the upper left and select "Programmer".
  3. Click on "DEC" if it is not already selected.
  4. Type in -1073741571.
  5. Click HEX. The number comes back as FFFF FFFF C000 00FD.
  6. Ignore the top two bytes and search in your favorite search engine for C00000FD. This turns out to be a stack overflow exception. The exception is being caused by some bug in the code contracts when the Emit contracts into XML doc file checkbox is checked.
The Windows Calculator Program
The programming mode in the Windows calculator is not very well known. The other way to do conversions is to buy a HP-16C programming calculator on Ebay. None of the other calculators HP made were nearly as good for programming. I dearly love that calculator, but it's more trouble than its worth to go dig it out when the calculator in Windows is always there and you can copy and paste from it.

HP-16C Calculator

2017-04-28

Change Visual Studio Task Bar Button to Run as Administrator

For some tasks, you need to run Visual Studio as an administrator. For example, if a build of a project needs to register for COM interop, Visual Studio will need to be run as an administrator. The easiest way to do that is to pin Visual Studio to the task bar, then change the button to run as an administrator.

To pin Visual Studio to the task bar, in the Start menu, right click on the shortcut for Visual Studio, click More, then select Pin to Task Bar. Then:
  1. Right click on the new Visual Studio icon on the task bar
  2. Right click on Visual Studio 2017 (or whatever version you are dealing with) from the menu that appears
  3. Left click on Properties from the menu that appears
  4. Click the Advanced button in the dialog
  5. Check the Run as Administrator check box
  6. Press OK on all the dialogs

Code Contracts in Visual Studio 2017

At the moment, the Code Contracts project has not caught up to Visual Studio 2017. As a work-around, you can copy a file from the install for Visual Studio 2015 into the proper place and it will allow the projects to build. It will not, however, enable the tab for setting the options in the Project Properties. For that you will still need to open the project in Visual Studio 2015.

The word Community will need to be replaced by the specific version of Visual Studio you have installed, Community, Professional, etc., in the script that follows.

This can be placed into a batch file (fixcontracts.bat) that can be checked into the project. Run the batch file as an administrator.

xcopy /y "C:\Program Files (x86)\MSBuild\14.0\Microsoft.Common.Targets\ImportAfter\CodeContractsAfter.targets" "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Microsoft.Common.targets\ImportAfter\"
setx /m TargetFrameworkSDKToolsDirectory "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.7 Tools\\"
pause

The line with TargetFrameworkSDKToolsDirectory sets a system environment variable to point to a directory where it can find sn.exe that re-signs the executable after adding the contracts. The defaults inside the targets file do not find sn.exe correctly (at least in some of my environments) without this variable. The two slashes at the end are not a mistake; the first escapes the second and only seems to be needed when preceding a quote mark. If you need to adjust this variable later, you can type System in the taskbar search box, then click the  Advanced system settings item, then click the Environment Variables button and edit it in the dialog that appears.
Contracts won't work with the .Net Framework 4.7, at least until they do a new release, so stick with 4.6 at the latest.

2017-04-21

WiX MSI ICE Errors

A MSI (Microsoft Installer) file is actually a relational database. You can use the ORCA (One Really Cool App) program to examine its content. While the WiX candle compiler and light linker can check the contents of individual records in the database, it is the job of ICE (Internal Consistency Evaluators) to check if the database is created correctly.

ICE is a set of rules (written with code) that verify things such as referential integrity within the MSI file. This is a list of all of the ICE errors, what they mean, and how to fix them.

2017-04-19

Colorizing C# Code in Blogger

I went back and revised all of the C# code in this blog using the info from here. I then colorized the keywords using this web page. This made all of the code examples a little nicer. Some day I might do the same thing for other languages and XML.

WiX Burn Command Line and Custom Bootstrapper Event Sequence

When a bootstrapper created with the WiX burn program executes, there are six possible Actions that it can handle: Install, Uninstall, Layout, Modify, Repair, and Help. (There are three more listed in the enum, but I don't know how they get triggered as they are not supported on the command line: Cache, UpdateReplace, and UpdateReplaceEmbedded.) The actions can be triggered by a switch on the bootstrapper application's command line:


setup [/install|/uninstall|/layout [path]|/modify|/repair|/help|/?] [/full|/none|/passive|/embedded] [other parameters]


The default is /install /full. The /help and /? options both cause the Help action. You can detect which Action was passed by checking this.Command.Action and comparing the values to the enum LaunchAction. You can detect which Display option was passed by checking this.Command.Display against the enum Display. If you are constructing your own interface, you can interpret the Action and Display switches as you see fit. If you don't want to support one, you should return an error. The /layout option may have an optional path, which can be retrieved from this.Command.LayoutDirectory.

Other information can be passed on the command line, which can be read from this.Command.GetCommandLineArgs(), which returns a string array. The string array will have the Action or Display options stripped.

The actual parsing of the command line is found in this C++ code in the ParseCommandLine() method.

When a WiX custom bootstrapper executes, there are a series of events that occur. Although you can add your own event handlers, there is no need. Instead, when you inherit from BootstrapperApplication, you can then override methods whose names all begin with On. If you override one of these method, you should call the base method at the end. For example:

protected override void OnStartup(Microsoft.Tools.WindowsInstallerXml.Bootstrapper.StartupEventArgs args)
{
    this.Engine.Log(LogLevel.Standard, "OnStartup");
    base.OnStartup(args);
}

Below is the sequence of events as they occur, and which Action causes them to execute. It is helpful to have these as comments in the Custom Bootstrapper file, so I've formatted them to be copied and pasted.

// I=Install U=Uninstall L=Layout M=Modify R=Repair
//// OnStartup (IULMR)
////  OnDetectBegin (IULMR)
////   // For each package
////   OnDetectPackageBegin (IULMR)
////   OnDetectPackageComplete (IULMR)
////  OnDetectComplete (IULMR)
////  OnPlanBegin (IULMR)
////   // For each package
////   OnPlanPackageBegin (IULMR)
////   OnPlanPackageComplete (IULMR)
////  OnPlanComplete (IULMR)
////  OnApplyPhaseCount (IULMR)
////  OnApplyBegin (IULMR)
////   OnCacheBegin (LR)
////    OnProgress (L)
////    OnCachePackageBegin (LR)
////     OnResolveSource (L)
////     OnCacheAquireBegin (LR)
////      OnCacheAcquireProgress (repeats) (LR)
////     OnCacheAquireComplete (LR)
////     OnCacheVerifyBegin (LR)
////      OnCacheAcquireProgress (repeats) (LR)
////     OnCacheVerifyComplete (LR)
////     OnProgress (LR)
////    OnCachePackageComplete (LR)
////   OnCacheComplete (LR)
////   OnElevate (IUM)
////   OnRegisterBegin (IUM)
////   OnRegisterComplete (IUM)
////   OnExecuteBegin (IULMR)
////    OnExecutePackageBegin (IUR)
////     OnExecuteProgress / OnExecuteMsiMessage (repeats) (IUR)
////     OnProgress (IUR)
////    OnExecutePackageComplete (IUR)
////   OnExecuteComplete (IULMR)
////   OnUnregisterBegin (IUMR)
////   OnUnregisterComplete (IUMR)
////  OnApplyComplete (IULMR)
//// OnShutdown (IULMR)

// Other events that may occur:
// OnDetectCompatiblePackage
// OnDetectForwardCompatibleBundle
// OnDetectMsiFeature
// OnDetectPriorBundle
// OnDetectRelatedBundle
// OnDetectRelatedMsiPackage
// OnDetectTargetMsiPackage
// OnDetectUpdate
// OnDetectUpdateBegin
// OnDetectUpdateComplete
// OnError
// OnExecuteFilesInUse
// OnExecutePatchTarget
// OnLaunchApprovedExeBegin
// OnLaunchApprovedExeComplete
// OnPlanCompatiblePackage
// OnPlanMsiFeature
// OnPlanRelatedBundle
// OnPlanTargetMsiPackage
// OnRestartRequired
// OnSystemShutdown

2017-04-18

Visual Studio 2017 Missing Guidgen.exe

I installed Visual Studio 2017 on a machine today. I installed the minimal set of things needed to perform the compile that I needed. When I went to create a GUID, selecting Tools > Create GUID from the Visual Studio menu, it complained that it was missing guidgen.exe. I went back to the Visual Studio installer (C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe) and selected Modify. From there I selected Individual components, then check the box for VC++ 2017 v141 toolset (x86,x64) in the Compilers, build tools, and runtimes section. When I clicked the Modify button, it then installed the guidgen.exe program, among other things.

This is actually a bug in the Visual Studio 2017 install. It should not add the Create GUID to the tools menu if it doesn't also install the guidgen.exe tool.

I reported this bug to Microsoft. They say that it is fixed in an upcoming release.

2017-04-12

Extract the Contents of a WiX Burn Setup Executable

To extract the contents of a setup executable created by the WiX Burn compiler, use this command line:

dark setup.exe -x extractfolder

This assumes that the dark.exe program is on the path, that the name of the executable that you want to extract is setup.exe and that extractfolder is the path to the folder in which you wish to extract the contents. It will create two sub-folders: AttachedContainer and Ux. The AttachedContainer folder will have the payloads that are installed into the executable. The Ux folder will contain all of the files necessary to build the user interface of the application.

2017-04-08

Be Careful About Formatting WiX XML Files

During the construction of a WiX MSI file, I was getting an error that looked like this

ICE17: Bitmap: '

It took a bit to debug what was causing it. I eventually used the WiX dark tool to decompile the MSI file, then looked for places where it might be referencing a bitmap. This is what I saw in the file (with some attributes removed):


<Control Id="BannerBitmap" Text="&#xD;&#xA;        WixUI_Bmp_Banner&#xD;&#xA;    "/> 
 
Makes perfect sense what was happening. This is what the XML in the source looked like:

    <String Id="BannerBitmap" Overridable="yes">
        WixUI_Bmp_Banner
    </String>

What had happened is that I had used Visual Studio to reformat the XML, which inserted Carriage Return/Line Feed (the &#xD;&#xA;) and spacing around the WixUI_Bmp_Banner. When WiX went to write the error into the error file, the first Carriage Return ended the error message, and it lost the rest of the message that had what it was complaining about. It's a bug in the WiX light compiler that it doesn't show the right error message, but perfectly understandable what is happening. All that is needed to fix the error is removing all the formatting.

<String Id="XocReadMeDlgBannerBitmap" Overridable="yes">WixUI_Bmp_Banner</String>


Looking at the output from the dark program shows several places where the formatting has messed up the strings being used in the files. I recommend using dark to find places in your files that might have subtle errors like this. It's important to know that WiX is actually putting all the white space in the MSI files, including Carriage Return/Line Feed and all spaces and tabs that appear in the XML files.

Transitioning from InstallShield LE to the WiX Toolset

I am making the transition from using InstallShield LE (Limited Edition) to using the WiX toolset for the installation of the Xoc Maya Calendar program that I developed. Both InstallShield and WiX use the Microsoft Windows Installer to do the heavy lifting. The Microsoft Windows Installer is a database with the extension .msi that has all of the information to install a program inside it. It cannot do a few things, though, such as install the .NET Framework before it runs. For that it needs a bootstrapper program that does the install of the framework, extracts the MSI file, then uses the Microsoft Installer exec (msiexec.exe) program to actually do the main install.

InstallShield provides an interface for creating both the MSI file and the bootstrapper. However, it has some limitations that have caused me to move to using WiX. These are the reasons I have moved:
  • The full version of InstallShield is expensive. It is $699 to $4999 depending on the version you get. There is a Limited Edition version, however, that was less, and that was what I was using.
  • The Limited Edition came with Visual Studio 2015 Professional. However, it did not work with the Community Edition. The Community Edition is the free version of Visual Studio for very small companies (like mine) and educational purposes. As a Microsoft Alumni, I can get Professional Edition at a steep discount, but it gives me no benefit over the Community Edition, except providing the license to use InstallShield LE.
  • InstallShield LE is just that, limited. There are many kinds of installs that it won't create. WiX can create anything that the MSI files support.
  • However, this is the killer: InstallShield LE is not out for Visual Studio 2017. It's not just that it isn't out now, and might be out in a month, it's that there might be significant periods in the future where it is unavailable when there is a lag between a Visual Studio release and an InstallShield release.

WiX is free and open source. The main drawback is that the learning curve to using it is steep. It is configured entirely by XML files; there is no user interface other than some minimal integration with Visual Studio. There are some attempts at providing a UI to creating the XML files, but none of them is as slick as InstallShield. It's not that XML configuration is difficult. I actually prefer XML to a UI, it's that there are a lot of options, and in some cases, it is not entirely clear how they interact. The UI can provide a list of Windows releases or .NET Framework releases rather than knowing exactly what value that needs to be entered into the XML.

After a few days of working with WiX, I broke down and essentially bought the documentation, which is the book WiX 3.6: A Developer's Guide to Windows Installer XML. (WiX is on 3.11 with 4.x in beta at this point, but the book is still the best there is.) This book is really essential for creating WiX installs. The documentation of each of the entries in the XML file is on the WiX web site, but the way to tie them together to make an install is found in the book. With a few days of working my way through the book, I am well on my way to having a new install process.

There is one tip that you may find useful if you make the transition from InstallShield to WiX. (I, of course, found this out after I had done most of the work by hand!) You can actually use some tools to create an equivalent WiX WXS file from the InstallShield executable. You will need the InstallShield setup file and the dark.exe tool that comes with the WiX toolset. Assuming that the setup program is setup.exe, that the MSI file that it extracts is setup.msi, and that the dark program is on the path, from an elevated command prompt execute the following commands:

setup.exe /x /b"." /v"/qn"
dark -x . "setup.msi" "setup.wxs"

The first line extracts the MSI file from the InstallShield executable into the current directory. The second line extracts the binary resources from the MSI file into sub-directories and creates a file called setup.wxs that has the WiX XML necessary for producing that MSI file. The WXS file will have a lot of cruft in it created by InstallShield that should be cleaned up, but it will save you a lot of work on trying to create an equivalent WiX file for the InstallShield setup.

2017-04-04

WiX Votive Preprocessor Variable Not Finding Project Name

In WiX, the votive preprocessor variables allow the use of syntax such as $(var.ProjectName.TargetDir), where the ProjectName is the name of a project within your solution. However, you may get an error that the variable is undefined. If so, the most likely reason is that you need to add a reference to the project mentioned in ProjectName to the WiX project that is using the variable.

I got on a wrong track thinking that maybe WiX was confused by the fact that I had periods inside the project name. That, however, was a red herring. It would have complained about any project name.

2017-04-03

The Design of Everyday Things

When I worked at Microsoft, they gave anyone who wanted one copies of several books. One was The Design of Everyday Things (originally called The Psychology of Everyday Things), by Don Norman. The Design of Everyday Things is still one of my favorite books. It is not really about computer interfaces, but all the rules apply to computers. It covers things like: why is it that when you come to a door in a building you are always pushing when you should be pulling, pulling when you should be pushing, or pushing on the side where the hinge is? It's because the door design told you to do that (and don't you feel stupid when you do!). If designers were allowed to design emergency exits, lots of people would die in fires. We don't let designers create emergency exits. Think about that the next time you click on something that looks like a hyperlink and you don't go anywhere.

2017-03-30

WiX Splash Screen Not Showing

I am working with WiX 3.11, the Windows Installer Toolkit. I was having a problem with the splash screen not showing when specified in the SplashScreenSourceFile attribute. The WiX log file recorded "Error 0x80070057: Failed to load splash screen bitmap." I eventually found this post that says that the version of the bitmap matters. Gimp and other programs output the wrong version for WiX. This is probably a bug in WiX. I opened the bitmap in Microsoft Paint and saved it again. After doing that, the splash screen started showing.

Note that the WiX splash screen must be a bitmap, not a png, jpg, or any other file format.

2017-03-18

C# Version 7 New Features

Visual Studio 2017 shipped earlier this month. Visual Studio itself has new features, but this article is concentrating on the C# programming language. I made a previous post on what they were talking about early in the development of C# version 7, but now we get to see what actually made it into the final product.

While these features are in the shipping version, there is yet to be a formal specification of the language for C# version 7. The last formal specification was for C# version 5. It would be good if they could get the spec updated.

This article describes the new features: https://blogs.msdn.microsoft.com/dotnet/2017/03/09/new-features-in-c-7-0/

1) Out variables. When specifying out variables, the definition of them can be placed at the point where they are called. We used to have to do this:

public void PrintCoordinates(Point p)
{
    int x, y; // have to "predeclare"
    p.GetCoordinates(out x, out y);
    WriteLine($"({x}, {y})");
}

but now we can do this:

public void PrintCoordinates(Point p)
{
    p.GetCoordinates(out int x, out int y);
    WriteLine($"({x}, {y})");
}

This is very nice especially for things like a TryParse method.

2) Pattern matching. This is now valid code:

switch(shape)
{
    case Circle c:
        WriteLine($"circle with radius {c.Radius}");
        break;
    case Rectangle s when (s.Length == s.Height):
        WriteLine($"{s.Length} x {s.Height} square");
        break;
    case Rectangle r:
        WriteLine($"{r.Length} x {r.Height} rectangle");
        break;
    default:
        WriteLine("<unknown shape="">");
        break;
    case null:
        throw new ArgumentNullException(nameof(shape));
}

The order of interpreting case statements has changed. Before it was undefined the order they were executed. Now, because order matters as the two Rectangle cases above show, case statements are executed in order, except the default is always executed last, regardless of order.

There is more stuff that matching can do.

3) Tuples. I think this is the one that might affect my code the most as it gives the ability to return more than one value from a method call. The Lua programming language has a way of doing that, and I found it cool. Tuples is actually more powerful than what Lua can do. Example of a declaration:

(string first, string middle, string last) LookupName(long id)
{
    // stuff
}

Then can be called with:

(first, middle, last) = LookupName(id2);

There are more variations on this.

4) Local functions. A method that is scoped so it is only accessible to an outer method. This is likely to get used incorrectly, so be careful. Here is a valid use:

public int Fibonacci(int x)
{
    if (x < 0) throw new ArgumentException("Less negativity please!", nameof(x));
    return Fib(x).current;

    (int current, int previous) Fib(int i)
    {
        if (i == 0) return (1, 0);
        var (p, pp) = Fib(i - 1);
        return (p + pp, p);
    }
}

5) Underscores in constant declarations of numbers. These are all valid now:

var d = 123_456;
var x = 0xAB_CD_EF;
var b = 0b1010_1011_1100_1101_1110_1111;

The underscores are ignored, but may improve readability of some numbers.

6) Ref returns and locals. Methods can return information by reference instead of by valid. See the example in the link above. This is particularly useful in returning pointers into a large data structure. For a few things this will vastly speed up code.

7) Async can now return things that are not Task. Creating things using these is not going to happen a lot, but calling things defined with them probably will.

8) You can now use expression bodies (=> definitions) in accessors, constructors, and finalizers. Some people love these...I just find them a weird syntactic sugar.

9) Throw from the middle of an expression. Okay fine. There might be a day where I find this useful...but it doesn't seem likely right now.

2016-11-12

Password Won't Work

Okay, this one was kind of trivial, but it took a little time to debug it, and I'm trying to document the things that I have to debug. I figure that if I have to debug it, someone else may be running into the same problem. That someone else might be me a few years from now.

A Windows Update of SQL Server on my web server was causing it to partially hang at boot. This was keeping other parts of the operating system from booting, and make the Remote Desktop part of the network driver to not get any CPU time. Remote Desktop to the web server stopped working. I went to the server rack and tried to log in. It didn't accept the password. Arg, now what do I do? I'm locked out of the server.

I booted from the operating system disk and created a command prompt. When I tried changing drives, something mysterious showed up. It was not echoing the key I was typing on the keyboard. Oh, duh! Due to space considerations at the server rack, what I was typing on was one of those small keyboards that has a keypad overlaid on some of the letter keys, instead of a separate keypad. NumLock was on! Thus I was typing numbers for part of the password. Reboot, turn off NumLock, and it logged in fine.

Next step is to disable NumLock permanently at startup. This link gives three ways of doing that. Then on to fix my hang problems.

2016-11-03

Windows Photos CPU Consumption

With the demise of Google Picasa, I need a new photo indexer and viewer. The Google replacement solution does not work for me. I have a fairly large amount of photos taken over the years, that I do not want to upload to the cloud. I cannot have Google performing lossy compression on them when they do get there, and I'm not paying them to not do compression. I need the original photos. Picasa was about perfect for what I needed, but now it's gone. I would even pay a small fee to keep Picasa around. This is not the first Google app that they've abandoned that I used.

So I looked around for what could replace it. One app that might work is Windows Photos, that comes for free with Windows. I fired it up. However, by default, it only searches in a few pre-defined places on the disk. So I changed the setting to have it search the entire disk. Which it did, very slowly, but it found all my photos. Still evaluating whether it meets my needs.

Several days later, I noticed that my laptop was running hot, with the fan running at full speed, every time that it went idle. I didn't make the connection that it was what I did in Photos. And, of course, if I went to look at what was running, that took it out of the idle state, which meant that whatever was running hot wasn't doing it when I looked. Stupid Schrödinger's cat getting in the way again...looking changes the state of things.

I eventually figured out how to find what was consuming all the CPU time. It was right there in Task Manager. Ctrl-Alt-Del brings up Task Manager. In a Remote Desktop environment, its Ctrl-Alt-End, as Ctrl-Alt-Del will bring up the Task Manager of the host system, not the remote system. Then select Task Manager from the list. In Task Manager, there is the App History tab. Click on that. Then click Delete usage history. Then let it go idle, come back later, and look at the list.

So what was sitting at the top of the list? The Windows Photos app! Running all really hot. It seems that when I told it to look over the entire disk, it does that any time the system goes idle, looking for new photos! Since it can't complete that task before the next time it should look, it just runs continuously. To make it so it only looks for new photos when it is running in the foreground, in Windows, click Start > Settings > Privacy > Background Apps, and move the slider on Photos to off. Problem solved.

So lessons learned: To monitor CPU usage, the App History tab in Task Manager is the right tool. Don't let Photos run in the background. And don't trust Google to keep making apps available...they abandon apps that are successful, unlike Microsoft...and when they do, they remove them completely, not just leaving the last version on their sites.

2016-05-04

Getting a Free SSL Certificate from Let's Encrypt to Use on IIS Server

Let's Encrypt (https://letsencrypt.org) provides a service that distributes free SSL (Secure Socket Layer) certificates for web sites with automatic creation, validation, signing, installation and renewal. SSL allows HTTPS secure encrypted connections to the web server. I first mentioned Let's Encrypt in this post in 2014. It is finally out of beta and available for prime time. The main purpose is to make it trivial to have encrypted communication between the web browser and web server, and make much more of the Internet secure.

Let's Encrypt only creates domain validated certificates. This means that you must have control over the web server, not that you control the business entity that purports to run the web site. Thus, someone can create a web site that looks like it is run by Microsoft and have SSL traffic to the site, but it is not controlled by Microsoft. Let's Encrypt currently has no plans to support Extended Validation Certificates that verify that a web site is run by the entity that it says that runs it.

Let's Encrypt has a program that you run on the web server that sets up the web site with the certificate. It validates that you have control of the web site, then gets a certificate and installs it. It also sets up the job to renew the certificate. Let's Encrypt believes in having certificates with short expiration dates, with frequent renewals. With the automatic process to perform the renewal, frequent renewals are painless.

The program that sets up the Let's Encrypt certificate on the web server will be maintained on the EFF (Electronic Frontier Foundation) web site, although for the moment it is still on the Let's Encrypt web site. However, that program is designed for Apache web servers running on Linux. Someone has released a version of Let's Encrypt for IIS servers running on Windows. I have tested this on a site on my web server, and it could not be more simple to use. You run the program, answer a couple of questions, and it does all the work of getting the certificate, installing it, changing the web site bindings to use the certificate, and setting up the scheduled job to renew the certificate.

There are still some problems. As I mentioned in this post, if there are multiple web sites running on the same IP address, a chicken-and-egg problem occurs with establishing the communication between the browser and the web server. The browser sends an encrypted request to the web server on port 443. The problem is that the web server doesn't know which web site it should direct the request to, because the request that has the web site to communicate to is encrypted, and it can't decrypt the packet until it can retrieve the certificate from the web site. This means that using the traditional technology you can only have one SSL encrypted web site per IP address. That would be fine if we had an abundance of IP addresses available, but we don't with IPv4. IPv6 will fix that problem, but the Internet infrastructure for it isn't there yet.

There is a solution to the chicken and egg problem. It's called Server Name Indication (SNI). The idea in SNI is that at the start of the handshaking procedure between the browser and the web server, a preliminary exchange occurs that directs the server to the correct web site to get the certificate. This requires that SNI be implemented in both the web browser and the web server. Anyone running an old web browser can't visit SNI enabled web sites. The web server also must support it.

This link gives which versions of browsers and web servers started supporting SNI. The two most significant entries are that Internet Explorer on Windows XP doesn't support it, and it first became available on Windows Server 2012. On my web sites, at this time, approximately 6% of my traffic does not use a browser that supports SNI. Do you want to exclude those people? More significant for me is that my server is running Windows 2008 R2, which doesn't support SNI at all. So unless I upgrade my server, I have a limit of one SSL site per IP address.

Let's Encrypt is a significant part of the solution to securing all communication between the browser and the server. It doesn't solve all the issues, but it is free and easy. Other than the technical limitations listed above, there is not much reason not to use it.

2016-03-25

Names That Break Databases

Database programmers frequently make false assumptions and compromises about names. Back when disk space was horrendously expensive, it was often for disk consumption reasons: If you allocated 15 characters for a last name, it consumed 15*numberOfRecords amount of bytes of disk space (not even considering non-ASCII names). The average last name in the U.S. is six characters, and almost all are less than 13, so 15 should be enough, right? These early systems also often didn't allow for variable length names. The person who ran the department for one of my first programming jobs had a hyphenated last name with 16 characters in it, and hated having it truncated. We bought more disk space just to handle that name.

You may think this all went away with the fact that disk space has become extremely cheap. (This dates me a little: My first 1 GB drive cost over $1000. Think about how expensive your cell phone, much less a terabyte+ database, would be at that rate!) Cheap disk space and variable length database fields have made things better, but database programmers still make many wrong assumptions and compromises regarding names.

Think about this: What if one of your customer's last name was Null? Would your system break? If so, it probably has other problems and is possibly open to a SQL Injection Attack. One of my favorite XKCD comics (If you don't understand why this is funny, see SQL Injection Attack on Wikipedia):


(In some places, because of this comic, SQL Injection Attacks are called Little Bobby Tables Attacks. I like this comic so much, I got one signed by Randall Monroe for my wife, who is an Oracle Database Administrator, for her desk at work.)

[General rule: Do not concatenate SQL with user supplied data...use parameters instead!]

This article discusses the many problems that people named Null run into: The names that break computer systems.

There are many other assumptions about names that turn out to be false. This is my favorite article about them: Falsehoods programmers believe about names

This is the start of the list from the article:
  1. People have exactly one canonical full name.
  2. People have exactly one full name which they go by.
  3. People have, at this point in time, exactly one canonical full name.
  4. People have, at this point in time, one full name which they go by.
  5. People have exactly N names, for any value of N.
  6. People’s names fit within a certain defined amount of space.
  7. People’s names do not change.
  8. People’s names change, but only at a certain enumerated set of events.
  9. ...
Take care that your system handles as many of these as possible. Almost all database systems have to make compromises to handle names, but they don't need to break just because they hit a name like 'Null'.

2016-02-02

HTML and CSS to Make Image Fill Window Height

I wanted to get an image to fill the entire window height in the browser and center horizontally, keeping the aspect ratio. There are a number of different ways of trying to do this, however, for my purposes (use in an epub, which restricts the HTML and CSS)  I could not use position:fixed or JavaScript. This vastly cut down on the number of methods possible. This is what I wound up with:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
    <head>
        <title>Cover</title>
    </head>
    <body>
        <div style="text-align:center">
            <img style="height:98vh" src="../Images/cover.jpg" />
        </div>
    </body>
</html>

The 98vh is a CSS3 unit that scales the height to 98% of the viewport height. This leaves a little border around the image. The vh will work in most modern browsers, but not in older ones.

2015-12-16

Using File.WriteAllText() with Encoding.UTF8 Writes Byte Order Mark (BOM) EF BB BF


First a little background on ASCII, Unicode, and UTF-8. ASCII (American Standard Code for Information Interchange) is a 50 year old standard, first adopted for teleprinters. It has 127 codes, and works rather well for representing English. As computers were used in other parts of the world, though, they needed some way to represent characters outside the ones available in ASCII. Various schemes were developed, but the one that has become the standard is Unicode.

Unicode represents each character as a numbered code point, allowing most characters in most languages to be represented. The first 127 code points are exactly same values as ASCII, making it a superset of ASCII. Unicode does not have a defined way of representing its code points in bytes, though, and various methods are used. The most popular encoding scheme is called UTF-8.

UTF-8 has the advantage that if the text characters are in the ASCII range, that the length in bytes is the same as ASCII. The length is only larger for representing characters outside the ASCII range.

So, given all that, you might think that the following four lines of C# code should all output the same bytes:

File.WriteAllText(@"c:\temp\Sample.txt", "Hello World!");
File.WriteAllText(@"c:\temp\Sample.txt", "Hello World!", Encoding.Default);
File.WriteAllText(@"c:\temp\Sample.txt", "Hello World!", Encoding.ASCII);
File.WriteAllText(@"c:\temp\Sample.txt", "Hello World!", Encoding.UTF8);

Since the "Hello World!" text is all in the ASCII range, you would expect that all four lines would write the same bytes. The first three lines, do write the same thing, but the fourth line writes something different. Here is a hex dump of the first output of the first three lines:

00000000  48 65 6C 6C 6F 20 57 6F 72 6C 64 21              Hello World!


Here is the hex dump of the Encoding.UTF8 file:

00000000  EF BB BF 48 65 6C 6C 6F 20 57 6F 72 6C 64 21     ...Hello World!


What are those first three bytes, EF BB BF? They are called the Byte Order Mark (BOM). They are supposed to indicate to a system reading the bytes how they are supposed to be read. When encoding the number 1 in binary, it could be encoded 1000000 or 00000001. The first is called Big Endian, and the second is called Little Endian. Most computers today use Little Endian ordering of bits.

Furthermore, when encoding the decimal number 400 in Little Endian, it could be encoded 00000001 10010000 or 10010000 00000001. In other words, the order of the bytes could change. The Byte Order Mark is meant to put a known three bytes at the beginning of the text so the system can figure out what the order of bits and bytes is being represented.

When a system reading Unicode text sees the Byte Order Mark, it is supposed to eat those bytes. However, if the system isn't expecting the BOM, then it displays what looks like three random letters at the beginning of the text, like .

So if you want to write UTF-8 with the BOM, then you should use:

File.WriteAllText(@"c:\temp\Sample.txt", "Hello World!", Encoding.UTF8);

On the other hand, if you don't want the BOM, then you should use:

File.WriteAllText(@"c:\temp\Sample.txt", "Hello World!");

They are not the same!

Incidentally, the output of the first four lines are way different from each other if the text included non-ASCII characters, but that is a whole other topic.

2015-12-04

Could Not Load the Assembly Because the Assembly Wasn't Signed

I was struggling for a few hours trying to get the Microsoft.Framework.Configuration code to run. It wouldn't because it kept complaining that:

FileLoadException was unhandled

As unhandled exception exception of type
'System.IO.FileLoadException' occurred in mscorlib.dll
Additional information: Could not load file or assembly
'Microsoft.Framework.Configuration.Abstractions,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or
one of its dependencies. A strongly-named assembly is
required. (Exception from HRESULT: 0x80131044)

The wasted hours came because I didn't scroll down in the dialog to see the last two lines of the error (for which I feel a little annoyed at myself). It was complaining that it couldn't load the assembly, and I kept trying to debug why the assembly wasn't being found. Except it was being found, just not loaded because it wasn't signed. The group working on this library made a mistake and failed to sign the assembly. As soon as saw the last two lines of the error, I immediately knew what was going on.

I've written about this exact problem in my book. Quoting from  my book, The Reddick C# Style Guide:

An assembly that has strong name can only reference other assemblies with strong names.

And a little further in the book:

⊗    There may be cases where a reference is needed to a required library has not been given a strong name, and there is no way of adding one. There is no other solution than not signing the assembly. If possible, though, work on acquiring a version of the library that has a strong name.

The simple solution to get the code working was to go into the Project Properties of my project and uncheck the "Sign the Assembly" check box on the signing tab. A more complex solution, since the library is open source, would be to get the sources and build it myself, signing the assembly. The best solution is to get the maintainers to sign the assembly so everyone doesn't run into this problem. Unchecking the "Sign the Assembly" check box has the drawback that it causes the Code Analysis CA2210 warning. It also leaves the assembly unsigned, which is a worse problem.

2015-11-17

Author Page on Amazon.com

I now have an author page on Amazon. http://amazon.com/author/gregreddick. You can find information about my books there. They don't allow me to list books where I contributed just an appendix, so there are quite a few that aren't listed.