Friday, November 17, 2017

Fixing 'VOC' testing on Windows

Over the Veteran's Day weekend, I was looking at the issue tracker for VOC and noticed several similar issues relating to testing on Windows. So I brought out a Windows (XP) machine, and set to work installing dependencies. Because it was XP, some of the components were not the most up-to-date.
  • Python 3.4.4 is the latest version build for XP.
  • Java 1.8 (u152) works, but warns loudly that it is not really supported on XP.
  • Apache Ant 1.10.1 is the latest, and seems to work fine.
  • Git 2.10.0 is this is the latest available for XP.
After getting these all configured, and the environment variables set up...
  1. The shlex module from ouroboros was failing to build. This was because of UTF-8 characters that were not being accounted for in voc\transpiler.py. I have submitted a PR for this, and it has been merged.
  2. I ran into an issue that appears to be a problem with the flush() in runAsJava. Once I dug down far enough to find the actual error, I found a StackOverflow question that pointed me in the right direction. Popen(['java'], ...) was picking up java.exe from C:\Windows\System32 instead of the one in the %PATH%, so I had to rename the former.
  3. Due to performing this work on an under-powered machine, I had to increase the timeout for calling 'ant java' from tests\utils.py from 30 seconds to a minute.
After this, I finally got to the issue that was being experienced by so many others. For test_str tests, Python was emitting a UnicodeEncodeError because it was trying to use the encoding 'cp437' and some of the characters in the test do not map. The Java version was emitting UTF-8 characters, and while the console doesn't understand that encoding and displays them incorrectly, it does not throw an exception. It is this difference which causes the tests to fail.

While Python 3.6 apparently gets UTF-8 working correctly, a work-around for this issue on earlier versions of Python is done by setting the environment variable PYTHONIOENCODING to 'UTF-8'. I have had a fix ready for several days but have only had a little time here and there to test it. I have submitted a PR and will just see where it goes.