mirror_edk2/BaseTools
Jordan Justen d80e451b18 BaseTools/UniClassObject: Verify valid UCS-2 chars in UTF-16 .uni files
Supplementary Plane characters can exist in UTF-16 files,
but they are not valid UCS-2 characters.

For example, refer to this python interpreter code:
>>> import codecs
>>> codecs.encode(u'\U00010300', 'utf-16')
'\xff\xfe\x00\xd8\x00\xdf'

Therefore the UCS-4 0x00010300 character is encoded as two
16-bit numbers (0xd800 0xdf00) in a little endian UTF-16
file.

For more information, see:
http://en.wikipedia.org/wiki/UTF-16#U.2B10000_to_U.2B10FFFF

This means that our current BaseTools code could be allowing
unsupported UTF-16 characters be used. To fix this, we decode the file
using python's utf-16 decode support. Then we verify that each
character's code point is 0xffff or less.

v3:
 * Based on Mike Kinney's feedback, we now read the whole file and
   verify up-front that it contains valid UCS-2 characters. Thanks
   also to Laszlo Ersek for pointing out the Supplementary Plane
   characters.

v4:
 * Reject code points in 0xd800-0xdfff range since they are reserved
   for UTF-16 surrogate pairs. (lersek)

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yingke Liu <yingke.d.liu@intel.com>

git-svn-id: https://svn.code.sf.net/p/edk2/code/trunk/edk2@17694 6f19259b-4bc3-4df7-8a09-765794883524
2015-06-23 23:34:19 +00:00
..
Bin CodeModule: BaseTools - Adding the svn:external to the https://svn.code.sf.net/p/edk2-toolbinaries/code/trunk/Win32 2014-07-25 14:07:00 +00:00
BinWrappers/PosixLike Sync BaseTools Trunk (version r2518) to EDKII main trunk. 2012-04-10 07:28:51 +00:00
Conf BaseTools: Append FILE_GUID to BaseName. 2015-06-10 07:50:59 +00:00
gcc Sync BaseTools Branch (version r2321) to EDKII main trunk. 2011-09-18 12:17:25 +00:00
Scripts 1. Update UpdateBuildVersion.py; 2015-02-06 03:40:27 +00:00
Source BaseTools/UniClassObject: Verify valid UCS-2 chars in UTF-16 .uni files 2015-06-23 23:34:19 +00:00
Tests BaseTools/Tests: Add unit test for AutoGen.UniClassObject 2015-06-23 23:34:14 +00:00
UserManuals Sync BaseTool trunk (version r2610) into EDKII BaseTools. 2013-11-18 07:41:21 +00:00
.gitignore BaseTools: Ignore BaseTools/Bin/Win32 directory in .gitignore 2015-04-17 07:40:44 +00:00
BuildEnv License header updated to match correct format. 2014-08-28 13:53:34 +00:00
building-gcc.txt Sync BaseTool trunk (version r2649) into EDKII BaseTools. 2014-01-27 05:23:15 +00:00
BuildNotes.txt Sync BaseTools Branch (version r2271) to EDKII main trunk. 2011-08-26 07:46:26 +00:00
Contributions.txt */Contributions.txt: Update example email address 2015-02-03 17:29:14 +00:00
get_vsvars.bat Sync BaseTool trunk (version r2670) into EDKII BaseTools. 2014-07-01 07:10:10 +00:00
GNUmakefile Sync BaseTool trunk (version r2599) into EDKII BaseTools. 2013-08-23 02:18:16 +00:00
License.txt Sync BaseTool trunk (version r2599) into EDKII BaseTools. 2013-08-23 02:18:16 +00:00
Makefile Sync EDKII BaseTools to BaseTools project r1971 2010-05-18 05:04:32 +00:00
ReadMe.txt Sync BaseTool trunk (version r2599) into EDKII BaseTools. 2013-08-23 02:18:16 +00:00
toolsetup.bat Sync BaseTool trunk (version r2670) into EDKII BaseTools. 2014-07-01 07:10:10 +00:00

This directory contains the next generation of EDK II build tools and template files.
Templates are located in the Conf directory, while the tools executables for
Microsoft Windows 32-bit Operating Systems are located in the Bin\Win32 directory, other 
directory contatins tools source.

1. Build step to generate the binary tools.

=== Windows/Visual Studio Notes ===

To build the BaseTools, you should run the standard vsvars32.bat script.

In addition to this, you should set the following environment variables:

 * EDK_TOOLS_PATH - Path to the BaseTools sub directory under the edk2 tree
 * BASE_TOOLS_PATH - The directory where the BaseTools source is located.
   (It is the same directory where this README.txt is located.)
 * PYTHON_FREEZER_PATH - Path to where the python freezer tool is installed

After this, you can run the toolsetup.bat file, which is in the same
directory as this file.  It should setup the remainder of the environment,
and build the tools if necessary.

Please also refer to the 'BuildNotes.txt' file for more information on
building under Windows.

=== Unix-like operating systems ===

To build on Unix-like operating systems, you only need to type 'make' in
the base directory of the project.

=== Ubuntu Notes ===

On Ubuntu, the following command should install all the necessary build
packages to build all the C BaseTools:

  sudo apt-get install build-essential uuid-dev

=== Python sqlite3 module ===
On Windows, the cx_freeze will not copy the sqlite3.dll to the frozen 
binary directory (the same directory as build.exe and GenFds.exe). 
Please copy it manually from <PythonHome>\DLLs.

The Python distributed with most recent Linux will have sqlite3 module
built in. If not, please install sqlit3 package separately.

26-OCT-2011