Adding new target platform and new tool chain
|This page describes step by step what do you do to add a new target
platform and a new tool chain to those already supported by SBuild.
Check this page to remember the porting effort overview.
|Table of content
Step 1: Support in the SBuild C/C++ toolkit
Let's first locate the C/C++ toolkit. It is stored in a subdirectory of the SBuild installation directory that is called tk_ccpp (it is a deep subdirectory, use file search by name).
The SBuild C/C++ toolkit stores the information about what to build for and what to build with in three global variants:
You don't have to learn first what a SBuild variant is (by the way, the 3 above are the most complex variants, way more complex than the average variant). You just have to make addition to them and that may be just 1-2 lines of code. The relationship between these 3 variants is the following:
First thing first, you have to choose names for your new target platform and your new toolchain. The name of the toolchain is usually the name of the provider/seller and the name of the platform is usually in the form <processor_operatingsystem>. These names will become new allowed values for the respective variants.
Step 1 requires you to make additions to the file ccpp_vars.py. This is the file that stores the Python descriptions of the 3 variants (location on your machine is probably <InstallDir>\scons-local-0.96.1\SCons\SBuild\tk_ccpp\ccpp_vars.py). In the future, we may decide to split this file in several different files (for several practical reasons).
Running any build script with argument explain=sum will tell you, among other things, which are the already supported platforms and tool chains (by listing the allowed values of all variants, including tgtplatform and toolchain).
You will notice that a variant has a number of attributes. Most of them are Python dictionaries keyed on the allowed values that a variant may take. For example, one attribute on the variant tgtplatform is descriptions and it is a dictionary containing a short, human readable text description for each platform name.
There are several attributes in tgtplatform variant that you need to look up and extend. Do not extend just yet, only take note. Important attributes in tgtplatform:
You should be informed at this point that SCons is NOT using the shell environment where you start the build script when building something. It is using instead it's own environment named the SCons Construction Environment. This attribute, scons_constr_var_to_set, is used to enrich the construction environment of SCons with variables and variables values that you want to be specific to your new target platform. More on this in Step 2.
The good news is that you don't have to touch directly all attributes of a variant. Adding a new value to a variant can be achieved in two alternative ways:
It is always a good idea to take a look first and check if you can find a similar platform already supported. Moreover, if that platform has been already cloned then, looking at that older clone, you already have a quick list of what you likely need to provide for your new platform.
|New toolchain value
Important attributes in toolchain:
Note that the SCons tools have access to the SCons Construction Environment. Therefore, you may and you will put in this attribute, scons_constr_var_to_set, things that you want to "pass over" to your SCons Tools.
|New bldopt entry
After you changed toolchain and tgtplatform you should also change the variant bldopt. This variant contains command line options to make a debug build or a release build with any toolchain. It is not about introducing a new bldopt value, but about changing what happens for the already existing allowed values dbg and rls. The important attributes of bldopt are:
As the name says, these are dictionaries keyed with the current value of the toolchain variant. You can use the local function clone_bldopt() to "copy" the options of one toolchain to another toolchain in 1 line of code.
Note that you should NOT stuff up, for example, all options to the compiler in ccflags_per_toolchain just because you can put them there. Here only those options that are different between the debug and the release build have their place. Those that are common may be placed in the attribute ccflags of the variant toolchain (or of tgtplatform, if specific to a target platform).
Note also that the variant bldopt has more allowed values, not only dbg and rls:
It is possible to introduce new bldopt allowed values for your toolchain (just provide them, the other toolchains will ignore them). It may be useful when you want to separate different kind of optimization (ex. for speed versus for space). But we don't recommend that path: multiplying the flavors also means multiplying the testing burden. Better decide here for only 1 kind of optimization and map that to rls, you may latter still customize it for individual products (in the SBuild project of that product).
Step 2: Support in SCons
SCons knows how to run an individual external executable from a tool chain (SBuild doesn't know, SCons knows). That means SCons knows how to make up the command line to run that. SCons stores this information in a Python file per external executable. That file is called a SCons Tool (a.k.a. SCons wrapper)
How can you check what SCons Tools are already available? SCons stores all the Python Tool files in one directory: <InstallDir>\scons-local-0.96.1\SCons\Tool. You will find there all kind of tools, some concerning C/C++ compilations (Microsoft, Borland, GNU, Sun, IBM, SGI, HP, etc) as well as tools completely unrelated to C/C++ (Java, PDF compilation, Zip making, etc.). There are more than 60 tools, most of them not used yet by SBuild.
But if you are here it is likely that you will have to write new SCons tools. Don't panic, "new" tools may be not that new. New tools can reuse existing tools; they can amount in the end to just a few lines of code carrying the genuine new information. Take a look, for example, to how the SCons Tool named "suncc.py" is reusing the tool named "cc.py".
The one-to-one relation between SCons Tools and wrapped executable is not mandatory (there may be a Tool wrapping several executables or several Tools wrapping different functionality of the same executable). Know that no external executable at all is possible (just Python code that you provide). It only happens that, for C/C++ builds, you always wrap existing external executables.
Making a new SCons tool, SCons Construction Environment
The structure of a SCons Tool is very simple. An SCons Tool is a Python script that is required to provide two file scope level functions
The function generate() is called during SCons initialization (if SBuild decides that it needs to start SCons, of course). It has only one in-out parameter: the so-called SCons Construction Environment. It's typically written as env, so this is how we will refer to it hereafter. Only one input parameter that is in fact a large namespace makes a quite loose contract between the SCons Tool and SCons, meaning a lot of freedom plus a lot of responsibility. The env is a dictionary and we call the entries in that dictionary construction variables.
Think of env as mainly an output parameter. The function generate() is supposed to add to the env some construction variables (and their values) specific to the executable that this SCons Tool is describing. Such a variable is, for example, the file name of the C compiler. More precisely, the function generate() for all the tools listed in the attribute scons_tools_to_set of the toolchain variant will be called in order to set up the SCons Construction Environment that will be used for most important actions, like building.
Here is how you set in the SCons Construction Environment the name of the compiler inside the generate(env) function:
env['CC'] = "my_great_compiler"As you see, the env behaves like an ordinary Python dictionary for all practical purposes. The code above supposes that the executable "my_great_compiler" will be found in the PATH at run-time. That path at run-time cannot (and shouldn't) be controlled by what you put in your shell environment where you run the build script. You can control it like this:
env.PrependENVPath('PATH', path_where_toolchain_executables_are_installed)Another alternative is to put an absolute path in the env['CC'] like this:
env['CC'] = path_where_toolchain_executables_are_installed+os.sep+"my_great_compiler"
The important thing in both cases is that it is your code that will somehow compute this path_where_toolchain_executables_are_installed. That's one chief mission of the SCons Tool and it may well be that the code for finding the wrapped executable and providing then the related file paths becomes larger than the code for putting construction variables in the env. See the paragraph below on automatic detection.
The construction environment is also an input parameter in the generate() function. That means that inside the function you can test construction variables and act accordingly. These construction variables are set on the env either by SCons Tools that have been initialized before this one or by SBuild from the scons_constr_var_to_set attribute of the variants tgtplatform and toolchain. For example, you can have in scons_constr_var_to_set PROCESSOR_ARCHITECTURE set to "armv4i" and then use that in your SCons Tool to set a compiler argument to compile for that kind of processor or choose the name of the compiler that compiles for that kind of processor, etc.
Note that the construction environment of SCons can be manipulated also from other places, not only from the SCons tools. One place is the SCons build platform description (look in embedasr\build\scons\scons-local-0.96.1\SCons\Platform).
Note also that the construction variables listed in the scons_constr_var_to_set are set BOTH before and after the generate() function of the tools is called by SCons. This way, what you put in scons_vars_to_set can be used both to control things inside the SCons Tools generate() and to overwrite things after all generate() calls.
You have to know that the value you set for the construction variables can be computed. This computation uses the syntax of the Perl string interpolation. Here is an example:
env['CCCOM'] = '$CC $CCFLAGS -c -o $TARGET $SOURCES'The construction variable CCCOM is the variable used by SCons to make the C compiler command line. The suffix COM is used by convention for all construction variables that are command lines for running executables.
The code above says that the compiler command line is made by expanding the construction variables CC, CCFLAGS, TARGET and SOURCES. If you wonder, know that the variables TARGET and SOURCES are automatically filled-in by SCons for each target. The construction variables CC and CCFLAGS will be provided most likely by your SCons tool. When working to get a command line right, don't forget that the -P option to a build script dumps all the command lines (without running any of them) for that build.
Know that the SCons reference manual gives an extensive list of construction environment variables with their purpose. The two SCons manuals (reference and user guide) are provided as HTML pages with your SBuild installation in <InstallDir>\scons-local-0.96.1\man. Know that you are also free to invent new variables in the Construction Environment (as long as not conflicting with documented ones, of course).
Suppose that your external executable, "my_great_compiler", is only slightly different from the ubiquitous Unix "cc" compiler. Than you will create an SCons tool named "my_great_compiler.py" that reuses the "cc.py" SCons tool (through a Python import statement). Here is your complete SCons tool:
All the other relevant Construction Environment settings are set by the generate() function of an existing SCons Tool, "cc.py", that's why your file is only 5 lines in total.As a side note, this kind of inheritance that doesn't use class inheritance may hurt the feelings of the OOP fans but it works well and SCons will not change it anytime soon.import cc def generate(env): "Add construction variables for my_great_compiler" cc.generate(env) env['CCFLAGS'] = "my_special_compiler options"
Note also that the example above is purely theoretical because if this is the only difference than you don't need a new SCons tool. An addition to the ccflags attribute of the toolchain variant would be enough.
Adding code for the automatic detection of the installed executables
You probably want to release the developers from the chore of telling SBuild/SCons where did they put on their development machine the executables of the toolchain. This requires some code addition to your SCons tools. It can be as simple as adding one fixed, hard coded directory to env['ENV']['PATH'] (the path used at build time to search for executables) or it can be complex code that scans the Windows registry and/or all the hard disks inside out. It is up to you to choose how you support your developers
In an enterprise environment, it also makes sense to not detect anything and enforce instead one unique location for the compiler, linker, etc. across all computers of all developers in the team (a shared location on a file server or a same path location on all individual machines). This avoids that a "non-official" compiler is selected by mistake. That is usually done by manipulating construction variables in the file _sb_custom.py of _sb_users.py of an SBuild project (they are executed before the generate() of the SCons Tools). It is of course possible to combine policies (for some platforms enforce one single official toolchain while for other platforms leave it to whatever is installed on the developer's machine).
It is also possible to provide in your SCons Tool some interactive code. That will start a dialog with the user if all attempts to detect failed, asks the user for the path to the installation and then save it in a local file (so that next run doesn't fire the dialog anymore).
Whatever you choice for specifying the relevant locations of the installed tool chain, don't forget to assert that each directory exists on disk and provide adequate error message if not. It is understandable if, under terrible dead line pressure, you just hardcode in your SCons tool some absolute path. But is not allowed to skip the code that tells to the next user that the installation of the toolchain in that fixed location is a hard requirement of your SCons Tool. Here is an example:
root_dir = r"c:\Program Files\ARM\rvct" assert os.path.isdir(root_dir), "We expect RVCT to be installed in: "+root_dir
If you hardcode, don't forget to hardcode in only one place, by assigning to a Python string variable that you will use afterwards where you need the same path. That variable will hold the root directory of the installation (even if you don't need the root directly but some subdirectory of it).
Step 3: Support in the product build description
This last step depends largely on what C/C++ product you want to port. Some may require no change in the build scripts, some may require a lot of changes. Ideally there is no change needed (meaning the port becomes a mere build) but that remains theoretical for real life products.
The task is to spot all the places in the build description that are customized per target platform. You need to make sure that in all those places, the build description does the right thing for your new target platform. As an example of such place, let's say that executable A is linked with some system library to support audio output. This audio library is target platform specific and also target specific (you don't want any executable you build to link with that audio library). So, you will need to specify what to link with for your executable A for your new platform in the target specification of that executable.
If you miss one such place, don't worry. SBuild will detect the missing information and it will stop at that place reporting what attribute (target attribute or prebuilt external attribute) hasn't been configured for the current target platform. This feature may be disabled by using the so-called "default platform support" for your new platform, see below.
Support in targets
As a reminder, the target specifications are those constructs in the build script introduced as sb.T(). The fact that the build scripts for one product are all in one directory comes in handy at this point. You can scan the build description for targets with attributes mentioning the platform (they all have a name like ***_per_platform, for example ccppdefines_per_platform) and you have a draft list of the places that you may need to touch to finish your porting.
To extend a target specification you have to change directly the target description and add an entry to all attributes with a name like *_per_platform. You have to add an entry even if you have nothing to do in particular for this target and this target platform. You then add an empty list, . If a list is not present, empty or not, SBuild will complain that this is an customization point that was not adapted to all supported target platforms. This mechanism is somewhat heavy but it is precise and clear. Note that empty lists may be added automatically by SBuild using the "default platform support" described below.
Support in prebuilt externals (a.k.a. C/C++
Prebuilt externals or, with more words, SBuild externals of type "sdk", are parts of your product that, for some reason, you don't built from sources. They are explained in more detail on this page: How to use C/C++ SDKs. As a reminder, the prebuilt externals are introduced as sb.E().
Since these prebuilt externals contain code that is already compiled for one target platform, you will obviously need to extend their specification with information about where to find the code compiled for the new target platform that you introduce now. You can search for "sb.E(" in all Python files in the SBuild project and quickly get the list of these specifications (although that will show also other SBuild externals, not only prebuilt externals). The sb.E() items are often present in just one file: _sb_vars.py.
To extend prebuilt externals to new values of tgtplatform, you may use the same "cloning" that you used for extending a variant. This way, assuming that your prebuilt doesn't need any single change for the new platform compared to an already existing platform, you only need 1 line of code: a call to the clone() method.
Default platform support
You can instruct SBuild that, throughout this SBuild project, it should provide automatically something where customization per target platform is expected. This default platform support is of two kinds:
Know that whatever the default platform provides, empty list or copied list, has lower priority than what you give explicitly in the target specification. This mechanism will change a target only if the entry is not present already. For prebuilt externals also what is provided by clone() method call takes precedence over the default platform support.
As a general note, the default platform support mechanism is sometimes the solution (for example when a new platform is very similar to an already supported one). But some other times is no real help: it shuts off all SBuild errors related to not-yet-ported targets it and leaves you with link-time errors or, even worse, run-time errors. Therefore, it is always a good idea to review all places where customization per platform has been provided before. That suggests that your entry in the default platform support list shouldn't be the very first step of your port.
In all build descriptions that are reused for several platforms, there is some danger that a change for one platform breaks the build for another platform. SBuild is no exception. It is good to know that the use of the default platform support can greatly reduce this risk.