Stubs: A dynamic linking mechanism for Tcl, Tk and extensions.
==============================================================

What does it do ?
-----------------

1)  Adds support for backlinking to all platforms, thus making it possible for
    static executables to dynamically load extensions on any platform.

2)  Eliminates all the operating system specific problems associated with
    dynamically linking an extension to Tcl/Tk and any other library which
    exports a stub interface. This may sound like a wild claim but it is true 
    simply because the operating system does not have to do anything.

3)  Extensions and libraries built with different compilers will work together
    even if it is not possible to link them normally. The only requirement for
    them to be able to work together is that function calls are compatible.

4)  As extensions do not have a hard coded reference to the library it is 
    possible to use them with any library which is compatible. e.g. an extension
    which was originally built for Tcl 8.0 should work with Tcl 8.1, and an
    extension built for Tcl 8.1 could work with Tcl 8.0.

How does it do it ?
-------------------

Basically the stub mechanism provides a cross platform dynamic linking 
mechanism using tables of function pointers. 

 o  A library which wishes to provide a stub interface populates a function 
    table with the addresses of its interface functions.

 o  A pointer to that table is then made available to any extension which needs 
    to access that library's interface.

 o  Once an extension has obtained the pointer it simply uses it to get the 
    address of the function it wants to call and calls it.

 o  At no time in this process does the operating system become involved.

To see the differences take a look at the following which describes what happens
when Tcl loads two different extensions.

 o  Loading an extension which is linked directly to Tcl.

    1)  Tcl asks the operating system to load the extension.
    2)  The operating system loads the extension and then tries to resolve any
	undefined symbols. This process is very operating system dependent but 
	involves one or more of the following steps.
	1)  Resolve any symbols which are defined in the current process 
	    context. (Backlinking).
	2)  Find and load any libraries that the extension is dependent on. 
	    This involves searching paths defined through a variety of 
	    operating system dependent methods.
	3)  Resolve any symbols which are defined in the new libraries.
    3)  Tcl then calls the extension's initialisation entry point.

 o  Loading an extension which uses Tcl's stub interface.

    1)  Tcl asks the operating system to load the extension.
    2)  The operating system does so, resolving any non Tcl symbols.
    3)  Tcl calls the extension's initialisation entry point.
    4)  The extension obtains the pointer(s) to Tcl's stub table(s) and uses 
	that to call Tcl.

Where is it ?
-------------

ftp://www.neosoft.com/pub/tcl/sorted/packages-8.0/devel/StubPatch804.tar.gz

How to apply it ?
-----------------

The StubPatch804.tar.gz file contains the following files.
	Tcl_StubPatch.diff	The patch for Tcl.
	Tk_StubPatch.diff	The patch for Tk.
	README			This file.

Unpack them as usual. e.g.
	gunzip -c StubPatch804.tar.gz | tar -xvf -

These patches should only be applied to vanilla 8.0.4.

Move into the Tcl directory which contains the generic directory and apply the
patch making sure to specify -p1. e.g.
	cd Tcl8.0.4
	patch -p1 < ../Tcl_StubPatch.diff

And do the same for Tk. e.g.
	cd Tk8.0.4
	patch -p1 < ../Tk_StubPatch.diff

You are now free to build it.

How to build it ?
-----------------

 o  Unix

     o  Tcl

	There is one new configure option for Tcl.

	--enable-stub
	    If this option is disabled (default) then you will simply get a 
	    normal tcl installation plus a few enhancements. 

	    If it is enabled then Tcl will provide a stub interface for
	    extensions to use. It will also build an archive library called 
	    libtclstub*.a which is used to initialise and access Tcl's stub 
	    interface.

	    You should delete the following files (if they exist) if you have 
	    changed the value of this option.
		tclBasic.o

     o  Tk

	There are two new configure options for Tk; both of the require that Tcl
	is built with stub support.

	--enable-tcl-stub
	    If this option is disabled then Tk will link to Tcl in the normal
	    way.

	    If it is enabled then Tk will use Tcl's stub interface.

	    You should delete the following files (if they exist) if you have 
	    changed the value of this option.
		tkMain.o tkConsole.o

	--enable-stub
	    If this option is disabled then Tk will not provide a stub 
	    interface.

	    If it is enabled then Tk will provide a stub interface for 
	    extensions to use. It will also build an archive library called
	    libtkstub*.a which is used to initialise and access Tk's stub 
	    interface.

	    You should delete the following files (if they exist) if you have 
	    changed the value of this option.
		tkWindow.o

 o  Windows

     o  Tcl

	Two new makefiles have been provided, one for Borland and one for
	Visual C++.

	makefile.vc.stub	(The Visual C++ version)
	makefile.bc.stub	(The Borland version)

	    These makefiles build a version of Tcl which provides a stub
	    interface for extensions to use and will also build a static
	    library tclstub80.lib which is used to initialise and access Tcl's
	    stub interface.

	    You should delete the following files if they exist when building
	    with these makefiles for the first time.
		tclBasic.obj

     o  Tk

	Four new makefiles have been provide, two for Borland and two for
	Visual C++.

	makefile.vc.stub	(The Visual C++ version)
	makefile.bc.stub	(The Borland version)

	    These makefiles build a version of Tk which uses Tcl's stub 
	    interface.

	    You should delete the following files if they exist when building
	    with these makefiles for the first time.
		tkMain.obj tkConsole.obj
	    (Visual C++ users will need to delete all .obj files)

	makefile.vc.stub2	(The Visual C++ version)
	makefile.bc.stub2	(The Borland version)

	    These makefiles build a version of Tk which uses Tcl's stub 
	    interface and also provides one of its own, it will build a static
	    library tkstub80.lib which is used to initialise and access Tk's
	    stub interface.

	    You should delete the following files if they exist when building
	    with these makefiles for the first time.
		tkMain.obj tkConsole.obj tkWindow.obj

 o  Macintosh

    At the moment there is no 'makefile' available to build a Macintosh version.
    However the code has been designed to work with the Macintosh and therefore
    it should not be too difficult to create one. If anyone would like to try
    then I will be happy to provide them with some more details about what needs
    to be done.

Compatability of the stub interfaces with the standard interfaces ?
-------------------------------------------------------------------

The stub interfaces of Tcl and Tk provide access to almost all of the external 
and internal functions both generic and platform dependent. The exceptions to
these rules are ones which it makes no sense to make available through the stub
interface. These include all Tcl command functions plus the following functions:
	Tcl_Main
	Tcl_AppInit
	Tk_Main

Neither Tcl nor Tk provides access to the global variables which are available
with the standard interface. This is for two reasons, the first being that it
is bad practice to access global variables and the second being that they could
only really be accessed using macros which can be very dangerous, especially in
C++.

On Windows and Macintosh the emulated X calls are included as part of the Tk
stub interface.

How to enable an extension to use a stub interface ?
----------------------------------------------------

Using Tcl's stub interface.

1)  Before the extension calls any Tcl function is must call Tcl_Required. This 
    is essentially the same as Tcl_PkgRequire except that the package name is 
    implicitly "Tcl". As this function is supported by both the standard and the
    stub interface the call to it does not need to be conditional. The stub
    interface version checks that stubs are supported and initialises the stub
    table pointers.

2)  The makefile needs to be modified to use the stub library (libtclstub*.a)
    instead of the standard library, and the compile flags need to be changed
    to define ACCESS_TCL_THROUGH_STUB.

3)  An inline versions of the stub interface is available, it uses macros in C
    and inline functions in C++ to access Tcl's interface. These will be used
    if INLINE_TCL_STUB is defined before including tcl.h, alternatively you
    can include tclStubInls.h yourself. The stub library is still needed to
    ensure that the stub initialisation is done correctly.

Using Tk's stub interface.

1)  Before the extension calls any Tk function it must call either Tk_Required
    or Tk_Present. These are essentially the same as Tcl_PkgRequire and 
    Tcl_PkgPresent respectively except that the package name is implicitly "Tk".
    These functions are implemented as a macro in the standard version of the
    interface because Tk may not be present when they are called. The stub
    interface version checks that stubs are supported and initialises the stub
    table pointers.
    
2)  The makefile needs to be modified to use the stub library (libtkstub*.a)
    instead of the standard library, and the compile flags need to be changed
    to defined ACCESS_TK_THROUGH_STUB.

Inline versions of the stub interfaces are available, they uses macros in C
and inline functions in C++ to access the library's interface. The Tcl inline
interface is used if INLINE_TCL_STUB is defined before including tcl.h, or you
can include tclStubInls.h directly. Similarly for Tk. The stub library is still
needed to ensure that the stub initialisation is done correctly.

Unix users should take a look at the tclConfig.sh and tkConfig.sh file to see
what variables are available to help configuring stubs.

How to create an extension with a different compiler ?
------------------------------------------------------

All that is needed to do this is to build the stub library with your compiler.
e.g. assuming that you have a Borland compiler and want to build an extension
which works with a Visual C++ built core you need to do the following.

1)  Get a patched version of the source and make tclstub80.lib. e.g.
	make -f makefile.vc.stub tclstub80.lib
2)  Build your extension making sure that you use the Borland version of the
    stub library.

Although it is possible to link objects built with different compilers together
in most cases, each combination of compilers requires a different process to do
it and starts to get very complicated when you have to link an extension to two
or more libraries which have been built with a combination of compilers and
compiler versions.

This method should work for all combinations of compilers because it only 
requires that function calls are compatible; if they are not then the compilers
cannot be used together anyway.

Changes
-------

The following lists the changes made to Tcl.

 o  Improved the support for AIX shared libraries. By making use of 
    export/import files I was able to create Tcl and Tk libraries which are 
    shared objects rather than archives containing shared objects. This makes 
    it possible to dynamically load Tk on AIX now. Extensions which are 
    correctly configured should not need to make any changes, extensions which 
    are also libraries will need to create an export file and use that instead 
    of the library when linking. See the Tk configure.in file to see how it 
    does it.

 o  Extended the package mechanism to support stubs, and added the ability to
    determine whether a package has been loaded without causing it to be loaded.
    See the PkgRequire.3 man page for more details.

 o  Added ByteArray type. There is a major problem when porting extensions which
    wish to handle binary data from 8.0 to 8.1. In 8.0 they can use string 
    objects to hold the binary data, unfortunately this will not work on 8.1
    because the internal representation of strings is now UTF. Binary data in
    8.1 is handled using ByteArray objects. Therefore I have added them to
    improve the upward compatibility.

 o  For each function which takes a variable number of arguments I have added
    a function which takes a va_list instead. This is so that I can create
    wrapper functions around them. These new functions names are the original
    function names appended with "VA"

The following lists the changes made to Tk.

 o  Modified Tk_Main and TkConsoleCreate to work properly with stubs. The 
    problem is that those functions and the other Tk functions that they use 
    should not actually be in the standard Tk library because they are only 
    really needed by applications. The problem that they cause with stubs is 
    that they are called before the Tk initialisation function is called and 
    therefore the pointers to the Tcl stub tables have not been properly set up
    in the Tk library. The solution was to make them take a Tcl_Interp parameter
    (which is used to get the Tcl stub table pointers) and to get them to 
    initialise the pointers as well. The change to Tk_Main is both binary 
    compatible and source compatible but the change to TkCreateConsole is not 
    because no interface is provided for those functions.

Future
------

There is one more thing that I have left to do which will make this mechanism
even more useful; that is to make the libraries use their stub interface to
access their own functions. This will allow dynamic extensions to change the
behavior of the libraries, either to fix a problem or to add some function.

Plus Patch
----------

The plus patches also have a stub interface available for them although it only
supports a subset of Tcl functions and one Tk function. They are available from
	<http://home.wxs.nl/~nijtmans/plus.html#files80>

Acknowledgements
----------------

Jean-Claude Wippler <jcw@equi4.com>
	For the initial seed idea from which this all came from and also for
	his continued support and feedback during development.

Jan Nitjmans <Jan.Nijtmans@wxs.nl>
	For his ideas and feedback which improved the final result and made
	me think.

Contact
-------

Paul Duffin <pduffin@mailserver.hursley.ibm.com>
