Xlib and X Protocol Test Suite X Version 11 Release 6.1 User Guide for the X Test Suite July 1992 Copyright (C) 1991, 1992 UniSoft Group Limited Permission to use, copy, modify, distribute, and sell this software and its documentation for any purpose is hereby granted without fee, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation, and that the name of UniSoft not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission. UniSoft makes no representations about the suitability of this software for any purpose. It is provided "as is" without express or implied warranty. User Guide for the X Test Suite User Guide for the X Test Suite 1. Introduction This document is a guide to installing and running X Version 11 Release 6.1 of the X Test Suite. In order to do this, please work through all of the steps described in this guide in order. Information on the content, purpose and goals of the X Test Suite is found in a series of appendices, which follow the installation instructions. Please read the "Release Notes for the X Test Suite", which describe particular features of this release. Further information, which would be required by a programmer to modify or extend the X Test Suite, is contained in a separate document, "Programmers Guide for the X Test Suite". Included in this release is a version of the "Test Environment Toolkit" (TET). This is required to build and execute the X Test Suite. The "Test Environment Toolkit" is a software tool developed by X/Open, UNIX International, and the Open Software Foundation. More details of the TET appear in appendix E. The contents of this document cover the installation and use of the included version of the TET. - 1 - User Guide for the X Test Suite 2. Preparation This section of the User Guide describes how to check that the system on which you want to build the X Test Suite has the required utilities and sufficient disc space, how to check the version of the X Window System1 you wish to test, and how to extract the software from the supplied distribution media. 2.1 Utilities The X Test Suite assumes that the following utilities are available on your system. 2.1.1 Bourne shell The configuration and building stages include example instructions which have only been tested using the Bourne shell. The build configuration file sets the SHELL variable so that the Bourne shell will be used by make. No other settings for this variable have been tested. 2.1.2 make The building stages assume the existence of make. 2.1.3 awk The report writer rpt uses awk. 2.1.4 Compiler A C compiler and link editor are required. The X Test Suite assumes that when these utilities execute successfully, they return a value of zero. The names of these utilities may be set in build configuration parameters. 2.1.5 Library archiver A library archiver and a means of ordering the libraries are required. The ordering software may be part of the library archiver, the ranlib utility, or the utilities lorder and tsort. The names of these utilities may be set in build configuration parameters. 2.1.6 File utilities ____________________ 1. The X Window System is a trademark of the Massachusetts Institute of Technology. X Window System Version 11 Release 4 is abbreviated to X11R4 in this document. X Window System Version 11 Release 5 is abbreviated to X11R5 in this document. - 2 - User Guide for the X Test Suite The X Test Suite uses utilities to copy, move, remove and link files during the build stages. The names of these utilities may be set in build configuration parameters. 2.2 Checking your version of the X Window System If your version of the X Window System supports the XTEST extension, you will be able to perform tests for some assertions which are otherwise untestable. The XTEST extension has been produced by MIT since the initial release of X11R5, based on a specification2 produced by UniSoft. The extension provides access to the X server to enable testing of the following areas of the X Window System: -- Those which rely on the simulation of device events. -- Those requiring access to opaque client side data structures. -- Those requiring information on the cursor attribute of windows. Before you configure the X Test Suite, you should determine whether your version of the X Window System includes the XTEST extension, and, if so, whether you wish to configure and build the X Test Suite to enable these features to be tested. There are two things to check: 1. Check whether your X server supports the XTEST extension. This can be done by printing the list of extensions available in your X server using the X utility xdpyinfo. Note - the name of the extension should be printed exactly as in this User Guide - there are other testing extensions for X which are not compatible with the XTEST extension. 2. Check whether you have the required libraries to link the test suite clients so as to access the XTEST extension. All test suite clients must be linked with Xlib, which is normally named libX11.a. If you want to access the XTEST extension, you will need two further libraries. These are the XTEST library (normally named libXtst.a) and the X extension library (normally named libXext.a). 2.3 Installing the X Test Suite Change to the directory in which you wish to install the distribution. Set an environment variable TET_ROOT to the full path name of that directory. ____________________ 2. Drake, K.J., "Some Proposals for a Minimal X11 Testing Extension." UniSoft Ltd. June 1991 - 3 - User Guide for the X Test Suite Load the software from the media supplied into that directory. The precise commands you should use depend on the format of the media supplied to you, the utilities available on your system, the options supported by the utilities, and the names of the tape devices on your system. See the Release Notes for more information about installation. - 4 - User Guide for the X Test Suite 3. Configuring the X Test Suite This section contains instructions on all the procedures you should go through in order to configure the X Test Suite, before attempting to build it. There is a description of the TET build tool in section 3.1, and the relationship between the TET build scheme and the Imake scheme in section 3.2. Sections 3.3 and 3.4 contain details of build and clean configuration parameters, which you should edit to reflect the configuration of the target platform on which the X Test Suite is to be built. Section 3.5 contains details of source files and include files which contain system dependent data which cannot be specified via the build configuration parameters. You should check these files before configuration and if necessary edit them to be suitable for your system. 3.1 The TET build tool The TET provides a scheme to execute a build tool, which builds the tests in the X Test Suite. The execution of the build tool in the TET is controlled by a small number of TET configuration parameters, contained in a build configuration file. These are described in section 3.3.1. A build tool has been developed and is provided as part of the X Test Suite. This is a shell script named pmake, which is supplied in the directory $TET_ROOT/xtest/bin. The shell script pmake is an interface to the make system command, and when invoked from the TET it builds a test using the rules provided in a Makefile. Each Makefile in the X Test Suite is written portably, using symbolic names to describe commands and parameters which may vary from system to system. The values of these symbolic names are all obtained by pmake from additional parameters in the build configuration file which are described in sections 3.3.2-3.3.7. The pmake utility may be invoked directly from the shell, as well as via the TET, to build individual parts of the X Test Suite. This is described further in subsequent sections of this guide. There is also a clean tool pclean which is an interface to the system make clean system command. This uses parameters in a clean configuration file. 3.2 Relationship between TET build scheme and Imake The TET is designed to provide a simple and self contained interface to configure and build tests. The X Test Suite can be configured and built with no specialised knowledge of the - 5 - User Guide for the X Test Suite X Window System beyond that contained in the X Test Suite documentation, and using a limited set of commonly available system commands. The only information required to configure and build the X Test Suite is the location of the X Window System Xlib and include files. The X Window System itself includes a configuration scheme which is known as Imake. This uses a utility imake supplied as part of the X Window System to create Makefiles from portable description files called Imakefiles. If you are familiar with the Imake scheme, and have used it to configure and build the X Window System on the platform being used to build the X Test Suite, you may be able to set a limited number of the TET build configuration variables described in section 3.3 to the same value you used for an Imake variable. Where this is possible, the name of the corresponding Imake variable is cross referenced. 3.3 Build configuration parameters All build configuration parameters are contained in a configuration file that forms part of the TET. This file should be edited to reflect the configuration of the target machine. The file $TET_ROOT/xtest/tetbuild.cfg contains all the parameters that are needed to build the X Test Suite. The parameters are grouped in seven sections within the configuration file. 3.3.1 Configuration Parameters defined by the TET None of these parameters require changing. They are already set to defaults which are correct for the X Test Suite. TET_BUILD_TOOL The name of the program that the TET will execute in build mode. You should use the pmake command that is supplied in the directory $TET_ROOT/xtest/bin. Eg: TET_BUILD_TOOL=pmake TET_BUILD_FILE Any flags required by the build tool. This parameter should be empty. Eg: TET_BUILD_FILE= TET_CLEAN_TOOL The name of the program that the TET will - 6 - User Guide for the X Test Suite execute in clean mode. You should use the pclean command that is supplied in the directory $TET_ROOT/xtest/bin. Eg: TET_CLEAN_TOOL=pclean TET_CLEAN_FILE Any flags required by the clean tool. This parameter should be empty. Eg: TET_CLEAN_FILE= TET_OUTPUT_CAPTURE This flag is used by the TET to enable the output from the build tool to be saved and copied into the journal file. This line should not be altered. 3.3.2 Configuration for system commands In this section the names of system commands are specified. SHELL The following line should cause the Bourne shell to be used by make. Eg: SHELL=/bin/sh CC A command to invoke the C compiler. Eg: CC=cc Imake variable: CcCmd RM A command to remove a file without interactive help. Eg: RM=rm -f Imake variable: RmCmd AR A command to generate a library archive. Eg: AR=ar crv Imake variable: ArCmd LD A command to link object files. Eg: LD=ld - 7 - User Guide for the X Test Suite Imake variable: LdCmd LN A command to make hard links. Eg: LN=ln NB: This does not correspond to the Imake variable: LnCmd RANLIB If the system supports a command to order library archives into random access libraries, then set the parameter to that command. Otherwise it should be set to echo (or a command that does nothing). Eg: RANLIB=ranlib Imake variable: RanlibCmd TSORT Set to cat if AR was set to a command which inserts a symbol table in the library archive, or if RANLIB was set to a command which creates a random access library, otherwise set to tsort. LORDER Set to echo if AR was set to a command which inserts a symbol table in the library archive, or if RANLIB was set to a command which creates a random access library. otherwise set to lorder. CP A command to copy files. Eg: CP=cp Imake variable: CpCmd CODEMAKER A utility to produce C source files from dot- m files. The supplied utility mc should always be used. This line should not be altered. Eg: CODEMAKER=mc 3.3.3 Configuration for the TET This section contains the locations of various parts of the TET. Usually only the first four parameters will need changing, unless files have been moved from their default locations. TET_ROOT The directory that contains all the files in the X Test Suite. This should be set to the - 8 - User Guide for the X Test Suite path to which TET_ROOT was set (see the section entitled "Installing the X Test Suite"). It must be written out as a full path without using any variable notation. TETBASE The directory that contains all the files in the TET system. This is used for convenience in defining the other directories. This should be set to ${TET_ROOT}/tet. PORTINC An option that can be given to the C compiler that will cause it to search all directories that are required to allow portability to systems that do not support POSIX. Should be empty for POSIX systems. If compiling on a BSD system using the supplied compatibility library, then the following line should be used. (See section entitled "The portability library"). Eg: PORTINC=-I${TET_ROOT}/port/INC PORTLIB A library containing POSIX.1 and C library functions that are not supplied by the system. This should be empty for a POSIX system. If compiling on a BSD system using the supplied compatibility library, then the following line should be used. (See section entitled "The portability library"). Eg: PORTLIB=${TET_ROOT}/port/libport.a TETINCDIR The directory containing the TET headers. Eg: TETINCDIR=${TETBASE}/inc/posix_c TETLIB The directory containing the TET library. Eg: TETLIB=${TETBASE}/lib/posix_c TCM The Test Control Manager. This is part of the TET. It is an object file that is linked with each test. Eg: TCM=${TETLIB}/tcm.o TCMCHILD The Test Control Manager. This is part of the TET. It is an object file that is linked with each program that is executed within a test by tet_exec(). Eg: TCMCHILD=${TETLIB}/tcmchild.o - 9 - User Guide for the X Test Suite APILIB The TET API library. Eg: APILIB=${TETLIB}/libapi.a 3.3.4 Configuration parameters for the X Test Suite Only the first two of these parameters require changing unless directories have been moved from their default locations. XTESTHOST The name of the host on which test suite clients are to be executed. This may be set to the value returned by a command which can be executed using the PATH you have set on your host, or may be set to a specific name. This is used to produce a resource file named .Xdefaults-$(XTESTHOST) in the test execution directory. The resource file is created when building the test for XGetDefault. This parameter is only used in the Makefile of the test for XGetDefault. Eg. XTESTHOST=`hostname` Eg. XTESTHOST=`uname -n` Eg. XTESTHOST=triton XTESTFONTDIR The directory in which to install the test fonts. Eg: XTESTFONTDIR=/usr/lib/X11/fonts/xtest XTESTROOT The directory that is the root of the X Test Suite. Eg: XTESTROOT=${TET_ROOT}/xtest XTESTLIBDIR The directory containing libraries for the X Test Suite. Eg: XTESTLIBDIR=${XTESTROOT}/lib XTESTLIB The xtest library. This library contains subroutines that are common to many tests in the X Test Suite. Eg: XTESTLIB=${XTESTLIBDIR}/libxtest.a XSTLIB The X Protocol test library. This library contains subroutines that are common to many tests in the X Protocol section of the X Test Suite. - 10 - User Guide for the X Test Suite Eg: XSTLIB=${XTESTLIBDIR}/libXst.a XTESTFONTLIB The fonts library. This library contains font descriptions that are common to many tests in the X Test Suite. Eg: XTESTFONTLIB=${XTESTLIBDIR}/libfont.a XTESTINCDIR The xtest header file directory. This directory contains headers that are local to the X Test Suite. Eg: XTESTINCDIR=${XTESTROOT}/include XTESTBIN The xtest binary file directory. This directory contains utility programs that are used by X Test Suite. Eg: XTESTBIN=${XTESTROOT}/bin 3.3.5 System Parameters Location of system libraries and include files. SYSLIBS Options to cause the C compiler to search any system libraries that are required for the X Test Suite that are not searched by default. This will probably include Xlib. Eg: SYSLIBS=-lX11 If you wish to build the X Test Suite to make use of the XTEST extension, you will need to include the XTEST library and the X extension library (in that order). Eg: SYSLIBS=-lXtst -lXext -lX11 Imake variables: ExtraLibraries XP_SYSLIBS Any system libraries that are needed, to link the X Protocol tests. This will include Xlib, since libXst.a (which is part of the test suite) will include at least one call to XOpenDisplay. Eg: XP_SYSLIBS=-lX11 Imake variables: ExtraLibraries - 11 - User Guide for the X Test Suite SYSINC Any commands that should be given to the C compiler to cause all relevant system include files to be included. This will probably include /usr/include/X11. Eg: SYSINC=-I/usr/include/X11 3.3.6 C Compiler Directives Directives to the C compiler. Usually only the first four parameters will need changing. The remainder are internally used parameters, which are an amalgam of previously set parameters. COPTS Options to the C compiler. Eg: COPTS=-O Imake variables: DefaultCDebugFlags and DefaultCCOptions DEFINES Options required by the C compiler to set up any required defines. For example in strict ANSI Standard-C systems you will need to define _POSIX_SOURCE. Additionally on an X/Open conformant system it may be necessary to define _XOPEN_SOURCE. Eg: DEFINES=-D_POSIX_SOURCE If there is no symbol NSIG defined in the system header file signal.h, then this has to be supplied for use by the TET API. It should be the number of signal types on the system. Eg: DEFINES=-D_POSIX_SOURCE -DNSIG=32 If you wish to build the X Test Suite to make use of the XTEST extension, you will need to define XTESTEXTENSION. XTESTEXTENSION is only used when building the X Test Suite library. Eg: DEFINES=-D_POSIX_SOURCE -DNSIG=32 -DXTESTEXTENSION Imake variables: StandardDefines XP_DEFINES C compiler defines specific to the X Protocol tests. This can be set as DEFINES, but you can build support for additional connection methods beyond TCP/IP, using the following defines, - 12 - User Guide for the X Test Suite if XP_OPEN_DIS is XlibNoXtst.c (R4/R5 XOpenDisplay emulation): -DDNETCONN - Connections can also use DECnet3. -DUNIXCONN - Connections can also use UNIX4 domain sockets. Refer to your documentation for building and installing Xlib on your platform. If XP_OPEN_DIS is one of XlibXtst.c or XlibOpaque.c then none of the defines listed above will be required. Eg: XP_DEFINES=-D_POSIX_SOURCE -DUNIXCONN Imake variables: StandardDefines LINKOBJOPTS Options to give to the LD program to link object files together into one object file that can be further linked. Eg: LINKOBJOPTS=-r INCLUDES Options to cause C compiler to search the correct directories for headers. This should not need changing as it is just an amalgam of other parameters. INCLUDES=-I. ${PORTINC} -I${TETINCDIR} -I${XTESTINCDIR} ${SYSINC} CFLAGS Flags for the C compiler. This should not need changing as it is just an amalgam of other parameters. Note that CFLOCAL is not defined in the configuration file; it is available for use in makefiles, to define parameters that only apply to a particular case. (It intentionally uses parentheses rather than braces) CFLAGS=$(CFLOCAL) $(COPTS) $(INCLUDES) $(DEFINES) XP_CFLAGS Flags for the C compiler. This parameter is used by the X Protocol tests in the X Test Suite. This should not need changing as it is just an amalgam of other parameters. XP_CFLAGS=$(CFLOCAL) $(COPTS) $(INCLUDES) $(XP_DEFINES) ____________________ 4. DEC and DECnet are registered trademarks of Digital Equipment Corporation. 4. UNIX is a registered trademark of UNIX System Laboratories, Inc. in the U.S. and other countries. - 13 - User Guide for the X Test Suite LDFLAGS Flags used by the loader. This is needed on some systems to specify options used when object files are linked to produce an executable. Eg. LDFLAGS=-ZP LIBS List of libraries. This should not need changing as it is just an amalgam of other parameters. LIBS=${XTESTLIB} ${XTESTFONTLIB} ${APILIB} ${PORTLIB} XP_LIBS List of libraries. This parameter is used by the X Protocol tests in the X Test Suite. This should not need changing as it is just an amalgam of other parameters. XP_LIBS=${XSTLIB} ${XTESTLIB} ${XTESTFONTLIB} ${APILIB} ${PORTLIB} XP_OPEN_DIS A choice of which code to build in the X Protocol library to make an X server connection. This must be set to one of three possible values: XlibXtst.c Use this option only if your Xlib includes post R5 enhancements to _XConnectDisplay ensuring maximum portable protocol test coverage. These enhancements include arguments to _XConnectDisplay to return authorisation details on connection. If you use this option when your Xlib does not have these enhancements to _XConnectDisplay, the results of running the X Protocol tests will be undefined. XlibOpaque.c You have a normal R4 Xlib or early R5 Xlib which you cannot patch to include the enhancements to _XConnectDisplay, and you cannot emulate these by building XlibNoXtst.c, so only client-native testing can be done portably, and no failure testing of XOpenDisplay can be done. This option uses XOpenDisplay to make the connection, from which the file descriptor is recovered for our own use. XCloseDisplay shuts down the connection. XlibNoXtst.c As for XlibOpaque.c but you can use the R4/R5 connection emulation supplied. - 14 - User Guide for the X Test Suite (Note: R4/R5 independent) This will ensure maximum protocol test coverage but may not be portable to all platforms. Reasons for not being able to build XlibNoXtst.c might include: i) different interfaces to connection setup and connection read/write; ii) different access control mechanisms. Refer to your Xlib documentation for further details. Eg. XP_OPEN_DIS=XlibOpaque.c 3.3.7 Pixel validation section This section defines a number of parameters that are used only when generating known good image files. These are not intended to be modified and need not be used when running the test suite. They are only used in the development environment at UniSoft when generating known good image files. 3.4 Clean configuration parameters The TET provides a scheme to execute a clean tool, which removes previously built tests and object files. All clean configuration parameters are contained in a configuration file that forms part of the TET. The file $TET_ROOT/xtest/tetclean.cfg contains all the parameters that are needed to clean the X Test Suite. To save configuration effort, we have arranged that the build and clean configuration files may contain identical parameter settings. Both files are needed, since the TET requires both a default build and clean configuration file. Copy the build configuration file into the clean configuration file: cd $TET_ROOT/xtest cp tetbuild.cfg tetclean.cfg 3.5 System dependent source files This section describes source files and include files provided in the X Test Suite which contain data, which you may need to edit to reflect the system under test. 3.5.1 Host address structures - 15 - User Guide for the X Test Suite The file xthost.c in the directory $TET_ROOT/xtest/src/lib contains three items, which you may need to edit to reflect the system under test. These are all related to the mechanisms provided by the X server under test to add, get or remove hosts from the access control list. These are only used in the tests for those Xlib functions which use or modify the access control list. The host access control functions use the XHostAddress structure. You should refer to the Xlib documentation for your system, to determine the allowed formats for host addresses in an XHostAddress structure. You may also find it helpful to refer to the X Window System Protocol documentation supplied with the X server under test. The section describing the ChangeHosts protocol request gives examples of host address formats supported by many X servers. The symbols FamilyInternet, FamilyDECnet, FamilyChaos and FamilyUname are defined on many systems in the include files X.h and Xstreams.h. The X server under test is not guaranteed to support these families, and may support families not listed here. You should find out which families are supported for the X server under test, by examining the header files supplied with your system, and consulting the documentation supplied with the X server. Some default declarations are contained in the file, but there is no guarantee that they will work correctly on your system. The three items are as follows: 1. You should ensure that there is a declaration for an array xthosts[] of at least 5 XHostAddress structures containing valid family,length,address triples. 2. You should ensure that there is a declaration for an array xtbadhosts[] of at least 5 XHostAddress structures containing invalid family,length,address triples. You should ensure that there is a declaration for an array xtbadhosts[] of at least 5 XHostAddress structures containing invalid family,length,address triples. If you cannot use the supplied examples, the simplest way to do this is to use an invalid family, which is not supported by the X server under test, in each structure of the array. 3. You should ensure that there is a declaration for a function samehost() that compares two XHostAddress structures and returns True if they are equivalent. (It is unlikely that the sample function will need modification - no systems requiring modification have yet been identified). - 16 - User Guide for the X Test Suite 4. Building the TET The X Test Suite runs under the Test Environment Toolkit (TET). This section of the User Guide tells you how to build and install the supplied version of the TET. The following instructions assume the use of a Bourne shell. The PATH variable should have the directory $TET_ROOT/xtest/bin prepended to it. PATH=$TET_ROOT/xtest/bin:$PATH export PATH 4.1 The portability library The current version of the TET used by the X Test Suite is designed to run on a POSIX.1 system. Since many systems running the X Window System currently run on BSD based systems a portability library that emulates the required routines using BSD facilities has been provided. This library is not part of the TET itself. The portability library source is kept in $TET_ROOT/port. The portability library may be useful in porting the X Test Suite to other environments as described below. Please refer to the "Release Notes for the X Test Suite" for details of the target execution environments, and a list of systems to which it has already been ported. 4.1.1 Porting to a POSIX.1 system If your system conforms to POSIX.1 and has an ANSI Standard- C compiler then this library should not be built and so this section can be skipped. The exception is that putenv() may not exist on a POSIX.15 system, however it is in the SVID6 and the XPG7, so in practice most non-BSD machines will have this function. 4.1.2 Porting to a BSD system ____________________ 5. IEEE Std 1003.1-1990, Portable Operating System Interface for Computer Environments 6. System V Interface Definition, Issue 1, AT&T, Spring 1985. 7. X/Open Portability Guide Issue 3, Volume 2: XSI System Interface and Headers - 17 - User Guide for the X Test Suite If the system is a standard BSD one, then the portability library can be used as it is; build it as follows. cd $TET_ROOT/port pmake 4.1.3 Porting to other systems The portability library may be useful as a base for porting the TET to other non-POSIX systems, however the portability library is designed to run on a BSD system, and will not necessarily build without change on other systems. The following routines are emulated for use under a BSD system, and these may be needed on other systems: getcwd() getopt() putenv() sigaction() sigaddset() sigdelset() sigemptyset() sigfillset() sigismember() sigpending() sigprocmask() sigsuspend() strchr() strcspn() strftime() strpbrk() strrchr() strspn() strtok() toupper() upcase() vsprintf() waitpid() Only the features that are used by the X Test Suite are emulated. They are not meant to mimic completely the standard behaviour. There is also an include directory $TET_ROOT/port/INC that contains header files that are required that are not found on a BSD system. These files contain only items that are needed for the X Test Suite, they are not designed to replace completely the standard ones. To adapt the portability library to other systems, the following hints may be found useful: o Examine the directory $TET_ROOT/port/INC. If the system already provides a standard conforming header file of the same name as one in the INC directory, then remove the version from the INC directory. o The header files contain the bare minimum required to compile the X Test Suite, and use BSD features. It may be necessary to alter them to suit the local system. This applies particularly to signal.h. o It may be necessary to add other header files. o In the library Makefile remove any function that is already provided by the system in a standard conforming form. o Examine the code of the remaining functions to make sure that they will work on the target system. - 18 - User Guide for the X Test Suite 4.2 Building libraries and utilities There is a top level Makefile which can be used to automatically perform a number of the following steps. You should still check through the User Guide and perform the steps which need to be done manually. In particular you need to build and install the test fonts as described in the section entitled "Compiling and installing the test fonts". The top level Makefile enables the following steps to be performed: Building the TET: 1. The Test Case Controller (TCC) 2. The API library Building the X test suite libraries and utilities: 1. Building the X test suite library 2. Building the X Protocol library 3. Building the X test fonts library 4. Building the mc utility 5. Building the blowup utility To use the top level Makefile, move to the top level directory: cd $TET_ROOT Make the utilities and libraries with the command: make 4.3 The Test Case Controller (TCC) Move to the directory containing the TCC source. cd $TET_ROOT/tet/src/posix_c/tools Make the TCC with the command: pmake install Note: the supplied version of the TCC assumes that the cp utility on your system supports recursive copy using the option -r. There are two occurrences of cp in the file exec.c which use this option. - 19 - User Guide for the X Test Suite In the X Test Suite, recursive copying is not required. If your system does not support this option, you can remove the use of this option in the source code before building the TCC. If you do this you may not be able to use the supplied TCC with other test suites. Alternatively, you can provide a shell script in the directory $TET_ROOT/xtest/bin which copies files using cp but ignores any option -r. 4.4 The API library Move to the API library source directory. cd $TET_ROOT/tet/src/posix_c/api Run the command pmake install which should produce the files libapi.a and the Test Case Manager files tcm.o and tcmchild.o. 5. Building the X test suite libraries and utilities 5.1 The X test suite library A library of common subroutines for the X Test Suite has source in $TET_ROOT/xtest/src/lib. This is built automatically when building tests in the X Test Suite. Should it be required to build it separately for any reason run the command. cd $TET_ROOT/xtest/src/lib pmake install The list of source files in this library is described in the "Programmers Guide". 5.2 The X Protocol library A library of common subroutines for the X Protocol tests in the X Test Suite has source in $TET_ROOT/xtest/src/libproto. This is built automatically when building tests in the X Test Suite. Should it be required to build it separately for any reason run the command. cd $TET_ROOT/xtest/src/libproto pmake install The list of source files in this library is described in the "Programmers Guide". - 20 - User Guide for the X Test Suite 5.3 The X test fonts library A library of common subroutines defining the characteristics of the test fonts for the X Test Suite has source in $TET_ROOT/xtest/fonts. This is built automatically when building tests in the X Test Suite. Should it be required to build it separately for any reason run the command. cd $TET_ROOT/xtest/fonts pmake install The list of source files in this library is described in the "Programmers Guide". Note that the directory $TET_ROOT/xtest/fonts also contains the test fonts themselves in bdf format, which must be compiled and installed. Instructions for performing these steps are included in the next section entitled "Compiling and installing the test fonts". 5.4 Compiling and installing the test fonts The X Test Suite contains a series of test fonts which are used to test the correctness of the information returned by the graphics functions in the X Window System. This is done by comparing the information returned by those functions with the expected font characteristics which are compiled into the tests via the X test fonts library. The X test fonts library is described in an earlier section of this document. There are seven test fonts whose descriptions are contained in the files xtfont0.bdf xtfont1.bdf xtfont2.bdf xtfont3.bdf xtfont4.bdf xtfont5.bdf xtfont6.bdf These files are located in the directory $TET_ROOT/xtest/fonts. The manner in which fonts should be compiled and installed for any particular X server is system dependent, and you should refer to the instructions supplied with your release of the X Window System for details of how to do this. Some sample instructions are given here which may be useful on many systems. These may not be appropriate for your system, or they may need adaptation to work properly on your system and so are provided only as a guide. 1. Move to the directory $TET_ROOT/xtest/fonts. cd $TET_ROOT/xtest/fonts - 21 - User Guide for the X Test Suite 2. Compile the seven bdf files into snf, pcf, or fb format, as appropriate for your system. pmake comp_snf or pmake comp_pcf or pmake comp_dxpcf or pmake comp_fb 3. Copy the compiled fonts into the server font directory (the XTESTFONTDIR configuration parameter). pmake install_snf or pmake install_pcf or pmake install_dxpcf or pmake install_fb 5.5 Building the mc utility The mc utility is used to generate test set source files and Makefiles from a template file, known as a dot-m file. The file naming scheme is described further in appendix B. The file formats are described further in the "Programmers Guide". The Makefiles and test set source files will be created using mc whenever test sets are built, if the dot-m file is found to be newer than the source file or Makefile, or if these files do not exist. Build mc and install in the xtest bin directory as follows. cd $TET_ROOT/xtest/src/bin/mc pmake install cd $TET_ROOT/xtest/src/bin/mc/tmpl pmake install - 22 - User Guide for the X Test Suite 5.6 Building the blowup utility The blowup utility is required for examining any incorrect image files generated by the X server during a test run. Instructions for running the blowup program are given in the section entitled "Examining image files". Build blowup and install in the xtest bin directory as follows. cd $TET_ROOT/xtest/src/pixval/blowup pmake install - 23 - User Guide for the X Test Suite 6. Building the tests 6.1 Building tests using the TET The entire X Test Suite can be built by using the build mode of the TCC. In this mode, the build configuration parameters in the file $TET_ROOT/xtest/tetbuild.cfg are used to build each test set in the X Test Suite separately. cd $TET_ROOT/xtest tcc -b [ -s scenario_file ] [ -j journal_file ] [ -y string ] xtest all -b This invokes the TCC in build mode. (If you have just finished building the TCC from the csh, you will probably have to rehash to get tcc in your path.) -s scenario_file This option builds the test sets in the named scenario file. The default is a file named tet_scen in the directory $TET_ROOT/xtest. For more details refer to the section entitled "Building modified scenarios using the TET". -j journal_file This option sends the output of the build to the named journal file. The default is a file named journal in a newly created sub-directory of $TET_ROOT/xtest/results. Sub-directories are created with sequential four digit numbers, with the TCC flags (in this case "b") appended. The TCC will exit if the specified journal file already exists, thus the journal file should be renamed or removed before attempting to execute the TCC. -y string This option only builds tests which include the specified string in the scenario file line. This may be used to build specific sections or individual test sets. xtest This is the name of the test suite. It determines the directory under $TET_ROOT where the test suite is to be found. all This is the scenario name in the default scenario file $TET_ROOT/xtest/tet_scen. For more details refer to the section entitled "Building modified scenarios using the TET" This will execute the TET build tool in the TET configuration variable TET_BUILD_TOOL (which is normally pmake), in each test set directory of the X Test Suite. - 24 - User Guide for the X Test Suite The journal file should be examined to verify that the build process succeeded. The report writer rpt cannot interpret the contents of a journal file produced during the build process. Note: If the TCC terminates due to receipt of a signal which cannot be caught, the TCC may leave lock files in the test source directories. Subsequent attempts to restart the TCC may give error messages saying that a lock file was encountered. At this point TCC may suspend the build. It may be necessary to find and remove files or directories named tet_lock before continuing. 6.1.1 Signal handling in the TET An interrupt signal (caused for example by typing the system interrupt character on the controlling terminal) will cause the TCC to abort the currently executing test case. The journal file output records the fact that the test case was interrupted. Any other signal which can be caught by the TCC causes it to terminate. By default, the system suspend character will also cause the TCC to terminate. If you wish to be able to suspend the TCC, you can add the relevant signals to the parameter SIG_LEAVE in the Makefile for the TCC. Signals in this list will not be caught, but will cause their default action. This is explained further in the Test Environment Tookit Release Notes. 6.2 Building, executing and cleaning tests using the TET Each test in the X Test Suite may be built, executed and cleaned before the next test set in the scenario. This mode of use has the advantage that the entire X Test Suite may be executed, without necessarily building all the test sets in advance. This mode of use has the disadvantage that you will need to rebuild a test set before rerunning, which will take considerably longer than when it is built in advance. To do this, skip to the section entitled "Executing the X Test Suite", and refer to the instructions in the sub- section entitled "Building, executing and cleaning tests using the TET" 6.3 Building modified scenarios using the TET 6.3.1 Format of the scenario file The TET uses a scenario file to determine which test sets to build. The file $TET_ROOT/xtest/tet_scen is the default scenario file. The format is basically a scenario name starting in column one, followed by list of test sets to be built (each starting beyond column one). Only one scenario named "all" is provided in the default scenario file. The names of the test sets are given relative to the - 25 - User Guide for the X Test Suite directory $TET_ROOT/xtest, and must commence with a leading slash. 6.3.2 Modifying the scenario file The file $TET_ROOT/xtest/tet_scen may be modified by removing lines corresponding to test sets which are not wanted. These will then simply not be built by the TCC. Alternatively, unwanted lines may be commented out by placing # in column one of a line. It is recommended that the supplied scenario file should be saved if it is modified. 6.3.3 Creating new scenario files A new scenario file may be created in the directory $TET_ROOT/xtest. The TCC will use this scenario file instead of the file $TET_ROOT/xtest/tet_scen if it is passed via the -s option. For example cd $TET_ROOT/xtest tcc -b -s scenario_file [ -j journal_file ] [ -y string ] xtest all 6.4 Building tests without using the TET See section 11, entitled "Building, executing and reporting tests without using the TET". 6.5 Building tests in space-saving format It is possible to build the tests in the X Test Suite such that all the executable files in one section are links to a single executable file. This normally allows a considerable reduction in the disc space requirements for the X Test Suite when fully built. Note that the names of the files built in space-saving format are different to the names of the separate executable files built using the instructions in previous sections. There is nothing to prevent both sets of executables being built (although there is no value in this, and unnecessary disc space will be consumed). 6.5.1 Building tests in space-saving format using the TET Before reading this section, read the section entitled "Building the tests using the TET". This gives an explanation of the build mode of the TET, and the structure of scenario files. A scenario named linkbuild is provided in a scenario file named link_scen in the directory $TET_ROOT/xtest. This enables the TCC to build the space-saving executable files and create all the required links for each test set in each section of the X Test Suite. The -y option allows a - 26 - User Guide for the X Test Suite particular space-saving executable for a single section to be built. Execute the command: cd $TET_ROOT/xtest tcc -b -s link_scen [ -j journal_file ] [ -y string ] xtest linkbuild This command will execute the TET build tool in the TET configuration variable TET_BUILD_TOOL (which is normally pmake), in the top level directory of each section of the test suite. 6.5.2 Building tests in space-saving format without using the TET This section describes how to build the space-saving executable files for a particular section of the X Test Suite directly without using the TET. This can be simply done by calling pmake in the required directory. For example, to build all the space-saving executable files for section 5 of the X Test Suite, execute the command: cd $TET_ROOT/xtest/tset/CH05 pmake - 27 - User Guide for the X Test Suite 7. Executing the X Test Suite Once you have built the X Test Suite as described in the previous sections, work through the following sections to execute the tests. 7.1 Setting up your X server The first step is to ensure that the X server to be tested is correctly set up. 7.1.1 Formal verification testing A number of the tests within the X Test Suite can only give reliable results if there is no window manager and no other clients making connections to the X server. Thus, when conducting formal verification tests, there should be no window manager and no other clients connected to the X server. It is recommended that you close down and restart your X server before a formal verification test run, in order to ensure that results produced are repeatable and are not affected by earlier tests, although this is not strictly necessary. You should switch off the screen saver if possible before starting formal verification tests. This is because some X servers implement the screen saver in a way which interferes with windows created by test suite clients, which may cause misleading results. If the screen saver cannot be switched off, the time interval should be set so large as to prevent interference with the tests. You should also ensure that access control is disabled for the server under test, so that the test suite can make connections to the server. Also (if the X server allows this) you should ensure that clients on the host system (as specified in the build configuration parameter XTESTHOST) can modify the access control list. Some X servers support the -ac option which disables host-based access control mechanisms. If this option is supported, you should use it. 7.1.2 Informal testing and debugging Although no guarantee can be made that the tests within the X Test Suite will give correct results if there are window managers and other clients connected to the X server, it is still possible to run many tests satisfactorily. This section gives some guidelines which may be helpful in running tests with a window manager present, and still deriving correct results. The guidelines have been derived from the experience gained during the development of the tests. Using these guidelines in connection with the instructions - 28 - User Guide for the X Test Suite in section 11, entitled "Building, executing and reporting tests without using the TET", gives a rapid means to investigate the results of particular tests in detail. 1. Set XT_DEBUG_OVERRIDE_REDIRECT=Yes in your execution configuration file. This is described in more detail in the next section. 2. Do not raise any windows on top of those created by running tests. 3. Avoid having any windows at position (0,0). Note that some window managers such as tvtwm create their own "root" window at position (0,0). This mainly affects tests for section 8 of the X11R4 Xlib specifications. 4. Be prepared to lose the input focus when tests are running and don't forcibly restore it. This mainly affects tests for section 8 of the X11R4 Xlib specifications. 7.2 Execute configuration parameters The next step is to set up the execution configuration file. All execution configuration parameters are contained in a configuration file that forms part of the TET. This file should be edited to reflect the configuration of the X server to be tested and the underlying operating system on which Xlib is implemented. The file $TET_ROOT/xtest/tetexec.cfg contains all the parameters that are needed to execute the X Test Suite. The parameters are grouped in eight sections within the configuration file. Numeric execution parameters may be specified in decimal, octal, or hexadecimal. Octal values must be a sequence of octal digits preceded by 0. Hexadecimal values must be a sequence of hexadecimal digits preceded by 0x or 0X. 7.2.1 Configuration parameters defined by the TET TET_EXEC_IN_PLACE Setting this variable to False indicates that files will be executed in a temporary execution directory. Use of a temporary execution directory for each test enables parallel execution of the test suite against multiple servers. Setting this variable to True will give you improved performance if you are not attempting parallel execution of the test suite against multiple servers. Eg: TET_EXEC_IN_PLACE=False - 29 - User Guide for the X Test Suite TET_SAVE_FILES This indicates which files generated during execution of tests are to be saved for later examination. This line should not be altered. Eg. TET_SAVE_FILES=Err*.err,*.sav 7.2.2 Configuration Parameters for the X Test Suite The following parameters are used in many places in the X Test Suite. These should be set to match the X server to be tested and the underlying operating system on which Xlib is implemented. XT_DISPLAY This should be set to a display string that can be passed to XOpenDisplay, to access the display under test. It must include a screen; all testing is done for a particular screen. Eg: XT_DISPLAY=:0.0 XT_ALT_SCREEN If the display supports more than one screen, this parameter should be set to the number of a screen that is different from that incorporated in the XT_DISPLAY variable. Set to the string UNSUPPORTED if only one screen is available. Note that this should be a screen number, not a display string that can be passed to XOpenDisplay. Eg: XT_ALT_SCREEN=1 XT_FONTPATH This should be set to a comma separated list that is a valid font path for the X server. It should include at least the components of the default font path for the X server, enabling the cursor font to be accessed. One of the components must be the directory in which the test fonts were installed (see the section entitled "Compiling and installing the test fonts"). This parameter will be used to set the font path for specific test purposes which access the test fonts. The font path is restored on completion of the specific test purposes. Eg: XT_FONTPATH=/usr/lib/X11/fonts/xtest/,/usr/lib/X11/fonts/misc/ XT_SPEEDFACTOR This is a speedfactor which should be set to reflect the relative delay in response of the underlying - 30 - User Guide for the X Test Suite operating system and X server combined. Co-operating processes which must synchronize allow a time delay in proportion to this speedfactor, to account for scheduling delays in the underlying operating system and X server. This should be set to a number greater than or equal to one. There should be no need to change the default unless the round trip time to the X server can be very long ( >15 seconds); in this case set this parameter to a value larger than the maximum round trip time divided by 3. Eg: XT_SPEEDFACTOR=5 XT_RESET_DELAY Specifies a delay time in seconds. Set this to be a time which is greater than or equal to the maximum time required by your server to reset when the last client is closed. The test suite pauses for this time whenever a connection is about to be opened and the server may be resetting. The server may be resetting when the test case is entered (in startup()) as a result of closing the last connection in the previous test case. The server also resets in a few places in the test for XCloseDisplay(). Eg. XT_RESET_DELAY=1 XT_EXTENSIONS Specifies whether you wish to test the extended assertions which require the XTEST extension. Set this to Yes if the XTEST extension is available on your system, and you have configured the test suite to use the XTEST extension, and you want to execute these tests, otherwise set to No. Eg. XT_EXTENSIONS=No 7.2.3 Configuration parameters for specific tests The following parameters are used to control one or more specific test purposes in the X Test Suite. These should be set to appropriate values for the X server to be tested. These parameters may cause temporary changes in the settings of the X server under test (such as the font path). Settings are restored on completion of the specific test purposes. XT_VISUAL_CLASSES A space separated list of the visual classes that are supported for the screen given by XT_DISPLAY. Each visual class is followed by a list of depths at which the class is supported (enclosed by brackets and separated by commas with no spaces). Visual classes - 31 - User Guide for the X Test Suite and depths that are supported only by other screens should not be included. Note that this parameter is only used to check the correctness of the values returned by XMatchVisualInfo and XGetVisualInfo. Other tests which loop over visuals obtain the values by calling these functions. Eg. XT_VISUAL_CLASSES=StaticGray(8) GrayScale(8) StaticColor(8) PseudoColor(8) TrueColor(8) DirectColor(8) (This must be typed as one line.) XT_FONTCURSOR_GOOD This specifies the number of a glyph in the default cursor font known to exist. XT_FONTCURSOR_GOOD+2 should also be a glyph in the default cursor font. Neither of these should be the same as the X server's default cursor. Eg: XT_FONTCURSOR_GOOD=2 XT_FONTCURSOR_BAD This specifies the number of a glyph in the default cursor font known not to exist. If no such value exists, set to UNSUPPORTED. Eg: XT_FONTCURSOR_BAD=9999 XT_FONTPATH_GOOD This should be set to a comma separated list that is a valid font path for the X server. It should be different from XT_FONTPATH. It need not contain the test fonts. Eg: XT_FONTPATH_GOOD=/usr/lib/X11/fonts/100dpi/,/usr/lib/X11/fonts/75dpi/ XT_FONTPATH_BAD This should be set to a comma separated list that is an invalid font path for the X server. If you cannot determine a suitable value, set to UNSUPPORTED. There is no default value - by default, tests which use this parameter will be reported as UNSUPPORTED. Eg: XT_FONTPATH_BAD=/jfkdsjfksl XT_BAD_FONT_NAME This should be set to a non-existent font name. XT_BAD_FONT_NAME=non-existent-font-name XT_GOOD_COLORNAME This should be set to the name of a colour which exists - 32 - User Guide for the X Test Suite in the colour database for the X server. Eg: XT_GOOD_COLORNAME=red XT_BAD_COLORNAME This should be set to the name of a colour which does not exist in the colour database for the X server. Eg: XT_BAD_COLORNAME=nosuchcolour XT_DISPLAYMOTIONBUFFERSIZE This should be set to a non-zero value (the value returned by XDisplayMotionBufferSize) if the X server supports a more complete history of pointer motion than that provided by event notification, or zero otherwise. The more complete history is made available via the Xlib functions XDisplayMotionBufferSize and XGetMotionEvents. Eg: XT_DISPLAYMOTIONBUFFERSIZE=256 7.2.4 Configuration parameters for Display functions The following parameters are used to control one or more test purposes for Xlib Display functions which are in section 2 of the X11R4 Xlib specifications. These should be set to match the display specified in the XT_DISPLAY parameter. Some of these parameters are specific to the particular screen of the display under test. This is also specified in the XT_DISPLAY parameter. Settings to these parameters will not cause any change in the settings of the X server under test. Suitable values for most of these parameters can be obtained from the output of the X11 utility xdpyinfo. XT_SCREEN_COUNT This parameter should be set to the number of screens available on the display as returned by XScreenCount. Eg: XT_SCREEN_COUNT=2 XT_PIXMAP_DEPTHS A space separated list of depths supported by the specified screen of the display that can be used for pixmaps. Eg: XT_PIXMAP_DEPTHS=1 8 - 33 - User Guide for the X Test Suite XT_BLACK_PIXEL This parameter should be set to the black pixel value of the specified screen of the display. Eg: XT_BLACK_PIXEL=1 XT_WHITE_PIXEL This parameter should be set to the white pixel value of the specified screen of the display. Eg: XT_WHITE_PIXEL=0 XT_HEIGHT_MM This parameter should be set to the height in millimeters of the specified screen of the display. Eg: XT_HEIGHT_MM=254 XT_WIDTH_MM This parameter should be set to the width in millimeters of the specified screen of the display. Eg: XT_WIDTH_MM=325 XT_PROTOCOL_VERSION This should be set to the major version number (11) of the X protocol as returned by XProtocolVersion. Eg. XT_PROTOCOL_VERSION=11 XT_PROTOCOL_REVISION This should be set to the minor protocol revision number as returned by XProtocolRevision. Eg. XT_PROTOCOL_REVISION=0 XT_SERVER_VENDOR This should be set to the X server vendor string as returned by XServerVendor. Eg: XT_SERVER_VENDOR=MIT X Consortium XT_VENDOR_RELEASE This should be set to the X server vendor's release number as returned by XVendorRelease. Eg. XT_VENDOR_RELEASE=5000 - 34 - User Guide for the X Test Suite XT_DOES_SAVE_UNDERS Set this to Yes if the specified screen of the display supports save unders (indicated by XDoesSaveUnders returning True) otherwise set to No. Eg. XT_DOES_SAVE_UNDERS=Yes XT_DOES_BACKING_STORE Set this to the following value: 0 - the specified screen supports backing store NotUseful 1 - the specified screen supports backing store WhenMapped 2 - the specified screen supports backing store Always The way the specified screen supports backing store is indicated by the return value of XDoesBackingStore. Eg. XT_DOES_BACKING_STORE=2 7.2.5 Configuration parameters for connection tests The following parameters are used to control one or more test purposes for XOpenDisplay, XCloseDisplay and XConnectionNumber. These should be set to match the display specified in the XT_DISPLAY parameter and the characteristics of the underlying operating system. Settings to these parameters will not cause any change in the settings of the X server under test. These parameters are not used when making connections to the X server in other tests. XT_POSIX_SYSTEM This may be set to Yes to indicate that the underlying operating system is a POSIX system. If this parameter is set to Yes, some extended assertions which describe implementation dependent functionality will be tested assuming POSIX concepts. Eg. XT_POSIX_SYSTEM=Yes XT_DECNET Set this to Yes if clients can connect to the X server under test using DECnet. This will be used (on a POSIX system) in the tests for XOpenDisplay. Eg. XT_DECNET=No XT_TCP Set this to Yes if clients can connect to the X server under test using TCP streams. This will be used (on a POSIX system) in the tests for XOpenDisplay. - 35 - User Guide for the X Test Suite Eg. XT_TCP=Yes XT_DISPLAYHOST Set this to the hostname of the machine on which the display is physically attached. This will be used instead of XT_DISPLAY (on a POSIX system) in the tests for XOpenDisplay which specifically test the hostname component of the display name. Note that this may not be the same as the machine on which the test suite clients execute (XTESTHOST). Eg. XT_DISPLAYHOST=xdisplay.lcs.mit.edu XT_LOCAL Set this to Yes if clients can connect to a local X server without passing a hostname to XOpenDisplay. This will be used (on a POSIX system) in the tests for XOpenDisplay. This is usually the case when the X server under test is running on the same platform as the X Test Suite. When a hostname is omitted, the Xlib implementation of XOpenDisplay can use the fastest available transport mechanism to make local connections. Eg. XT_LOCAL=No 7.2.6 Configuration Parameters which do not affect test results There are a number of execution configuration parameters which can be used to reduce the size of the journal file, or dump out more information from the test suite. They will not alter the behaviour of the tests or the test results. XT_SAVE_SERVER_IMAGE When set to Yes, the image produced by the server that is compared with the known good image is dumped to a file with suffix ".sav" . Eg: XT_SAVE_SERVER_IMAGE=Yes XT_OPTION_NO_CHECK This may be set to Yes to suppress the journal file records containing CHECK keywords. Refer to appendix D for information on the contents of these messages. Eg: XT_OPTION_NO_CHECK=Yes XT_OPTION_NO_TRACE This may be set to Yes to suppress the journal file records containing TRACE keywords. Refer to appendix D - 36 - User Guide for the X Test Suite for information on the contents of these messages. Eg: XT_OPTION_NO_TRACE=Yes 7.2.7 Configuration Parameters for debugging tests There are a number of execution configuration parameters which should not be set when performing verification test runs. These are intended for debugging purposes. These parameters may affect the behaviour of some test purposes if they are set to assist debugging. XT_DEBUG This may be set to a debugging level. A higher level produces more debugging output. Output is only produced by the test suite at levels 1, 2 and 3. Setting this variable to 0 produces no debug output, and 3 gives everything possible (setting this variable to 3 can give an enormous volume of output so you should not do this when running large numbers of test sets). Eg: XT_DEBUG=0 XT_DEBUG_OVERRIDE_REDIRECT When set to Yes, windows are created with override_redirect set. This enables tests to be run more easily with a window manager running on the same screen. This should not be set to Yes for verification tests. Eg: XT_DEBUG_OVERRIDE_REDIRECT=No XT_DEBUG_PAUSE_AFTER When set to Yes, the test pauses after each call to the Xlib function being tested, until Carriage Return is entered. This is useful to enable the results of graphics operations to be observed. This should not be set to Yes for verification tests. Eg: XT_DEBUG_PAUSE_AFTER=No XT_DEBUG_PIXMAP_ONLY When set to Yes, tests which would normally loop over both windows and pixmaps are restricted to loop over just pixmaps. This is useful for speeding up the execution of the test set. This should not be set to Yes for verification tests. If XT_DEBUG_WINDOW_ONLY is also set to Yes, some tests will report UNRESOLVED due to the fact that nothing has been tested. Eg: XT_DEBUG_PIXMAP_ONLY=No - 37 - User Guide for the X Test Suite XT_DEBUG_WINDOW_ONLY When set to Yes, tests which would normally loop over both windows and pixmaps are restricted to loop over just windows. This is useful for speeding up the execution of the test set. This should not be set to Yes for verification tests. If XT_DEBUG_PIXMAP_ONLY is also set to Yes, some tests will report UNRESOLVED due to the fact that nothing has been tested. Eg: XT_DEBUG_WINDOW_ONLY=No XT_DEBUG_DEFAULT_DEPTHS When set to Yes, tests which would normally loop over multiple depths are restricted to test just the first visual returned by XGetVisualInfo and/or the first pixmap depth returned by XListDepths (depending on whether XT_DEBUG_PIXMAP_ONLY or XT_DEBUG_WINDOW_ONLY is also set). This is useful for speeding up the execution of the test set. This should not be set to Yes for verification tests. Note that the first visual returned by XGetVisualInfo may not be the default visual for the screen. Eg: XT_DEBUG_DEFAULT_DEPTHS=No XT_DEBUG_VISUAL_IDS When set to a non-empty string, tests which would normally loop over multiple depths are restricted to test just the visuals ID's listed. Note that visual ID's for visuals on more than one screen may be entered, but those used will depend on whether the test being executed uses visuals on the default screen or alternate screen. The visuals ID's should be entered in decimal, octal or hexadecimal and separated with commas and with no intervening spaces. This should not be set to a non-empty string for verification tests. Eg. XT_DEBUG_VISUAL_IDS=0x22,0x24,0x27 XT_DEBUG_NO_PIXCHECK When set to Yes, tests which would normally perform pixmap verification omit this (all other processing is performed in those tests as normal). Pixmap verification is a scheme which compares the image produced by the X server with a known good image file which is part of the X Test Suite (this is described further in the section entitled "Examining Image Files"). This should not be set to Yes for verification tests. Eg: XT_DEBUG_NO_PIXCHECK=No - 38 - User Guide for the X Test Suite XT_DEBUG_BYTE_SEX When set to NATIVE, REVERSE, MSB or LSB, the X Protocol tests will only be executed with the specified byte sex. When set to BOTH, the X Protocol tests make connections to the X server using both the native and reversed byte sex. Note: The parameter should always be set to NATIVE when the build configuration parameter XP_OPEN_DIS was set to XlibOpaque.c Eg: XT_DEBUG_BYTE_SEX=NATIVE XT_DEBUG_VISUAL_CHECK When set to a non-zero value, the X Protocol tests will pause for the specified time interval (in seconds), to enable a visual check to be performed on the displayed screen contents. Eg: XT_DEBUG_VISUAL_CHECK=5 7.2.8 Configuration Parameters used only during test development This section defines a number of parameters that are used only when generating known good image files. These are not intended to be modified and need not be used when running the test suite. They are only used in the development environment at UniSoft when generating known good image files. XT_FONTDIR The directory in which the xtest fonts are located (before being installed). This must be set such that appending a string gives a valid file name. This is normally set to $TET_ROOT/xtest/fonts/. Eg: XT_FONTDIR=/usr/mit/testsuite/xtest/fonts/ 7.3 Executing tests using the TET The X Test Suite is executed by invoking the execute mode of the Test Case Controller. cd $TET_ROOT/xtest tcc -e [ -s scenario_file ] [ -j journal_file ] [ -x config_file ] [ -y string ] xtest all -e This invokes the TCC in execute mode. -s scenario_file This option executes the test sets in the named - 39 - User Guide for the X Test Suite scenario file. The default is a file named tet_scen in the directory $TET_ROOT/xtest. For more details refer to the section entitled "Executing modified scenarios using the TET". -j journal_file This option sends the test results to the named journal file. The default is a file named journal in a newly created sub-directory of $TET_ROOT/xtest/results. Sub- directories are created with sequential four digit numbers, with the TCC flags (in this case "e") appended. The TCC will exit if the specified journal file already exists, thus the journal file should be renamed or removed before attempting to execute the TCC. -x config_file This is an option to run the test suite using the information in a modified execution configuration file named config_file. The default is tetexec.cfg. -y string This option only executes tests which include the specified string in the scenario file line. This may be used to execute specific sections or individual test sets. xtest This is the name of the test suite. It determines the directory under $TET_ROOT where the test suite is to be found. all This is the scenario name in the default scenario file $TET_ROOT/xtest/tet_scen. For more details refer to the section entitled "Executing modified scenarios using the TET". A journal file will be produced. More information on the contents of the journal file is given in appendix C. Note: If the TCC terminates due to receipt of a signal which cannot be caught, the TCC may leave lock files in the test source directories. Subsequent attempts to restart the TCC may give error messages saying that a lock file was encountered. At this point TCC may suspend the build. It may be necessary to find and remove files or directories named tet_lock before continuing. 7.4 Building, executing and cleaning tests using the TET Each test in the X Test Suite may be built, executed and cleaned before the next test set in the scenario. This mode of use has the advantage that the entire X Test Suite may be executed without necessarily building all the test sets in advance, thus ensuring disc space is conserved throughout. This mode of use has the disadvantage that you will need to - 40 - User Guide for the X Test Suite rebuild a test set before rerunning, which will take considerably longer than if it is built in advance. The X Test Suite is built, executed and cleaned by simultaneously invoking the build, execute and clean modes of the Test Case Controller. cd $TET_ROOT/xtest tcc -bec [ -s scenario_file ] [ -j journal_file ] [ -x config_file ] [ -y string ] xtest all -b This invokes the TCC in build mode. -e This invokes the TCC in execute mode. -c This invokes the TCC in clean mode. The other options are as described in the earlier section entitled "Executing tests using the TET". A journal file will be produced. This contains for each test set in order the results of the build, followed by the test results, followed by the results of the clean. More information on the contents of the journal file is given in appendix C. The default journal file is named journal in a newly created sub-directory of $TET_ROOT/xtest/results. Sub-directories are created with sequential four digit numbers, with the TCC flags (in this case "bec") appended. 7.5 Executing modified scenarios using the TET 7.5.1 Format of the scenario file The TET uses a scenario file to determine which test sets to execute. The file $TET_ROOT/xtest/tet_scen is the default scenario file. The format is basically a scenario name starting in column one, followed by list of test sets to be executed (each starting beyond column one). Only one scenario named "all" is provided in the default scenario file. The names of the test sets are given relative to the directory $TET_ROOT/xtest, and must commence with a leading slash. 7.5.2 Modifying the scenario file The file $TET_ROOT/xtest/tet_scen may be modified by removing lines corresponding to test sets which are not wanted. These will then simply not be executed by the TCC. Alternatively, unwanted lines may be commented out by placing # at the start of the line. If you wish to execute just a subset of the test purposes in - 41 - User Guide for the X Test Suite a test set, refer to the section below entitled "Executing individual test purposes using the TET". It is recommended that the supplied scenario file should be saved if it is modified. 7.5.3 Creating new scenario files A new scenario file may be created in the directory $TET_ROOT/xtest. The TCC will use this scenario file instead of the file $TET_ROOT/xtest/tet_scen if it is passed via the -s option. For example cd $TET_ROOT/xtest tcc -e -s scenario_file [ -j journal_file ] [ -x config_file ] [ -y string ] xtest all 7.6 Executing individual test purposes using the TET Each assertion in the X Test Suite has separate test code which is known as a test purpose. We have arranged that each test purpose is also a separately invocable component, and that the invocable component number is identical to the test purpose number. The expression within the braces at the end of a line within a scenario file is an invocable component list (or IC_list). The default invocable component list all causes the TCC to execute all invocable components in a test set. By altering the invocable component list for a test set, particular invocable components of interest can be executed. The invocable component list consists of one or more elements separated by commas. Each element is either an invocable component number, or a range of invocable component numbers separated by a dash. This is useful for quickly executing a particular test purpose of interest for example: /tset/CH05/stclpmsk/Test{3} This is also useful for executing all test purposes except one known to cause a system error. This may be useful if a particular test purpose causes your X server to exit (at present the TET provides no high level control facilities to conditionally cancel later test sets). For example: /tset/CH05/stclpmsk/Test{1-2,4-6} Note that the placement of windows used by the test suite may differ when an earlier test purpose is not executed. It is intended that test purposes produce the same results regardless of window placement. - 42 - User Guide for the X Test Suite 7.7 Executing tests without using the TET See section 11, entitled "Building, executing and reporting tests without using the TET". 7.8 Executing tests in space-saving format using the TET Before reading this section, read the section entitled "Building tests in space-saving format". When you have built all the sections of the test suite in space-saving format, you can execute all the tests in the test suite using the instructions in this section. A scenario named linkexec is provided in a scenario file named link_scen in the directory $TET_ROOT/xtest. This enables the TCC to execute the space-saving executable files which have been built. Execute the command: cd $TET_ROOT/xtest tcc -e -s link_scen [ -j journal_file ] [ -x config_file ] [ -y string ] xtest linkexec - 43 - User Guide for the X Test Suite 8. Report writer A basic report writer rpt is included with the X Test Suite. It extracts and formats the main information from a TET journal file produced by executing the TCC in execute mode, or build-execute-clean mode. It does not format the TET journal file produced by the TCC in build only or clean only mode. The main features of the TET journal file produced by the TCC in execute or build-execute-clean mode are described in appendix C. Execute the report writer as follows: rpt [ -t ] [ -d ] [ -p ] [ -s ] [ -f file ] With only the -f argument, rpt lists the results of each test purpose for all test sets that appear in the journal file file. The default is the file named journal in the highest numbered subdirectory of the $TET_ROOT/xtest/results directory that has an 'e' suffix. The reason for any test result code which is other than PASS is printed out. This is done by copying the test information messages of type REPORT. For further details, see appendix D. A warning message is printed if a test information message of type REPORT is given in a test purpose which produced a test result code PASS. The results for each test set are followed by a summary of the number of test purposes in the test set which produced each result code type. There is no overall summary list of results for all test sets in the journal file. -t Test information messages of type TRACE in the test purposes specified are printed. For further details, see appendix D. -d Test information messages of type TRACE or DEBUG in the test purposes specified are printed. For further details, see appendix D. -p Output is restricted to omit reporting on test purposes that resolve to PASS, UNSUPPORTED, UNTESTED or NOTINUSE -- thereby reporting only tests showing possible errors. -s The result summaries after the end of each test set are omitted. - 44 - User Guide for the X Test Suite 9. Examining image files 9.1 Generating pixmap error files During the test run, discrepancies may be encountered between the image displayed by the server and the known good image. The known good image may have been obtained from a known good image file supplied with the release, or it may have been determined analytically. Should a discrepancy be encountered, the test purpose will give a result code of FAIL. The failure reason message will name a pixmap error file in which is contained both the known good image and the server image. A debug option has been provided, which skips any verification of the image produced by the server with known good image files. This is done by setting the execution configuration parameter XT_DEBUG_NO_PIXCHECK to Yes. 9.2 Pixmap error file naming scheme Each invocation of the TCC creates a sub-directory in $TET_ROOT/xtest/results. Sub-directories are created with sequential four digit numbers, with the TCC flags ("e" or "bec") appended. The default TET journal is a file named journal created in this directory. Pixmap error files are stored in a directory tree created within the newly created results sub-directory. So, for example, when the line /tset/CH06/drwln is executed in a scenario file, pixmap error files might be produced in a directory named $TET_ROOT/xtest/results/0001bec/tset/CH06/drwln. The creation of a new results directory tree for each execution of the TCC enables results to be obtained in parallel against multiple X servers. Pixmap error files are named Errnnnn.err, where nnnn is a four digit number. This number does not correspond to the number of the test purpose which caused the error. Note - when tests are executed without using the TCC the error files are produced in the current directory. 9.3 Known good image file naming scheme All the required known good image files for the test programs in the X Test Suite (as supplied) have been created in advance. The known good image files for each test program are supplied in the X Test Suite in the test set directory in which the dot-m file is supplied. They are named annn.dat, where nnn is the number of the test purpose - 45 - User Guide for the X Test Suite for which the known good image file was generated. More details of the contents of this release are in appendix A. 9.4 Using blowup to view image files The contents of the two images in a pixmap error file may be compared by using the blowup utility. Also, a known good image file may be viewed directly. The file formats of the error file and the known good image file are the same. The blowup utility detects which file type is being viewed by means of file name. For this reason, do not rename the pixmap error or known good image files. 9.4.1 Blowup command The blowup utility may be used to view one or more pixmap error files or known good image files as follows: blowup [-z zoom_factor] [-f font] [-d display] [-colour] file(s) -z zoom_factor This option sets the magnification factor in all the blowup windows. -f font This option ensures that font is used rather than the default font. The default font is 6x10. -d display This option uses the display named display for the display windows. -colour On a colour display, this option will display different pixel values in different colours corresponding to that server's colour table. No attempt is made to preserve colours between different servers. 9.4.2 Blowup windows Two windows are created. The first is called Comparison, and the second is called Blowup. The Blowup window shows a magnified version of a portion of the Comparison window, which is indicated in the Comparison window by a rectangle. A user interface menu is shown in the Blowup window. The title of the Comparison window will change to "Server Data", then to "Pixval Data" and then back to "Comparison" when the "B/G/Both" option on the menu is used. - 46 - User Guide for the X Test Suite 9.4.3 Selection of a viewing region This may be done in one of three ways: 1. Click in the Comparison window. 2. Click in a square in the Blowup window. This becomes the new centre square in that window. 3. Choose the "next error" option on the menu. The next pixel at which there is a discrepancy will become the new centre square in the Blowup window. 9.4.4 Information displayed The value stored in the centre pixel and its coordinates are shown as the top items in the menu. Under some circumstances, the expected pixel value will be shown to the right of the actual value. 9.4.5 Display of errors When the "B/G/Both" option is set to Both, and the title of the Comparison window is Comparison, errors are displayed in two ways: one for each window. In the Comparison window pixels set to non-zero in the "good image" but set incorrectly in the "server data" are shown as a cross (X). In the blowup window these are shown as a white square with a cross (X) through it. In the Comparison window pixels set to zero in the "good image" but set incorrectly in the "server data" are shown as shaded squares. In the blowup window these are shown as a black square with a white cross through it. The reason that we have proposed the two different methods of displaying errors is as follows. One normally has a higher magnification in the Blowup window and the use of a cross (X) through incorrect pixels is good, and simple to remember at this level of zoom. In the Comparison window this style of display does not work well at the lower magnification levels; all the crosses merge to a blur so it is hard to see what type of error is being displayed. 9.4.6 Commands (via menu in the Blowup window) All of the commands are invoked by clicking the left mouse button when the corresponding menu item is highlighted (inverted). The available commands are, from top to bottom: B/G/Both Show Bad (Server Data), Good (Pixval Data) or Both - 47 - User Guide for the X Test Suite (Comparison). Clicking in this advances around the cycle Bad ----> Good -> Both ------<------<----/ The Comparison window's name changes to reflect the current state. If a known good image file is being displayed then only the Good option is available. A pixmap error file is required for this command to be useful. color/mono Use colour/monochrome in the Blowup window. next error Advance centre pixel point to be next pixel at which there is a discrepancy. sub-zoom + Zoom in (make bigger by zoomfactor) on the Blowup window sub-zoom - Zoom out (make smaller by zoomfactor) on the Blowup window quit Quit from the blowup utility. big-zoom + Zoom in (make bigger by zoomfactor) on the Comparison window big-zoom - Zoom out (make smaller by zoomfactor) on the Comparison window next View next file in the list. The Blowup window will be removed and a new one created for each file. The size, and zoom factor, of the Comparison window will be preserved across files. - 48 - User Guide for the X Test Suite 10. Cleaning the tests 10.1 Cleaning tests using the TET The entire X Test Suite can be cleaned by using the clean mode of the TCC. In this mode, the clean configuration parameters in the file $TET_ROOT/xtest/tetclean.cfg are used to clean each test set in the X Test Suite separately. Previously built test set executables and object files are removed. cd $TET_ROOT/xtest tcc -c [ -s scenario_file ] [ -j journal_file ] [ -y string ] xtest all -c This invokes the TCC in clean mode. -s scenario_file This option cleans the test sets in the named scenario file. The default is a file named tet_scen in the directory $TET_ROOT/xtest. For more details refer to the section entitled "Cleaning modified scenarios using the TET". -j journal_file This option sends the output of the clean to the named journal file. The default is a file named journal in a newly created sub-directory of $TET_ROOT/xtest/results. Sub-directories are created with sequential four digit numbers, with the TCC flags (in this case "c") appended. The TCC will exit if the specified journal file already exists, thus the journal file should be renamed or removed before attempting to execute the TCC. -y string This option only cleans tests which include the specified string in the scenario file line. This may be used to clean specific sections or individual test sets. xtest This is the name of the test suite. It determines the directory under $TET_ROOT where the test suite is to be found. all This is the scenario name in the default scenario file $TET_ROOT/xtest/tet_scen. For more details refer to the section entitled "Cleaning modified scenarios using the TET". This will execute the TET clean tool in the TET configuration variable TET_CLEAN_TOOL (which is normally pclean), in each test set directory of the X Test Suite. - 49 - User Guide for the X Test Suite The journal file should be examined to verify that the clean process succeeded. The report writer rpt cannot interpret the contents of a journal file produced during the clean process. Note: If the TCC terminates due to receipt of a signal which cannot be caught, the TCC may leave lock files in the test source directories. Subsequent attempts to restart the TCC may give error messages saying that a lock file was encountered. At this point TCC may suspend the build. It may be necessary to find and remove files or directories named tet_lock before continuing. 10.2 Cleaning modified scenarios using the TET 10.2.1 Format of the scenario file Refer to the earlier section "Building modified scenarios using the TET". 10.2.2 Modifying the scenario file Refer to the earlier section "Building modified scenarios using the TET". 10.2.3 Creating new scenario files A new scenario file may be created in the directory $TET_ROOT/xtest. The TCC will use this scenario file instead of the file $TET_ROOT/xtest/tet_scen if it is passed via the -s option. For example cd $TET_ROOT/xtest tcc -c -s scenario_file [ -j journal_file ] [ -y string ] xtest all 10.3 Cleaning tests without using the TET See section 11, entitled "Building, executing and reporting tests without using the TET". 10.4 Cleaning tests built in space-saving format It is possible to clean the tests in the X Test Suite which were previously built in space-saving format. 10.4.1 Cleaning tests in space-saving format using the TET A scenario named linkbuild is provided in a scenario file named link_scen in the directory $TET_ROOT/xtest. This enables the TCC to clean the space-saving executable files and remove all the required links for each test set in each section of the X Test Suite. The -y option allows a particular space-saving executable and its links for a single section to be removed. Execute the command: - 50 - User Guide for the X Test Suite cd $TET_ROOT/xtest tcc -c -s link_scen [ -j journal_file ] [ -y string ] xtest linkbuild This command will execute the TET clean tool in the TET configuration variable TET_CLEAN_TOOL (which is normally pclean), in the top level directory of each section of the test suite. 10.4.2 Cleaning tests in space-saving format without using the TET This section describes how to clean the space-saving executable files for a particular section of the X Test Suite directly without using the TET. This can be simply done by calling pclean in the required directory. For example, to clean all the space-saving executable files for section 5 of the X Test Suite, execute the command: cd $TET_ROOT/xtest/tset/CH05 pclean - 51 - User Guide for the X Test Suite 11. Building, executing and reporting tests without using the TET 11.1 Building tests An individual test set can be rebuilt without the need to use the build mode of the TCC. This is done by executing pmake directly, rather than as a TET build tool. This is a useful facility for building a single test set after a previous build has failed. The build configuration parameters used by pmake are obtained from a file named $TET_BUILDCONFIG, or, if TET_BUILDCONFIG is not set in your environment, from the file named $TET_ROOT/xtest/tetbuild.cfg. The pmake command should be executed in the directory containing the source code for the test set which is to be rebuilt. For more details of the names of the directories containing the source code for the test sets, refer to appendix A. For example cd $TET_ROOT/xtest/tset/CH05/stclpmsk pmake No journal file is created when pmake is executed directly. The test set can also be rebuilt using the command pmake Test If there is a macro version of the Xlib function, this may be rebuilt using the command pmake MTest 11.2 Executing tests An individual test set can be executed without the need to use the execute mode of the TCC. This is done by executing a shell script pt. This is a useful facility for executing a single test set repeatedly when investigating a particular test result. The execution configuration parameters used by pt are obtained from a file named $TET_CONFIG, or, if TET_CONFIG is not set in your environment, from the file named $TET_ROOT/xtest/tetexec.cfg. The pt command is a shell script, which attempts to execute - 52 - User Guide for the X Test Suite the binary file named Test in the current directory. If the file Test is not found, the pt command attempts to execute the space-saving executable file built in that directory. The pt command should be executed in the directory containing the test set which has been built. Unless you have manually installed the test set elsewhere, this will be the directory containing the source code for the test set. For more details of the names of the directories containing the source code for the test sets, refer to appendix A. For example cd $TET_ROOT/xtest/tset/CH05/stclpmsk pt A TET results file is created when pt is executed directly. This is a file named tet_xres located in the directory in which the test was executed. There are a number of options which may be passed to pt which alter the manner in which the test set is executed. Execute pt as follows: pt [ -v XT_VARIABLE_NAME ] [ -d display ] [-i IC_list ] [ -p ] [ -w ] [ -P ] [ -D ] [ -x debug_level ] [ -g ] [ -m ] -v XT_VARIABLE_NAME=Value Modifies the value of the execution configuration parameter named XT_VARIABLE_NAME, assigning it a value of Value. -d display Sets the display string to be used for the test. The default value is taken from the environment variable DISPLAY, or, if this is not set, from the execution configuration parameter XT_DISPLAY. -i IC_list The invocable components executed will be those specified in IC_list. Each assertion in the X Test Suite has separate test code, which is known as a test purpose. We have arranged that each test purpose is also a separately invocable component, and that the invocable component number is identical to the test purpose number. The invocable component list consists of one or more elements separated by commas. Each element is either an invocable component number or a range of invocable component numbers separated by a dash. - 53 - User Guide for the X Test Suite This is useful for quickly executing a particular test purpose of interest for example: pt -i 37 This is also useful for executing all test purposes except one known to cause a system error. For example: pt -i 1-36,38-57 Note that the placement of windows used by the test suite may differ when an earlier test purpose is not executed. It is intended that test purposes produce the same results regardless of window placement. -p This option is equivalent to setting the execution configuration parameter XT_DEBUG_PIXMAP_ONLY to Yes. -w This option is equivalent to setting the execution configuration parameter XT_DEBUG_WINDOW_ONLY to Yes. -P This option is equivalent to setting the execution configuration parameter XT_DEBUG_PAUSE_AFTER to Yes. -D This option is equivalent to setting the execution configuration parameter XT_DEBUG_DEFAULT_DEPTHS to Yes. -x debug_level This option is equivalent to setting the execution configuration parameter XT_DEBUG to debug_level. -g The binary file pvgen will be executed instead of the binary file Test. This option should not be used, since binary files named pvgen are only used in the development environment at UniSoft when generating known good image files. -m The binary file MTest will be executed instead of the binary file Test. Files named MTest contain tests for the macro version of an Xlib function. Note that pt creates a temporary file CONFIG in the current directory containing the configuration parameters, so write permission is required to this file (or if no file is there, to the current directory). Note also that the binary file Test creates a temporary file .tmpresfd in the current directory containing the configuration parameters, so write permission is required to this file. - 54 - User Guide for the X Test Suite 11.3 Reporting tests The TET results file produced for an individual test set can be formatted using the basic report writer rpt, which is described in more detail in the section entitled "Report writer". The argument -f tet_xres formats the contents of the tet_xres file. For convenience, a separate report writer prp is provided, which is identical to rpt, except that the default file used is tet_xres in the current directory. This is a useful facility for quickly formatting the results from the execution of a test set, and looking at the summary of the result codes for each test purpose executed. The prp command should be executed in the directory containing the TET results file named tet_xres. Unless you have manually installed and executed the test set elsewhere, this will be the directory containing the source code for the test set. For more details of the names of the directories containing the source code for the test sets, refer to appendix A. For example cd $TET_ROOT/xtest/tset/CH05/stclpmsk prp 11.4 Cleaning tests An individual test set can be cleaned without the need to use the clean mode of the TCC. This is done by executing pclean directly, rather than as a TET clean tool. The clean configuration parameters used by pclean are obtained from a file named $TET_CLEANCONFIG, or, if TET_CLEANCONFIG is not set in your environment, from the file named $TET_ROOT/xtest/tetclean.cfg. The pclean command should be executed in the directory containing the test set which was built. For more details of the names of the directories containing the source code for the test sets, refer to appendix A. For example cd $TET_ROOT/xtest/tset/CH05/stclpmsk pclean No journal file is created when pclean is executed directly. - 55 - User Guide for the X Test Suite 12. Appendix A - Contents of X Version 11 Release 6.1 This section describes the contents of the directories in the TET_ROOT directory which are supplied in this release of the X Test Suite. The revised X Test Suite has been developed from the T7 X Test Suite. This section therefore also explains how the arrangement of the revised X Test Suite compares with the T7 X Test Suite. 12.1 tet This contains the source files and include files needed to build the Test Environment Toolkit (TET). The contents of the subdirectories are as follows: 12.1.1 tet/src This contains the source files for the TET. 12.1.2 tet/inc This contains the include files for the TET. 12.1.3 tet/lib This contains the libraries and object files when the TET has been built. 12.1.4 tet/bin This should be empty since the TET utilities will be copied into $TET_ROOT/xtest/bin, rather than this directory, when using the modified Makefiles supplied with the TET. 12.1.5 tet/doc This contains the release notes and man pages for the TET. 12.1.6 tet/demo This contains a demonstration program for the TET. 12.2 xtest This contains the tests included in the revised X test suite which are stored as a complete TET test suite. This includes all necessary configuration files and scenario files to enable you to use the TET following the instructions in the documentation. 12.3 xtest/bin This contains commands you will need to install, configure, build and execute the X Test Suite. After installation, this directory contains shell script commands. After configuration and building the X Test Suite, this directory also contains executable programs built for your system. - 56 - User Guide for the X Test Suite 12.4 xtest/doc This contains the documentation. It contains this user guide, the programmers guide and the release notes. These are supplied in troff(1) format requiring the mm macro package, and also in PostScript format. It also contains a template for error reports and a description of how to submit them. It also contains a file paper.mm, which is a copy of the file Xproto_verf/doc/paper.ms originally supplied in the T7 X Test Suite, converted to use the mm macro package. This file contains a paper entitled "An Approach to Testing X Window System Servers at a Protocol Level". This is a technical paper, which defines in outline terms the areas of the X Window System server which should be tested at the X Protocol level rather than the Xlib level. The approach recommended in this paper, and adopted in the design of the T7 X Test Suite, has been maintained in the revised X Test Suite. The paper explains the choice of test cases and division of tests between the X Protocol tests and Xlib tests. Before the revision of the X Test Suite, UniSoft recommended that this paper should be left "as is". As a result, some sections of this paper are out of date in that they refer to development schedules for a previous software development project, which have now been superseded with the production of the revised X Test Suite. 12.5 xtest/fonts This contains test fonts which should be installed using the instructions in this user guide. It also contains a software library describing the fonts which is used by the tests for text drawing. 12.6 xtest/include This contains include files for the software in the xtest/src and xtest/tset directories. 12.7 xtest/lib This contains libraries and other common software which are used by the tests in the xtest/tset directory. The libraries are built using the instructions in this user guide. 12.8 xtest/results This is an empty directory which is used by the TET to store journal files produced when executing the X Test Suite. 12.9 xtest/src - 57 - User Guide for the X Test Suite This contains the source for the libraries and utilities. These are built using the instructions in this user guide. 12.10 xtest/tset This contains the source for the tests for sections 2 to 10 of the X11R4 Xlib specifications, in directories CH02 to CH10. These are built using the instructions in this user guide. In the rest of this document, these are refered to as the "Xlib tests". Each of the directories CH02 to CH10 contains further subdirectories which are known as test set directories. Each of these contain all the test code for a single Xlib function. The name of the directory is derived from the name of the Xlib function by a scheme which is described in appendix B. So, for example for XSetClipMask we have the following: tset/CH05/stclpmsk tset/CH05/stclpmsk/stclpmsk.m The file stclpmsk.m is the source file, which is also known as a dot-m file. The format of the dot-m file is described further in the "Programmers Guide". The Xlib tests are designed to accomplish the following: 1. Test the ability of the Xlib function to behave as specified in the X11R4 Xlib specifications, in situations where no X Protocol error should be generated. This is tested by a series of separate tests known, as "test purposes", each of which is designed to test a single statement in the Xlib specifications. The statement which is tested is contained in an "assertion", which is also contained in the dot-m file, and precedes the test code for that test purpose. 2. Test the ability of the Xlib function to produce the expected X Protocol errors in specified situations. This is tested in further test purposes, each preceded in the dot-m file by an assertion describing the situation which should produce the error. 12.11 xtest/tset/XPROTO This contains the source for the touch tests for the X Protocol (version 11). These are built using the instructions in this user guide. In the rest of this document, these are refered to as the "X Protocol tests". These tests were in a separate test suite in the T7 X Test Suite, which was located in a directory Xproto_verf. This included separate documentation, drivers, parameter files, - 58 - User Guide for the X Test Suite include files and libraries. In the revised X Test Suite, the directory XPROTO only contains the source of the tests - the other items are integrated within the X Test Suite as described in this user guide. The directory XPROTO contains further subdirectories which are known as test set directories. The structure of test set directories is exactly as for the Xlib tests, described in the previous section. The X Protocol tests tests are designed to accomplish the following: 1. Test the ability of the X Window System server to accept all legal message types and respond appropriately. 2. Ensure that the server capabilities which Xlib testing depends on work in the simplest cases. 3. Test that the X server adheres to the canonical byte stream of the X Protocol, independent of the host byte sex or compiler structure packing. For further details of the choice of test cases and division of tests between the X Protocol tests and Xlib tests, refer to the document entitled "An Approach to Testing X Window System Servers at a Protocol Level" contained in file xtest/doc/paper.mm. - 59 - User Guide for the X Test Suite 13. Appendix B - File naming scheme A file naming scheme has been devised which is for naming the directories containing dot-m files and the dot-m files themselves. The file naming scheme converts from an X Window System name to an abbreviated name. This is done as follows: -- Remove leading X. -- Replace: Background -> Bg Subwindow -> Sbwn String -> Str Window -> Wdw -- Remove all lowercase vowels aeiou. -- Truncate to 10 chars. We have already checked that truncation to ten characters still gives uniqueness. -- If the string before truncation ended in "16" then force truncated string to end in "16". -- convert to lowercase. -- add ".m" suffix to get name of source file containing C code, assertions and strategies. - 60 - User Guide for the X Test Suite 14. Appendix C - Format of the TET journal file This appendix describes the manner in which the X Test Suite uses some of the TET journal file facilities. The format of the TET journal file is not fully described here - only the main features used by the X Test Suite are described. In a future release of the X Test Suite the format is expected to be described fully in the TET documentation. Journal files are produced by the TCC during the build and execute stages. The journal file produced during the execute stage contains two basic sections: 1. Details of the configuration parameters and environment in which the tests were executed. This may also be preceded by build configuration parameters and/or followed by clean configuration parameters, if the TCC was invoked in build-execute-clean mode. 2. Details of the test results. This includes the test result codes and test information messages output by the test suite. Each line in the second section of a journal file is made up from three components separated by a vertical bar: 1. Message type. There is a unique numeric code for each message type which is always the first field on a line. 2. Message parameters. These contain serial number and similar information. 3. Message area. The format of this area is specific to the Message Type. An example of the second section is as follows: 10|53 /tset/CH02/vndrrls/vndrrls 13:41:12|TC Start, scenario ref 85-1, ICs {all} 15|53 1.9 1|TCM Start 520|53 0 7457 1 1|TRACE:NAME: XVendorRelease 400|53 1 1 13:41:14|IC Start 200|53 1 13:41:14|TP Start 520|53 1 7457 1 1|REPORT:XVendorRelease() returned 5000 instead of 1. 220|53 1 1 13:41:14|FAIL 410|53 1 1 13:41:14|IC End 80|53 0 13:41:15|TC End This consists of a block of information for each test set executed which contains the following lines: 1. Message Type 10 - Test Case Start (TC Start) A single message indicating that the Test Case Controller (the part of the TET which executes test sets) is about to execute a test set. This also - 61 - User Guide for the X Test Suite indicates the start of the results for this test set (which in TET terminology is known as a test case). This line indicates the name of the test set obtained from the scenario file. 2. Message Type 15 - Test Case Manager Start (TCM Start) A single message indicating that the Test Case Manager (the part of the TET which calls the test purposes) has started executing. 3. Each assertion in the X Test Suite has separate test code, which is known as a test purpose. We have arranged that each test purpose is also a separately invocable component, and that the invocable component number is identical to the test purpose number. For each test purpose these lines follow: 1. Message Type 400 - Invocable Component Start (IC Start) The second field of the Message Parameters gives the IC number. 2. Message Type 200 - Test Purpose Start (TP Start) The second field of the Message Parameters gives the number of the test purpose within the test set. 3. Message Type 520 - Test Case Information The second field of the Message Parameters gives the number of the test purpose within the test set. The Message Area contains a text message output by the X Test Suite - the possible types of message are described further in appendix D "Interpreting test results" in the section entitled "Test information messages". 4. Message Type 220 - Test Purpose Result The second field of the Message Parameters gives the number of the test purpose within the test set. The Message Area contains the test result code - the possible test result codes are described further in appendix D "Interpreting test results" in the section entitled "Test result codes". 5. Message Type 410 - Invocable Component End (IC End) The second field of the Message Parameters gives the IC number. 4. Message Type 80 - Test Case End (TC End) A single message indicating the end of the results for this test set (which in TET terminology is known as a test case). - 62 - User Guide for the X Test Suite 15. Appendix D - Interpreting test results This section includes information describing the significance of the test result codes and the accompanying test information messages that may appear in a TET journal file. 15.1 Categorisation of assertions The test result codes which are output for each test purpose are dependent on the category of the assertion. The model for the categorisation of assertions which is used in the X Test Suite is described in POSIX.3. There are four categories of assertions described by POSIX.3 which are designated A, B, C and D. If the assertion tests a conditional feature, it is categorised as type C or D, otherwise it is categorised as type A or B. If the assertion is classified as an "extended assertion", it is categorised as type B or D. Otherwise it is categorised as type A or C and is known as a "base assertion". Tests are always required for base assertions. Tests are not required for extended assertions, but should be provided if possible. There are a number of "extended assertions" for which tests have been written in the X Test Suite. Extended assertions are used to describe features that may be difficult to test conclusively. +--------------------+----------------+--------------------+ | | Base Assertion | Extended Assertion | +--------------------+----------------+--------------------+ | Required Feature | A | B | +--------------------+----------------+--------------------+ |Conditional Feature | C | D | +--------------------+----------------+--------------------+ 15.2 Test result codes The following test result codes may be found within the TET journal file. These will be found in Test Purpose Result lines with Message Type 220 (described in appendix C). The reason for any result codes NORESULT, UNRESOLVED and UNINITIATED should be determined and resolved. PASS The implementation passed the test for the corresponding assertion. FAIL The implementation failed the test for the corresponding assertion. - 63 - User Guide for the X Test Suite UNRESOLVED The test for the corresponding assertion commenced, but was unable to establish the required conditions to complete all required stages of the test. The reasons for this should be investigated and if necessary the test rerun. In some tests, reliance is made on the successful behaviour of another area of the X Window System. Where this reliance is made, and that area of the X Window System does not behave as expected, a result code UNRESOLVED may occur. The test information messages should indicate the area of the underlying problem. It may be necessary to look at the test results for that area first and investigate and resolve the underlying problem before re-running the UNRESOLVED tests. Tests which give a result code of UNRESOLVED and the message "Path check error" normally contain programming errors. The test reached the point at which a PASS result would be assigned, but the number of check points passed in executing the test code differs from the expected number. Tests which give a result code of UNRESOLVED and the message "No CHECK marks encountered" may be due to programming errors. The test reached the point at which a PASS result would be assigned, but no check points had been passed. This can also occur when you execute the tests using debug options. For example, the message occurs when you execute tests which normally loop over windows and pixmaps and set XT_DEBUG_PIXMAP_ONLY=Yes or XT_DEBUG_WINDOW_ONLY=Yes or XT_DEBUG_DEFAULT_DEPTHS=Yes or XT_DEBUG_VISUAL_IDS=Yes. NOTINUSE Although there is an assertion within the test set, there is no specific test provided for the assertion. This might be because the assertion is tested adequately as part of the test for another assertion, or because the assertion has been automatically included into a test set in which it is not applicable. In either case, tests which report the result code NOTINUSE do not need to be investigated further. UNSUPPORTED This result code may only be used for assertions in category C or D (conditional - 64 - User Guide for the X Test Suite assertions). The implementation does not support some optional feature of the X Window System, which is needed to test the corresponding assertion. In this case, the assertion will normally make clear what optional feature is required, and there will be an accompanying test information message describing the feature which was found to be unsupported. UNTESTED This result code may only be used for assertions in category B or D (extended assertions). The implementation could not be conclusively tested to determine the truth of the corresponding assertion. Note that this does not mean that no testing was performed in the X Test Suite. There are a number of "extended assertions" for which we have provided tests where possible (to test some likely problem areas, for example). UNINITIATED The test for the corresponding assertion could not commence due to a problem in an earlier test purpose or an initialisation routine. Tests which produce this result code should be resolved as for those reporting UNRESOLVED. NORESULT Each test purpose should output a test result code before completing. This special result code will be inserted by the TET in the journal file, if the test purpose did not output a test result code. This indicates a major fault during the execution of the test set which should be investigated. WARNING The implementation passed the test for the corresponding assertion. Whilst the behaviour of the implementation was found to be acceptable, it behaves in a manner which is not recommended in the X11 specification on which the assertion is based. FIP The contents of the journal file must be examined by hand to determine whether the implementation passed the test for the corresponding assertion. This is used for testing functions which produce output whose correctness cannot be easily determined automatically by the test suite. - 65 - User Guide for the X Test Suite 15.3 Test information messages There are four types of test information messages which are output by the X Test Suite. Each one results in a TET journal file line with Message Type "Test Case Information" (520), and with the Message Area beginning with one of the following keywords: REPORT This keyword is used to report the reason for any test result codes whch are other than PASS. A warning message is printed by the report writer rpt, if a test information message of type REPORT is given in a test purpose which produced a test result code PASS. CHECK This keyword is used to record the passing of a particular checkpoint in the test suite code. These messages contain the checkpoint number within the test purpose, and the line number within the source code. These messages are not output to the journal file if the execution configuration parameter XT_OPTION_NO_CHECK is set to Yes. This option can reduce the size of the journal file considerably. TRACE This keyword is used for any messages describing the state of the test being executed which are not failure messages. When running the X Protocol tests, messages with this keyword are output to the journal file, to describe briefly the interaction between the X server and the test program. These messages are not output to the journal file, if the execution configuration parameter XT_OPTION_NO_TRACE is set to Yes. This option can reduce the size of the journal file considerably. DEBUG This keyword is used for debug messages inserted during the development of the X Test Suite. When running the X Protocol tests, messages with this keyword are output to the journal file, when the debug level is greater than zero, to describe in detail the interaction between the X server and the test program. This includes the contents of requests, replies and events. This is output if the value of the execution configuration parameter XT_DEBUG is greater - 66 - User Guide for the X Test Suite than or equal to the level of the debug message. XT_DEBUG may be set from 0 to 3. - 67 - User Guide for the X Test Suite 16. Appendix E - Outline of Test Environment Toolkit The "Test Environment Toolkit" is a software tool developed by X/Open8, UNIX International (UI)9, and the Open Software Foundation (OSF)10. The project which produced this software and the associated interface specifications was also known as Project Phoenix. The TET consists of a user interface program known as the Test Case Controller (TCC). This enables test software to be built and executed. The TCC uses configuration files to specify the environment for both the build and execute operations. The TCC also uses a scenario file to control which tests to build or execute. The TCC produces a journal file which is intended to be formated by a test suite specific report writing program. The TET also includes an Application Programming Interface (API) Part of the API is the Test Case Manager (TCM). This includes a main() function which calls user supplied test functions. The API also includes a library of functions to manage the test functions and perform output operations to the journal file in a structured fashion. Since the developers of the TET have indicated a commitment to develop software test suites that execute within this environment, the TET can be seen as an emerging de facto standard environment for test suite execution. During stage one of the X Test Suite development project we identified that the TET provides features which are required by the revised test suite. For this reason we have developed the revised test suite within the TET environment, and supplied a copy of the TET with the revised test suite. Release 1.9 of the TET was issued by the developers during March 1992, and is included in this release of the X Test Suite. The software is complete in that the functionality is stable and the implementation agrees with the TET specification. Documentation including release notes and man pages for the TET utilities, are included in this release of the X Test Suite. However, this release does not contain a Programmers Guide or Users Guide. These are under development by UNIX International, but are not complete at the time of this release, and so are not part of the current ____________________ 8. X/Open is a trademark of the X/Open Company, Ltd. in the U.K. and other countries. 9. UI is a trademark of UNIX International. 10.Open Software Foundation, OSF and OSF/1 are trademarks of the Open Software Foundation, Inc. - 68 - User Guide for the X Test Suite version of the TET. - 69 - User Guide for the X Test Suite 17. Appendix F - Glossary assertion A statement of functionality for an element of the X Window System, phrased such that it will be true for a system conforming to the X Window System specifications. An example would be a test description phrased according to the requirements of POSIX.3. assertion test Synonymous with test purpose. base assertion An assertion for which a test suite must provide an assertion test. Every assertion that is not an extended assertion is a base assertion. element A particular X Window System interface such as an Xlib function, header file or X11 Procotol. extended assertion An assertion for which a test suite is not required to provide an assertion test. An assertion test should still be provided if possible. Reasons why a test suite is not required to provide a test are given in appendix A of the Programmers Guide. POSIX.1 Part one of the IEEE POSIX 1003 standards, the document entitled System Application Program Interface (API) [C language]. Also known as P1003.1. POSIX.3 Part three of the IEEE POSIX 1003 standards, the document entitled Test Methods for Measuring Conformance to POSIX. Also known as P1003.3. Project Phoenix Synonymous with TET. SVID System V Interface Definition. TCC The Test Case Controller. This is part of the TET. This is a user interface program which enables the test software to be executed. See appendix E for more details. ____________________ Obtainable from Publication Sales, IEEE Service Center, P.O. Box 1331, 445 Hoes Lane, Piscataway, NJ 08854-1331, (201) 981-0060 - 70 - User Guide for the X Test Suite TCM The Test Case Manager. This is part of the TET. This is an object file containing a main() function which calls user supplied test functions. See appendix E for more details. TET The "Test Environment Toolkit". This is a software tool which provides a framework within which tests may be developed and executed. More information is given in appendix E. test case Synonymous with test set. test description The description of a particular test to be performed on an element. This is presented in functional terms and describes precisely what aspect of the X Window System is to be tested for that element. An assertion is an example of a test description, but the reverse is not the case. test program Synonymous with test set. test purpose The software which tests the conformance of an implementation of the X Window System to an assertion. test set The software containing all the test purposes for an element. test strategy A description of the design and method used to implement a test purpose. This should say how a test purpose is implemented rather than what feature is being tested. XPG The X/Open Portability Guide. Xlib tests These are the tests for sections 2 to 10 of the X11R4 Xlib specifications. They are stored in subdirectories of the directories CH02 to CH10 (which are to be found in the directory $TET_ROOT/xtest/tset). X Protocol tests These are the touch tests for the X Protocol (version 11). They are stored in subdirectories of the directory XPROTO (which is to be found in the directory $TET_ROOT/xtest/tset). CONTENTS 1. Introduction ........................................ 1 2. Preparation ......................................... 2 2.1 Utilities .................................... 2 2.2 Checking your version of the X Window System . 3 2.3 Installing the X Test Suite .................. 3 3. Configuring the X Test Suite ........................ 5 3.1 The TET build tool ........................... 5 3.2 Relationship between TET build scheme and Imake ........................................ 5 3.3 Build configuration parameters ............... 6 3.4 Clean configuration parameters .............. 15 3.5 System dependent source files ............... 15 4. Building the TET ................................... 17 4.1 The portability library ..................... 17 4.2 Building libraries and utilities ............ 19 4.3 The Test Case Controller (TCC) .............. 19 4.4 The API library ............................. 20 5. Building the X test suite libraries and utilities .. 20 5.1 The X test suite library .................... 20 5.2 The X Protocol library ...................... 20 5.3 The X test fonts library .................... 21 5.4 Compiling and installing the test fonts ..... 21 5.5 Building the mc utility ..................... 22 5.6 Building the blowup utility ................. 23 6. Building the tests ................................. 24 6.1 Building tests using the TET ................ 24 6.2 Building, executing and cleaning tests using the TET ..................................... 25 6.3 Building modified scenarios using the TET ... 25 6.4 Building tests without using the TET ........ 26 6.5 Building tests in space-saving format ....... 26 7. Executing the X Test Suite ......................... 28 7.1 Setting up your X server .................... 28 7.2 Execute configuration parameters ............ 29 7.3 Executing tests using the TET ............... 39 7.4 Building, executing and cleaning tests using the TET ..................................... 40 7.5 Executing modified scenarios using the TET .. 41 7.6 Executing individual test purposes using the TET ......................................... 42 7.7 Executing tests without using the TET ....... 43 7.8 Executing tests in space-saving format using the TET ..................................... 43 8. Report writer ...................................... 44 i 9. Examining image files .............................. 45 9.1 Generating pixmap error files ............... 45 9.2 Pixmap error file naming scheme ............. 45 9.3 Known good image file naming scheme ......... 45 9.4 Using blowup to view image files ............ 46 10. Cleaning the tests ................................. 49 10.1 Cleaning tests using the TET ................ 49 10.2 Cleaning modified scenarios using the TET ... 50 10.3 Cleaning tests without using the TET ........ 50 10.4 Cleaning tests built in space-saving format . 50 11. Building, executing and reporting tests without using the TET ...................................... 52 11.1 Building tests .............................. 52 11.2 Executing tests ............................. 52 11.3 Reporting tests ............................. 55 11.4 Cleaning tests .............................. 55 12. Appendix A - Contents of X Version 11 Release 6.1 .. 56 12.1 tet ......................................... 56 12.2 xtest ....................................... 56 12.3 xtest/bin ................................... 56 12.4 xtest/doc ................................... 57 12.5 xtest/fonts ................................. 57 12.6 xtest/include ............................... 57 12.7 xtest/lib ................................... 57 12.8 xtest/results ............................... 57 12.9 xtest/src ................................... 57 12.10 xtest/tset .................................. 58 12.11 xtest/tset/XPROTO ........................... 58 13. Appendix B - File naming scheme .................... 60 14. Appendix C - Format of the TET journal file ........ 61 15. Appendix D - Interpreting test results ............. 63 15.1 Categorisation of assertions ................ 63 15.2 Test result codes ........................... 63 15.3 Test information messages ................... 66 16. Appendix E - Outline of Test Environment Toolkit ... 68 17. Appendix F - Glossary .............................. 70 ii