FWKCS(TM) Contents_Signature System, Ver. 1.12, 1991 Jul 18. (C)Copyright Frederick W. Kantor 1988-1991. All rights reserved. Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS ........................ C A V E A T O P E R A T O R ........................ To the reader: ~~~~~~~~~~~~~~ This software is based on ideas which become easier to understand as you get hands_on experience. That is one of the advantages of using a computer. Take breaks in your reading and try it out, so that the words take on direct meaning. And for safety and peace of mind, be sure that you have backup copies of files. This file, FWKCS112.REF, contains descriptions of options and auxiliary functions provided in FWKCS.COM Version 1.12. Many powerful features are brought out for convenient use via special programs; various of those special programs are presented here as examples of the specific features they use. GO.BAT is provided for installing the FWKCS(TM) Contents_ Signature System. On_line help menus are provided, to assist you with information on special .BAT and .COM programs, and with literature: CSM.BAT, in the \CS Contents_Signature subdirectory (the working contents_signature lists and index files are kept there, together with special programs for manipulating the cs lists and finding duplicate or redundant files); and CSAM.BAT, in the \CSA Contents_Signature Activity place (a structure comprising one main subdirectory with 11 subdirectories under it, with .BAT programs, .COM programs, literature, upload_log, and other related material). Companion files in this zipfile (FWKCS112.ZIP) contain instructions for installing FWKCS (SETUP.112), overall discussion (FWKCS112.DOC), examples (EXAMPLES.112), step by step recipes (RECIPES.112), utilities (UTILS.112), registration information (REGISTER.DOC), and other material. FWKCS.COM options: ~~~~~~~~~~~~~~~~~~ The FWKCS command_line: ~~~~~~~~~~~~~~~~~~~~~~~ Because square brackets "[]" have literal meaning in the command line, optional material discussed herein is enclosed in parentheses "()". Some options have variable_length fields. This could be a variable_length numeric data_field, such as the decimal_value N's in the tNNNN.NN option, which specifies how long to keep retrying when a file is not available. Or, it could be a variable_length data_field which could contain non_numeric characters, such as d:\path\filename. If the last data_field for an option is a variable_length field which could contain a non_numeric character, then the data_field is ended with a semi_colon, ';', to separate it from the next option. All (sub)options (with their respective data_field(s), if any) are placed directly in sequence with no spaces between them, forming a single string. If you have run one of the FWKCS installation programs (GO, 2_SETUP, S_SETUP, CSAMACS), then you can type FWKCS /d , and see examples of command line structure in macros [b], [c], [d], [e], and [f] (macros are discussed further, below). Upper and lower case letters are OK on the FWKCS command line. You can mix upper and lower case for convenience in keeping track of options with variable_length data_field entries, etc. Format 1,2,3 grammar: /(r)(t)other_options. the r for redirect (or as part of the "revert" command, discussed below) and t for timespan options come in a definite order near the "/". This lets the letters be used with other meanings as suboptions for other functions. Format 1: FWKCS (CSLIST.NDX CSLIST.SRT) /option ((<)in ((>)out)) where CSLIST.NDX is index for CSLIST.SRT sorted contents_signature_list. FWKCS can open input and output files which come after the /option(s). Format 2: FWKCS file1 file2 (file3) /m This is used in merging files. The m merge option is discussed below. Format 3: FWKCS (files)([b])([+file1])[c](options)([def])([+file2]) (files) ([-file3]) where "[]" are literal characters; [b][c][d][e][f] are macros (see /d); [+file] includes file's contents (/d macros OK) in command line; [-file] prevents execution, writes (expanded) command line to file. There are 5 macros, each of which is a contiguous character sequence up to 73 bytes long. This gives a total of 365 bytes of rapidly accessible, internally stored, user_settable information for use in making a command line. This is more than DOS normally handles, and helps support the use of (many) options for the f and g functions and the /a auxiliary functions. Note: the total command line, with all external files included, and with all macros in the command line and in the external files replaced with their full strings, should not exceed 1024 bytes. To see these macros, type FWKCS /d . The command line can include one or several files, which are read into the internal command interpretation region. They are included in the command line by using square brackets "[]" and the "+" sign. For example, to load a macro from a file named MACROB into FWKCS macro [b], type FWKCS [+macrob] /db On the left edge of the /d (defaults) screen is a hexadecimal digit, 1...9, A...F, to label each of the externally changeable lines. For example, to change line 5 to be NEWENTRY, type FWKCS newentry /d5 . To change it back to the original condition as listed on the left side of the equal sign on line 5, type FWKCS /d5 . To change all the /d entries back to their original condition, type FWKCS /revert (A utility, REVERT.BAT, is provided, to warn you about making this change, and then easily change all of them back to their original conditions.) A file which is read into the command line for controlling execution can, itself, include citations to macros [b],[c],[d], [e], and/or [f]. The macros are expanded during interpretation of the file. (Macros cannot cite macros.) And, execution can be intercepted and the entire, expanded command line can then be written to a text file, by using a minus sign "-" in front of the filename, within square brackets, where that bracketed entry is the last entry on the line. To see what this command line is when fully expanded, up to the last non_blank text character of the part which would be used in executing the line, you can write it to text file ZZZ by typing, e.g., FWKCS d:\path\filename.ext upload /a5[cdef] [-zzz] You can then edit ZZZ, and use it as a configuration file for subsequent execution. If you wish, you could have FWKCS read ZZZ as the whole command line, by typing FWKCS [+zzz] and carry out the execution which you had intercepted when you captured the command line into ZZZ. This ability to read and write files can be used in editing macros. For example, if you wish to edit macro [e], you could type FWKCS [e] [-macroe] then edit the file MACROE, and then type FWKCS [+macroe] /de to load the edited material back into FWKCS as macro [e]. Or, you could use the revised file as another macro, e.g., as [b], and call either (or both) of the two macros from the command line, as appropriate. The ability to load macros still applies when FWKCS.COM is a compressed file, provided that its .COM structure has not been damaged. For example, if you compress FWKCS.COM using Phil Katz's PKLITE(R), and do not have the ATTRIBute flag set to read_only, then you can load the file MACROB into macro [b], by typing FWKCS [+macrob] /db . FWKCS makes the changes, and, if permitted, writes itself to wherever it was called from, with the new FWKCS.COM file in uncompressed form. The ability for FWKCS to write macros from FWKCS to a text file (for example, FWKCS [b] [-bwas] writes macro [b] to the ASCII file BWAS) does not depend on, nor affect, whether FWKCS.COM is stored on disc in compressed form. This flexibility helps support the applications which the FWKCS contents_signature system makes feasible, including, by way of example and without limitation, the /a auxiliary functions. For various suboptions, there is an initial letter, followed by a number or a comma ",", followed by further information used by that suboption (such as a subdirectory, d:\path\subdir; or where to find or put a file, d:\path\filename.ext; or some numbers to use, etc.) ending with a semicolon ";", after which the next option starts. Variable_length fields are ended with a semicolon. If the suboption does not need such information, then the next option is put immediately after it. The whole sequence of options forms one contiguous string. When the command options are pieced together from macros, if there is no space between the macros as shown on the command line, then they are put together to form one string. To see an example of this, you can put material into [c] [d] and [e], and then type FWKCS [cde] [-sample] and then type TYPE sample You get the same result if you replace the [cde] with [c][d][e] FWKCS [c][d][e] [-sample] but you get a different result if you put spaces in between, FWKCS [c] [d] [e] [-sample] because the macros are inserted into the structure of the command_line, which in this last case has intervening spaces. Error messages are logged to FWKCS.ERR in the local default directory. Information about actions taken, or errors, is returned on exit via the errorlevel, which can then be used by the calling program (e.g., a .BAT program). For more information on this, see FWKCS /e. For some examples, see /a5 and /a7.3 below, and BANAV.BAT, LISTAV.BAT, and TESTAV.BAT (included in FWKCS112.ZIP). a - Auxiliary Functions ~~~~~~~~~~~~~~~~~~~~~~~ The auxiliary functions provide an errorlevel on exit (see FWKCS/e for errorlevels and messages). Typing FWKCS /a brings up 2 screens listing auxiliary functions which make use of various parts of the program, and which may call other programs to perform work which this program sets up for them; and continues with credits to people whose programs are used with these auxiliary functions. For certain operations under the auxiliary functions, FWKCS calls PKZIP(R) and/or PKUNZIP(R). FWKCS interacts with these two programs is such way as to be able to accept zipfiles independent of what their extension is (including the case where the file does not have any extension), and can accept files which start with any permitted DOS filename character, including, for example, - and @. Because FWKCS can accept a zipfile which does not have any extension, it is necessary to indicate that an extension exists when there is one -- FWKCS will not add a default ".ZIP" to a file. Auxiliary Functions, FWKCS ... /an(...)#(lsuvz);u,tmp;, "n" at left of item, below. Swapping most of program out of memory ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The #(lsuvz);u,tmp; provides for swapping to disc or RAMdisc, to leave more room in memory for programs which are called by FWKCS. The temporary file created when FWKCS swaps out of memory is kept in the same place as other temporary files created by FWKCS. For example, by using u,tmp; (below), you can specify that swapping be to a RAMdisc. #; swap to default directory for all SHELL operations. #lsuvz; swap if List Sort Unzip scanV Zip. Specifying a subdirectory for temporary files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ u,tmp; Use subdirectory d:\path_tmp for temporary files and for swapping out of memory. The subdirectory must exist before this is executed. The same subdirectory can be shared by multiple nodes. Auxiliary Functions 1-6 and 7.3 use defaults #1-4 listed on FWKCS /d; Wildcards * and ? can be used with Auxiliary Functions /a2, /a3, and /a4. Auxiliary Function 1 ~~~~~~~~~~~~~~~~~~~~ Format: FWKCS filelist /a1(.N)#(lsuvz);u,tmp; (>)nonmatch to find novel 1 filenames. 2 contents_signatures. 3 do both tests. #(lsuvz);u,tmp; See discussion of Swapping most of program out of memory, and Specifying a subdirectory for temporary files, under a - Auxiliary functions, above. The test for filenames uses a local copy of DIRGUIDE.TXT(TM) (from FWKDG) and calls QSORT. FWKDG.COM is included in FWKCS112.ZIP. This is for finding which filenames and/or contents_ signatures, in a new list received from a BBS, are not on your system. Suppose, for, example, that you have downloaded 300k of text directory listings in the format used on PCBoard, including multi_line file descriptions, into a file called FILELIST, and removed their directory heading(s). Now you want to compare the list with, e.g., some 20 or 30 thousand names and contents of zipfiles or plain files on your system, and, e.g., the contents of some 175,000 different files contained in those zipfiles. This does that for you, preserving the multi_line comments to help you decide which you want to download. (The multi_line comments prepared at the source BBS under FWKCS Ver. 1.10 - 1.12, using the g option, contain the contents_signature information used by /a1.2 and /a1.3, and also provide, for each zipfile, the date of the latest file in the zipfile.) To run this, first use FWKDG to make DIRGUIDE.TXT. To do this, change to the default directory in which you have PCBoard text directories DIR0 - DIR999. Then, with FWKDG and QSORT (or its equivalent) on your path or in that subdirectory (using the name of whatever sort program you chose, where I have indicated QSORT) to make short form of DIRGUIDE.TXT containing only text directories, type FWKDG /t | QSORT >DIRGUIDE.TXT Option /a1.1 compares filenames with DIRGUIDE.TXT. For example, having made this shorter DIRGUIDE.TXT, type FWKCS filelist /a1.1 nonmatch to put the non_matched names, with their (multi_line) comments, into the file NONMATCH. Option /a1.2 compares contents_signatures (which are provided by the BBS as part of processing uploads using option g of FWKCS Ver. 1.10 - 112) with the cs's for all the plain files, and all the files in all the zipfiles, on your system. To use this, you must first install the FWKCS Contents_Signature System on your computer system. (see SETUP.112, RECIPES.112, and EXAMPLES.112) For example, having installed the FWKCS Contents_Signature System, type FWKCS filelist /a1.2 nonmatch to put the names of the files for which matching contents_ signatures were not found, with their (multi_line) comments, into the file NONMATCH. Option /a1.3 does both operations, first testing whether the names do not match names in use on your system, then testing the contents_signatures. For example, having made this shorter DIRGUIDE.TXT, and having installed the FWKCS Contents_Signature System, type FWKCS filelist /a1.3 nonmatch to put the names of the files for which neither matching names nor matching contents_signatures were found, with their (multi_line) comments, into the file NONMATCH. FWKDG and DIRGUIDE.TXT have other uses, discussed in detail in UTILS.112, section 1. Auxiliary Function 2 ~~~~~~~~~~~~~~~~~~~~ Format: FWKCS (d:)(\path\)filename(.ext) /a2#(lsuvz);u,tmp;(d) Note: the () means that one or more entries is/are optional; for example, this (d) is not macro [d]. Wildcards * ? can be used with /a2. #(lsuvz);u,tmp; See discussion of Swapping most of program out of memory, and Specifying a subdirectory for temporary files, under a - Auxiliary functions, above. Subcommands: d reDate zipfile(s) to the latest date of the files inside. /a2 is to make contents_signature(s), get matching, add new signatures, make new index, make report; this uses default files #1-4 (see FWKCS /d); may call QSORT, LIST. format: FWKCS /a2m - make Y.BAT = " FWKCS %1 /a2 " to call /a2. This is for finding existing files and adding new material, by a one_key command named Y. The version of Y.BAT provided in FWKCS112.ZIP contains on_line help. To see that on_line help, type Y /* To make a terse form of Y.BAT, called YT.BAT, which does not provide on_line help, suitable for use as a subroutine, type FWKCS /a2m (This defaults to not use the d subcommand. To insert the reDate subcommand, use a convenient text processor to edit Y.BAT or YT.BAT, to change FWKCS %1 /a2 to FWKCS %1 /a2d .) To use Y.BAT, just type Y file , where "file" = "(d:)(\path\)filename(.ext)" You can use wildcards in specifying the filename.ext. When the program is ready, it summarizes its findings, and asks if you would like to see the report called d:\path\MATCHING.RPT, thus telling you where the report is. If you type n, it skips displaying the report. If you type y, then, if you have LIST and have not specified the use of internal scroll, it calls LIST to show you the report; if you do not have LIST, or if you have specified that the display be by internal scroll (discussed below), then the report is scrolled on the screen, pausing between display_screens. Then, it asks you if you wish to delete the report. If you type y, it deletes the report. If you type n, then it does not delete the report. If you have used wildcard(s) in specifying the file(s) to process, the program looks for the next file. If it finds one, the new material may be used to overwrite the existing MATCHING.RPT. So, if you want to collect the reports, you may want to do them one at a time, or to write a .BAT program which adds the report to a collection file, and then call that .BAT program using a recursive DOS command line (for an example of recursive DOS call, see item 2 in EXAMPLES.112). Auxiliary Function 3 ~~~~~~~~~~~~~~~~~~~~ Format: FWKCS (d:)(\path\)filename(.ext) /a3#(lsuvz);u,tmp;aNNNNbc(,zcomb;) dj,junked;kNnN,NN(NN);o,oust;mpr,refs;sv(,vdat;)v1,telsysop;w (the options are all contiguous; can be entered using macros and/or via configuration file(s)) Note: FWKCS /a3m makes a short .BAT file, which does not contain on_line help (e.g., for use as a subroutine). To use the m suboption with only the o suboption, place the o before the m. (You need the o to impliment execution of the m, so this does not reduce your number of options.) Note: on exit from /a3, FWKCS sets an errorlevel, which can be used to control the calling program. For information about these errorlevels, type FWKCS /e . #(lsuvz);u,tmp; See discussion of Swapping most of program out of memory, and Specifying a subdirectory for temporary files, under a - Auxiliary functions, above. Suboptions under /a3 : aNNNN Age in months of latest file in zipfile, if older than that, rename file ".OLD" b test for Bad Authenticity Verification stamp. If find Banned AV_stamp, rename the file ".BAV". If find obsolete or deficient AV_stamp, rename the file ".OBD". Note: To prevent certain forms of abuse of the zipfile comment and the AVEXTRA.TXT comment together, it is urged that you use the c option (below) to strip or replace the zipfile comment when testing the AV_stamp on unknown files. To ban a specific zipfile Authenticity Verification stamp from your system, you can use BANAV, provided in FWKCS112.ZIP. For instructions on how to use BANAV, type BANAV /* . For discussion, see option /a7.3b, below. c(,zcomb;) strip (replace) zipfile Comment; else keep. "zcomb" can be in form of d:\path\zcomb. If there is no file specified for the replacement zipfile comment, and the c option is used, then the zipfile comment is removed from a zipfile. If the file containing the replacement is provided and found, then it is used (the replacement comment should end with the carriage_return line_feed required by PKZIP). If the c option is not used, then the zipfile comment is left as is. d reDate zipfile to latest file in it. j,junked; append to file (d:)(\path\)Junked(.ext) a listing for each zipfile entry which has been deleted, giving the name which it had in the zipfile and any path information which was zipped with the file, together with the file's contents_signature. This provides a record which may be used to support retrieval and reconstruction, using the same contents from other files on the system, regardless of the name used elsewhere on the system. kN treat zipfiles which contain at least one apparently encrypted file as follows: default is k0 (N = 0) N = 0 if NOT s, then, if a zipfile contains an apparently encrypted file, change the filename to ".ENC". N = 1, 2, 3 apparent encryption not used as a basis for changing the filename. m delete file from zipfile if a cs_match was found in CSLIST.SRT or CSLIST1.SRT. This selects such a file for inclusion in the list prepared under the o Oust option. It does not perform any deletion itself. nN,NN(NN); novelty test, based on comparison of contents_signature(s) of all entries in a zipfile against all the contents_signatures in CSLIST.SRT and CSLIST1.SRT, e.g., from all your current files and all the files which have been retired from your system since you first installed FWKCS. This is actually three, different tests: n1,NN bad if =NNNN cs_matched; this might be used in situations where people have a collection of items, such as music, and one wants to catch and stop duplication of internal files. Example: n3,1 applied to a zipfile with 10 files in it, would require that every file of the 10 be new. Example: n3,2 applied to a zipfile with 10 files in it, would require that at least 9 of the 10 be new. if Question as to Sufficient Novelty arises, rename file ".QSN". o,oust; make a list of files to be deleted from a zipfile, giving their names and zipped_paths inside the zipfile. "oust" can have full d:\path\filename.ext. To operate under /a3 or /a4, the "o" command must be accompanied by a filename to which to send names and zipped_paths for deletion from the zipfile, else it is cancelled. The o subcommand prepares a list for deletion according to subcommands m, p, and w. p Purge non_AV file(s) from Authenticity Verification Zipfile. This causes such files to be included in the list prepared under the o Oust option, above. It does not perform the deletions itself. r,refs; collect references with matching contents_signature cited in CSLIST.SRT and CSLIST1.SRT. In CSLIST.SRT and CSLIST1.SRT, these are marked with a lower_case "r" in decimal_column 17, and have this format: | 16 characters| | 16 characters| CONTENTSIGNATUREr CONTENTSIGNATURE |4s| (four spaces or markings) |<-- column 17 In each output line, the two contents_signatures are swapped, so that the one which was on the right is at the beginning of the line, where it can be used for finding another cs_match; and the lower_case "r" is capitalized to "R", to help avoid confusion. For example, FWKCS can search a roughly 18 Megabyte file in typically a fraction of a second: this ability to make cross_references against the entire system is potentially a remarkably powerful tool. s keep Same extension (don't rename file); but still sets exit errorlevel: if it would have renamed the file, then if s, it displays the errorlevel on the screen. In either case, it returns the errorlevel when it exits. This subcommand is useful for integrating the FWKCS(TM) Contents_Signature System into .BAT programs, etc. v(,vdat;) call SCAN for plain_file Virus (use external virus_data_file) vdat indicates (d:)(\path\)filename(.ext), an external virus_data_file made according to McAfee Associates instructions. Otherwise, it just calls SCAN. NOTE: "v" can be used without the external data file. It is a separate command, and has a different capability than "v1", below. If v is not explicitly given and "v1,telsysop;" is specified, v without external data is assumed. If find apparent virus, rename file ".VIR". v1,telsysop; d:\path\file for warning about "virus" If find apparent virus, post a warning in the specified file. If the file to use for warning the operator does not exist, then create it, else append to existing file. w Wipe (tell o to list for deletion) from a zipfile every file which has a contents_signature which matches a cs in CSLIST.SRT or CSLIST1.SRT which is marked in decimal_column 17 with a lower_case "w". For example, this can be used for removing known ads. Renaming files under /a3 ~~~~~~~~~~~~~~~~~~~~~~~~ s prevents renaming the file, as discussed above. if NOT s, renames files which did not pass test (alphabetized according to the extension): banned zipfile Authenticity Verification stamp ".BAV" duplicate_zcs ".DUP" - the file is a zipfile and all the files in it seem to be in one_to_one correspondence with a set of files appearing in a zipfile which is known to the system, regardless of name, order, comments, compression ratio, or dates. zipfile which contains an apparently encrypted file ".ENC" zipfile with obsolete or deficient AV_stamp ".OBD" >NNNN months old ".OLD" question as to sufficient novelty ".QSN" redundant ".RED" - the file is a zipfile and all of the files in it are not new, but they do not all appear as the only contents of a zipfile known to the system redundant_non_zip ".RNZ" - the file is non_zip and not new virus ".VIR" - in the case of /a3, this refers to the outer shell, because a zipfile would not be unzipped during an /a3 test exclusion_flagged ".XFL" - in searching for matches for the contents_signature(s) for a file or for the files in a zipfile, a lower_case "x" was found in decimal_column 17 of a matching cs in CSLIST.SRT or CSLIST1.SRT. The program, ACCESION.BAT as provided in FWKCS112.ZIP, contains on_line help. To see this help, type ACCESION /* A short form of ACCESION.BAT, called ACCESN.BAT, which does not contain on_line help, can be generated directly from FWKCS.COM. Foe example, it can be used more efficiently as a subroutine. To generate the short form of ACCESN.BAT = " FWKCS %1 /a3[b] " to call this collection of functions, type FWKCS /a3m Then, prepare the macro [b], by typing FWKCS ContiguousStringOfOptionsUpTo73BytesLong /db and check to see that it came out OK by typing FWKCS /d , looking down the left side of the screen to find item B, and then reading across. This is for accessioning files into the contents_signature system, and can process both zip and non_zip files together, including zipfiles which contain apprently encrypted files. It accepts wildcards. It summarizes its results to the screen as it goes, and treats the filename.ext for each file as follows: If the file passes the various tests which you have specified, the name is unchanged, and the contents_signature(s) are made part of the system; If the program tries to change a name and runs into a name conflict, it reports this to the screen; else, if not s, the filename is changed as listed above. To run it, type ACCESION *.* , or you can be more specific, if you wish. It can handle a single file, or, if you wish, be turned loose on part of, or a whole, (sub)directory. After running this on a collection of files, you may want to run FWKCS /v, as discussed below, to help keep your update_time conveniently short. Auxiliary Function 4 ~~~~~~~~~~~~~~~~~~~~ Format: FWKCS (d:)(\path\)filename(.ext) /a4#(lsuvz);u,tmp;aNNNNbc(,zcomb;) df,free;j,junked;kNnN,NN(NN);o,oust;mpr,refs;sv(,vdat;)v1,tsysop;wz (the options are all contiguous; can be entered using macros and/or via configuration file(s)) #(lsuvz);u,tmp; See discussion of Swapping most of program out of memory, and Specifying a subdirectory for temporary files, under a - Auxiliary functions, above. /a4 is like /a3, but PKUNZIPs zipfiles. /a4 can test for virus in file which was inside zipfile, and for zipfile in zipfile. All the options provided under /a3 are also provided under /a4. Additional or changed options: f,Free; subdirectory into which to unzip files; default = ##CELLAR If you are using /a4 on a local system, you can use a drive and path as part of specifying the free work area, f,d:\path\##cellar; and then automatically reach that free work area when you are working in directories other than the one directly above that ##cellar. The reason why the d:\path is omitted for multi_user systems, is to permit each user to automatically go to a different default ##cellar. For example, a system with 5 users can have a ##CELLAR under each of the default directories for the 5 users. Note that this can be set separately in macro [b] for a single local user, as distinct from operation of a BBS. In this way, a single local user (such as a sysop) can be free to use /a3[b] or /a4[b] in many different directories which do not have ##CELLARs, while the same system is running a multi_node BBS with a ##cellar under the default directory of each of the nodes, respectively. kN treat zipfiles which contain at least one apparently encrypted file as follows: default is k0 (N = 0) N = 0 if NOT s, then, if a zipfile contains an apparently encrypted file, change the filename to ".ENC". N = 1, 2, 3 provided that the structure of the zipfile meets the requirements set forth below, apparent encryption not used as a basis for changing the filename. Note: encrypted files are not virus_SCANned. Structure of zipfile containing an encrypted file: (Zipfiles containing both encrypted and non_encrypted files which do not conform to these requirements will be treated as having failed the ziptest, and if not s, will be renamed ".FZT".) EITHER all files in the zipfile must be encrypted OR the zipfile must contain zipped path information, AND every file in a given subdirectory is encrypted OR every file in a given subdirectory is NOT encrypted AND there are at least two files with the same name and different path information which have different contents. Example: In this example, each file in the first level contains non_secret material pertaining to the secret material in the file with the same name in the subdirectory below. PATNTWRK.01 SECRET\PATNTWRK.01 PATNTWRK.02 SECRET\PATNTWRK.02 PATNTWRK.03 SECRET\PATNTWRK.03 PATNTWRK.04 SECRET\PATNTWRK.04 In this example, every file which does not have path information is not encrypted, AND ~~~ every file in the SECRET subdirectory has the same name as the corresponding file in the directory above and is encrypted. ~~ z ZIPs_only, else can handle both zip and non_zip files. In addition to all the operations which can be performed by /a3, these are provided under /a4: if NOT s, renames failed ziptest ".FZT" virus ".VIR" - including both virus in shell and virus inside zipfile. See discussion, below. zip in zip ".ZIZ" - a zipfile was found inside a zipfile if z, nonzip ".NZ" - this extension is to avoid confusion which might be caused by forcing a zipfile test on a file which is non_zip. Discussion: It is possible to put two different files with the same filename into a zipfile, by using the PKZIP -r and -p options to recurse subdirectories and retain the paths. Because of this, it is not necessarily sufficient to just unzip all of a zipfile together into a single subdirectory. If you use the PKUNZIP -o option, so that the system does not "hang", a second occurrence of the same filename causes the first file with that name to be overwritten. Thus, SCANning the subdirectory for viruses would not find a virus hidden in the first of those two files. For many systems, it could be a security risk to let a node create a large number of subdirectories, and the individual nodes are not given CREATE rights for making subdirectories. Note: You have use option kN to control whether zipfiles containing apparently_encrypted file(s) are permitted on your system: encrypted files are not virus_SCANned. To block encrypted files, you can set option kN to k0. (That is automatically set to k0 if you run CSAMACS to install your \CSA Contents_Signature Activity subdirectory. You can change it, by editing macro [e]. See "The FWKCS command_line:", above.) When testing for viruses, or searching for zipfile(s) inside a zipfile, FWKCS uses contents_signatures to test for whether the (outermost) zipfile contains different files with the same filename. Under /a4, /a5, and /a6, such zipfiles are processed by separately unzipping and scanning groups of files which have the same inner_path and different filenames. In this way, the full DOS tree of permitted paths and subdirectories can be processed, using only one subdirectory as a work area, without having one file overwrite another. When operating a BBS with multiple nodes, a separate work area (default = ##CELLAR) is provided by the operator under the default directory of each node, and that one work area is enough for that node. By means of this special capability, which is coded in assembly_language, FWKCS is able to provide important protection for both the integrity of the uploaded file and the security of the BBS which receives the file. The author of the uploaded software is thus provided with the freedom to use the valuable structural capabilities which Phil Katz has provided in his PKZIP/PKUNZIP system, including the ability to zip an entire DOS tree of subdirectories if he or she wishes; and the author is assured that his or her signature, in the form of a zipfile Authenticity Verification stamp, will not be removed from his or her work during the testing. As provided in FWKCS112.ZIP, ACCESZIP.BAT contains on_line help. To see this help, type ACCESZIP /* . A short form of ACCESZIP, called ACCZIP.BAT, can be generated directly from FWKCS.COM, for example, for more efficient use as a subroutine: format: FWKCS /a4m to make ACCZIP.BAT = " FWKCS %1 /a4[b] " to call /a4, with the same procedure described for /a3, above, for making macro [b]. Auxiliary Function 5 ~~~~~~~~~~~~~~~~~~~~ auto test + accession files: Format: FWKCS file (UPLOAD) (UPDESC.nnn) /a5(options) (out) options: #(lsuvz);u,tmp;aNNNN(,(2),old;)b((2),bav;)c(,zcom;)de((2),ziz;) f,free;gN,N(,...;)h,msg;ij,junk;k0ke,enc;kf,fzt;l,uplog; nN,NN(NN)(,qsn);ompq,potvirus;r,refs;st,trashold;v(,vdat;) v1,telsysop;wxyz (options are contiguous; command_line can use macros + configuration file(s); see discussion of command_line usage, above, and see FWKCS/h) The default macros for this are [cdef], providing 292 bytes of user_settable, internal storage available for inclusion in the command line. If more internal storage is critically needed, [b] can be taken away from /a3 and /a4 and used here, e.g., [bcdef], for a total of 365 bytes. If you wish to use more space than is provided by the internal storage, you can incorporate an external file into the command line, using [+d:\path\filename.ext] (the square brackets "[]" and the plus sign "+" are literal). Note that the total command line, with all external files included, and with all macros in the command line and in the external files replaced with their full strings, should not exceed 1024 bytes. ~~~~~~~~~~ /a5 runs under PCBoard v14.n; may call QSORT, PKZIP, PKUNZIP, SCAN. default output: can make PCBPASS.TXT or PCBFAIL.TXT. (out) if present, receives message and suppresses PCBPASS.TXT, PCBFAIL.TXT. Whether or not UPLOAD, --- sets errorlevel (see /e screen below, or FWKCS/e. This is discussed in detail under option h, below.) #; swap all. #lsuvz; swap if List Sort Unzip scanV Zip. This provides a way to increase the amount of memory available for running programs called by FWKCS under /a5. For example, this may be especially helpful when running under a multi_tasking system. u,tmp; Use d:\path_tmp for temporary files, including swapping out of memory. d:\path_tmp can be a ramdisk, for speed. For safety, swapping is only to a subdirectory, not to a root directory. f,Free; empty subdir, defaults to ##CELLAR. Note that normally there is no path for the ##CELLAR. For a multi_node system, such as an electronic bulletin board, a ##CELLAR subdirectory is placed directly under the default directory of each node. All of these free workareas have the same name, for example, ##CELLAR. Because the free workarea does not have d:\path specification, each separate node automatically does its work in its own, separate ##CELLAR. This lets the system handle concurrent uploads without overlapping the workareas. If you are using /a5 on a local system, you can use a drive and path as part of specifying the free work area, f,d:\path\##cellar; and then automatically reach that free work area when you are working in directories other than the one directly above that ##cellar. For example, a configuration file can include such details, and then be incorporated into the command line via [+file] for local use. x(0) ascii output, else uses PCBoard v14.n @codes in message(s). (For customizing messages, see option h, below.) if x0, use filenames PASS.TXT and FAIL.TXT rather than PCBPASS.TXT and PCBFAIL.TXT. z Zipfiles only, forces PKUNZIP test; else handles both plain and zip. Note: under /a5, if UPLOAD, a zipfile which contains internal zipped_path information is subjected to a filename and contents_signature test, for whether a zipfile contains different files with the same filename. Such a zipfile is automatically processed in a way which prevents the accidental overwriting of one file onto another. See discussion, above, about virus testing. under /a5, if NOT upload, then FWKCS does not test for whether a zipfile contains different files with the same filename, nor test for zipfile in zipfile, nor run a virus test. Rather, for a zipfile already on the board, FWKCS provides a report to send to the user, reformatted from the results of running a PKUNZIP -t test of zipfile integrity. If the file is not a zipfile, then a contents_signature is provided. If the y option is used, then contents_ signature information is also provided for zipfiles. --- if UPLOAD: Upload Credit: Note: certain options provide a choice of whether you will give upload credit for a file which fails a test, and provide the ability to set aside the file for further study. The way in which upload credit is given, is to return to PCBoard a PCBPASS.TXT file. This may result in the file being listed as available for download, when in fact it has been set aside. Unless you are going to examine your target subdirectory frequently to move the sequestered material to a place from which it can be downloaded, or you make files in that target subdirectory available for downloading, then you may wish to set the option to not give upload credit. excludes (zip)file if an x flag is found in column 17 of CSLIST.SRT or CSLIST1.SRT, for any contents_signature which matches the contents_signature of a non_zip file, or which matches the contents_signature of any file in a zipfile. To mark material for exclusion from your BBS, you can use EXCLUDE, which is provided in FWKCS112.ZIP (see /a7.3, below). For instructions on how to use EXCLUDE, type EXCLUDE /* . aNNNN(,(2),old;)) Age test in months. If and only if a subdirectory is listed to put them in, then "2" means that upload credit is given for files which fail on this test. (See note on Upload Credit, above.) Subdirectory can have full d:\path\dirname. This command is slightly different from others, in that if any information is given after the NNNN number of months, then a comma appears after the number of months; this is because the number of months is a variable_length_field, which may be followed by another decimal digit. For example, if you do not wish to give upload credit for files which are more than 18 months old, but do wish to be able to look at them just in case, you would use a18,,d:\path\old; Note the use of two adjacent commas after the age, where the "2" for upload credit was omitted. b(2),bav; test for Bad_AV; if and only if the subdirectory exists, then "2" = give upload credit. The subdirectory can have full d:\path\dirname. Note: To prevent certain forms of abuse of the zipfile comment and the AVEXTRA.TXT comment together, it is strongly urged that you use the c option (below) to strip or replace the zipfile comment when automatically testing the AV_stamp on unknown files. (See note on Upload Credit, above.) (For instructions on how to ban a specific zipfile Authenticity Verification stamp from your system, type BANAV /* . See also option /a7.3b, below.) c(,zcom;) strip (replace) zipfile Comment; else keep. "zcom" can be in form of d:\path\zcom. If there is no file specified for the replacement zipfile comment, and the c option is used, then the zipfile comment is removed from a zipfile being processed as an upload. If the replacement is provided and found, then it is used (the replacement comment should end with the carriage_return line_feed required by PKZIP). If the c option is not used, then the zipfile comment is left as is. d reDate zipfile to latest file in it. e(2),ziz; upload crEdit for zipfile(s) with zipfile(s) inside: if and only if the specified subdirectory exists, then "2" means that upload credit is given. The subdirectory can have full d:\path\dirname. (See note on Upload Credit, above.) If the s option is not used, then, If the subdirectory exists, and the e option is used to designate that subdirectory for receiving zip_in_zip uploads, then a zip_in_zip upload is put into that subdirectory. Else, zip_in_zip uploads are deleted. gN,N(d(:\path(\filename(.ext)));) Get UPDESC.nnn ^ ^0,1 ; 0,2; ^ 0,4 ^ ^ 1 use internal defaults to find file. If the second entry is UPLOAD, FWKCS checks to see if there is a third entry. If it finds a third entry, that third entry is used for the full d:\path\filename.ext for the description to process. 2 drive is given, find the rest. 3 d:\path is given, find the rest. 4 (d:\path\)filename is given, find node in environment. 5 use exact d:\path\filename.ext as given. gc cancel any g option set earlier in command_line. (used in LUT.BAT. For on_line explanation of LUT, type LUT /* .) An entry of the g option on the command line can override a previous setting of the g option in the same string of options. This feature is used in modifying the g option executed when LUT.BAT processes a "description" (which can be an empty file, or a filename to be used for a description file which LUT then creates -- but note that an empty "description" file might not follow a rejected file into its target subdirectory). During processing of an upload, PCBoard provides a file called UPDESC.nnn, containing the filename, size, and date, and the description provided by the uploader. The nnn is the node number on a multinode BBS, and is different for each node. The g option tells FWKCS to modify UPDESC.nnn, so as to include the true file size after any changes are made, the file's new external date, the FWKCS contents_signature for the file and, if it is a zipfile, the date of the latest file in it. This contents_signature can be read using the /a1.2 and /a1.3 suboptions. This g option is the "sender" end, and /a1.2 or /a1.3 is the "receiver" end. This combination provides a simple way for a person who receives a "new uploads" directory from a BBS to compare the contents_signatures of files listed in it with the cs's for all the non_directory files on his/her system, independent of filenames and comments, before deciding to download any of those newly listed files. And each of those comparisons carries with it the enhanced resolution provided by the FWKCS contents_ signature system, typically more than two orders of magnitude better than that provided by a 32_bit CRC. In this g option, the first decimal digit specifies how FWKCS is to find UPDESC.nnn. The second digit, 0...7, is bit_mapped. It sets how the contents_signature is presented (upper or lower case); sets the format (YEAR-MO-DY or MO-DY-YR) for specifying either the latest date found inside a zipfile (marked as "Latest:") or the date of a nonzip file (marked as "Dated:"); and sets whether or not the cs_line is visually suppressed in graphics mode (note that the character string is sent, but the contrast for monochrome display is set to "black" on "black"). The combined results for the second digit are: 0 YEAR-MO-DY, UPPER_CASE. 1 MO-DY-YR, UPPER_CASE. 2 YEAR-MO-DY, lower_case. 3 MO-DY-YR, lower_case. 4 YEAR-MO-DY, UPPER_CASE, suppressed contrast in graphics. 5 MO-DY-YR, UPPER_CASE, suppressed contrast in graphics. 6 YEAR-MO-DY, lower_case, suppressed contrast in graphics. 7 MO-DY-YR, lower_case, suppressed contrast in graphics. When there is a separate processor for each node, the node number can be provided in the environment using the SET command, SET PCBNODE=NNN where NNN is the node number with no leading blanks. You can choose where UPDESC.nnn will be placed by PCBoard. The default in FWKCS is set to C:\UPLOADS\UPDESC.nnn. You can change the drive, or drive\path, etc. If you have set PCBoard to send them to the default directory of each node, then you can specify g4,N,UPDESC; where N is set as discussed above. Because there is no PATH specified, FWKCS would look in each node's default directory for the UPDESC used by that node, respectively, using the nnn provided in that node's environment. For example, g4,1,UPDESC; would tell FWKCS to look for UPDESC.nnn in each node's default directory, using an extension gotten from PCBNODE=NNN in that node's environment, respectively, and to format the date of the latest file in a zipfile as MO-DY-YR. In a multitasking system, under pre_14.5a versions of PCBoard, you might provide a different PCBTEST.BAT for each node, leave this g option out of the [cdef] macros, and provide a full file identification for, for example, node 2, using g5,N,d:\path\UPDESC.2; in that specific copy of PCBTEST.BAT. The ability to use a specific file named in the third entry on the command line is used in a utility for locally processing batches of files, FWKLOCAL, discussed in item 4 in UTILS.112 and provided in FWKCS112.ZIP. The calling program provides %3 on the command line. In order to detect misdirection, FWKCS does not create a missing file for that third entry on the command line. That file has to exist before FWKCS tries to open it, although a zero_length file can be used there. (As noted above, one can explicitly designate a description file within the g subcommand. That file must exist to be opened by FWKCS.) Under option g, if a file is rejected and a subdirectory has been specified to hold it (according to the reason for which it was rejected), then FWKCS puts a copy of its original description in the same subdirectory with the rejected file. The copy of the description is given the same name as the rejected file, but with the extension ".DSC". Option g also is used in supporting the remote inquiry option i, by sending a report to the private upload directory list. There three outcomes which it can report: a successful remote cs inquiry, a defectively formatted cs inquiry list, and a cs inquiry list which exceeded 512 entries. If you have set PCBoard to provide the uploader's name, then the name will also be provided with the modified description. (for processing remote inquiries, see option i, below.) Depending on the kN option (below), option g may also modify the upload_description to provide information about whether an uploaded zipfile contains apparently encrypted file(s). h,msg; msg is the subdirectory in which you have messages for FWKCS to use in preparing its reports to send to users. It can have full d:\path\msg specification. Because of the complexity of possible situations on different systems, the messages prepared by FWKCS for sending to the user are composed by FWKCS according to what it finds in carrying out its work. Thought has gone into keeping key parts of the FWKCS messages short, partly out of respect for people's time and the fact that people may be calling long distance. However, over the years, you may have devoted part of your efforts to creating a special, personal environment which you and the users of your board now find pleasant, comfortable, and productive. And, you may have specific instructions, etc., to add, in order to help the user use your board. For at least these reasons, you may wish to customize and personalize the messages. FWKCS can accept message material which you prepare in advance. You can use this to create messages in which you can control graphics, use personal names via the PCBoard @codes, select messages of your own, and/or send information provided by FWKCS in a framework of your own design. Customizing messages to users: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To use this option, put your message material in the d:\path\msg location specified in the h,msg; option. Except as noted below, FWKCS will incorporate material from those external files, according to the conditions it finds. Except as noted below, for each of the listed cases (with corresponding exit errorlevels), you can provide three messages: FIRST.nn the first part of your message that goes specifically with that case. MID.nn the mid_part of your message that goes specifically with that case. If this mid_part is found, then it is used >>------> to replace the message from FWKCS. ~~~~~~~~~~ LAST.nn the last part of your message that goes specifically with that case. You can also have general material which goes out with each message. You may have different preface and ending material, according to whether the message is about an upload or is in response to a call for the TEST function: PREFACE.TST used if message reports result of a TEST PREFACE.UPL used in response to an upload ENDING.TST used if message reports result of a TEST ENDING.UPL used in response to an upload >--> Note: the program tests for the existence of message material before using it. So, you can add material a little at a time. A program is included, MSGS.BAT, which makes dummy files for all the messages used under this option. Each of the files contains its name. To see which files to put your material in, run MSGS to make these short files. Then, upload a file locally and look at the resulting PCBPASS.TXT or PCBFAIL.TXT: it will contain a list of the message files to use for that case, in the order in which they will be incorporated into the output message. You can then remove the identifier line from each of those files, and put in your message (which can include graphics, PCBoard @codes, etc.). For instructions on how to use MSGS.BAT, type MSGS /* . When you are installing FWKCS, if you run CSAMACS to set up the \CSA Contents_Signature Activity place, or run one of the automatic setup procedures (which run CSAMACS), you will find a copy of MSGS.BAT in the subdirectory \CSA\MSG, where the messages go. Until you create those files, FWKCS uses its own internal message generation capabilities. You can delete or hide the MID.nn files. For example, DEL MID.* OR REN MID.* MUD.* OR PKZIP MIDPARTS MID.* -m If you remove the MID parts, then the FWKCS messages are used as the mid_part of the final message, while, for example, you are using FIRST.nn and LAST.nn to set graphics, add instructions to the user, etc. Here are the message options, case by case: except as noted, for any case nn you can provide FIRST.nn MID.nn LAST.nn ---------------------------------------------------------- Case 0, exit errorlevel 0: For TEST: a successful non_upload test. FIRST.T0 (for your use) Note: MID.T0 is reserved for use by FWKCS. LAST.T0 (for your use) For UPLOAD: a completely new upload, which was accepted. FIRST.00 (for your use) MID.00 (for your use) LAST.00 (for your use) ---------------------------------------------------------- Case 1, exit errorlevel 1: only some of the material was new, but was enough to pass. ---------------------------------------------------------- (Note: At present, FWKCS doess not use exit errorlevel 2.) ---------------------------------------------------------- Case 3, exit errorlevel 3: material would have been deleted if the file were not an Authenticity Verification zipfile. ---------------------------------------------------------- Case 4, exit errorlevel 4: material was deleted from within a zipfile, and the remaining material was accepted. ---------------------------------------------------------- Note: Case 5 does not apply under /a5 and /a6: if you have specified that only zipfile format is acceptable, by using the z option, then every file is tested using Phil Katz's PKUNZIP(R). A non_zip file would be reported under Case 7. ---------------------------------------------------------- Case 6, exit errorlevel 6: an exclusion flag was found (e.g., someone tried to upload commercial software). ---------------------------------------------------------- Case 7, exit errorlevel 7: the file failed the PKUNZIP(R) zipfile integrity test. it could have been an upload, or it could have been on the BBS and failed on TEST. For TEST: FIRST.T7 (for your use) Note: MID.t7 is reserved for conveying a reformatted report from PKUNZIP about a file that failed. LAST.T7 (for your use) For UPLOAD: FIRST.07 (for your use) Note: MID.07 is reserved for conveying a reformatted report from PKUNZIP about a file that failed. LAST.07 (for your use) ---------------------------------------------------------- Case 8, exit errorlevel 8: a contents_signature match was found for every contained file, but not for zipfile_contents_signature; the contents may have appeared in a larger prior ZIP, or been spread among more than one prior ZIPfiles. ---------------------------------------------------------- Case 9, exit errorlevel 9: a contents_signature match was found for each input file; the input file(s) might be not ZIPped. ---------------------------------------------------------- Case 10, exit errorlevel 10: a contents_signature match was found for every file, including zipfile_contents_signature; this material probably all appeared in a prior (ZIP)file. ---------------------------------------------------------- Case 11, exit errorlevel 11: the file appears older than the age test is set for. ---------------------------------------------------------- Case 12, exit errorlevel 12: a zipfile was found inside this zipfile. ---------------------------------------------------------- Case 13, exit errorlevel 13: this (zip)file may contain a "virus". ---------------------------------------------------------- Case 14, exit errorlevel 14: a question as to sufficient novelty has arisen. ---------------------------------------------------------- Case 15, exit errorlevel 15: zipfile has a Banned Authenticity_Verification_stamp. ---------------------------------------------------------- Note: errorlevels 16, 17, 18 are used locally for reporting the results of testing for an Authenticity Verification stamp. See /a7.3, below. ---------------------------------------------------------- Case 19, exit errorlevel 19: a remote contents_signature search was successfully performed. FIRST.19 (for your use) Note: MID.19 is reserved for use by FWKCS for answering remote request(s) for a contents_signature search. LAST.19 (for your use) ---------------------------------------------------------- Case 20, exit errorlevel 20: on a remote contents_signature search, the contents_signature list was defective as to format. ---------------------------------------------------------- Case 21, exit errorlevel 21: on a remote contents_signature search, too many entries: maximum per search is 512 entries. ---------------------------------------------------------- Case 22, exit errorlevel 22: a zipfile was rejected because it contained apparently encrypted file(s) (kN option set to k0). ---------------------------------------------------------- Case 23, exit errorlevel 23: a zipfile was rejected because it had an obsolete or deficient Authenticity Verification stamp. ---------------------------------------------------------- Note to Sysops: The version of PCBTEST.BAT created by FWKCS /a5m contains only one line: FWKCS %1 %2 %3 /a5[cdef]. A single copy of this .BAT program can serve many nodes. FWKCS can incorporate material which you supply, according to that it finds. FWKCS also makes its decision available to you for further or other use, by setting the exit errorlevel, as noted for each case, above. (also see screens FWKCS /e) In designing your messages, please bear in mind that there are people who are blind, and who use relatively simple computer reading software which may not respond well to graphics. They are welcome and productive members of our computer networks. They, and others, may choose an OFF setting for the graphics mode (e.g., on PCBoard). i process remote Inquiries requesting a contents_signature search. With option i, a person can ask ahead to find out if material which he/she is thinking of uploading is already on a BBS. Option i also provides potentially valuable cross_search capabilities. For example, a user could use this to find other files on a BBS which contain material related to a file he/she has, by learning in which zipfiles the specific file appears, and then use the y form of the TEST function to obtain full sets of contents_signatures for all the files in each of those zipfiles, and so on. This remote_Inquiries option helps the BBS sysop and the BBS users work together as a team. As a safety precaution, as presently configured, this remote cs_search is limited to 512 entries input per search. The format for the contents_signatures is provided by FWKCS/1. The contents_signatures must be presented in a zipfile which contains only one file, named FWKCSLOO.KUP. The zipfile which is sent can have any name permitted under DOS. If the i option is used on the receiving BBS, then that incoming zipfile is unzipped; the single FWKCSLOO.KUP file (if longer than one line) is sorted; and that list of contents_signature(s) is used as an input for finding matching contents_signature(s). Both CSLIST.SRT and CSLIST1.SRT are searched. As a precaution before uploading a file, one can set the default directory to be different from the directory which contains the file about which one wants to obtain information, and then use the same name when creating the inquiry_file to send. That way, one can also find out at the same time whether that filename is already taken on that BBS. (See example, below.) A utility is provided, LOOKUP.BAT, which the remote BBS user can use to automatically create the material to send, for the remote contents_signature inquiry to take place. See LOOKUP.DOC. For on_line help, type LOOKUP /* . Example: default <-- where you are now | sub1\FOO.ZIP <-- the file you want to get information about Type LOOKUP sub1\foo.zip foo.zip Then upload this new FOO.ZIP which is in the default directory, to find out whether you should upload the original FOO.ZIP. A related procedure, RCROSREF.BAT, is provided, for following up on contents_signature cross_reference(s) found in a remote contents_signature search. For on_line help, type RCROSREF /* . The results of the search are reported to the user by making a PCBFAIL.TXT (under /a5) or FAIL.TXT (under /a6) file, so that the uploaded search material does not get counted for upload credit. If the s option is not set, then the uploaded inquiry_file is automatically deleted, so that the same name can be used again. The procedure can be logged to the FWKCS uploads log, and the private uploads directory can receive a revised upload description, which includes a note saying what happened (to modify the report sent to the private uploads directory, see option g, above). Three possibilities are covered; corresponding errorlevel values are provided on exit, e.g., for controlling .BAT programs and for customizing messages (see h,msg; above): exit ERRORLEVEL Successful remote cs_search. 19 The cs list furnished by the user was defective as to format. 20 The cs list furnished by the user had more than 512 entries. 21 Using PCBoard's ability to identify the uploader in the uploads directory, a sysop can see who might need help in using this feature, and make it easier for those uploaders to provide interesting new material for the board. j (see suboptions under o (/a5)) kN treat zipfiles which contain at least one apparently encrypted file as follows: default is k0 (N = 0) 0 if zipfile contains at least one apparently encrypted file, reject the zipfile. 1 zipfiles which contain apparently encrypted file(s) according to the structural rules set forth below, are treated on UPLOAD similarly to non_encrypted files, but without the ability to virus_scan. With support from option g, the uploads description is modified to report that the zipfile contains apparently encrypted file(s). 2 like 1, but the message is not visible in graphics mode. 3 like 1, but no message about apparent encryption. Structure of zipfile containing an encrypted file: (Zipfiles containing both encrypted and non_encrypted files which do not conform to these requirements will be treated on UPLOAD as having failed the ziptest, and will be disposed of accordingly (see options kf,fzt; and s (below)).) EITHER all files in the zipfile must be encrypted OR the zipfile must contain zipped path information, AND every file in a given subdirectory is encrypted OR every file in a given subdirectory is NOT encrypted AND there are at least two files with the same name and different path information which have different contents. Example: In this example, each file in the first level contains non_secret material pertaining to the secret material in the file with the same name in the subdirectory below. PATNTWRK.01 SECRET\PATNTWRK.01 PATNTWRK.02 SECRET\PATNTWRK.02 PATNTWRK.03 SECRET\PATNTWRK.03 PATNTWRK.04 SECRET\PATNTWRK.04 In this example, every file which does not have path information is not encrypted, AND ~~~ every file in the SECRET subdirectory has the same name as the corresponding file in the directory above and is encrypted. ~~ ke,enc; Keep rejected zipfiles which contain apparently Encrypted files in ENC subdirectory. The subdirectory can have full d:\path\dirname. kf,fzt; Keep zipfiles which Failed zipfile test in FZT subdirectory. The subdirectory can have full d:\path\dirname. l,uplog; overrides the default upload Log (which is specified in line "A" on the default screen, FWKCS /d); this override can use full d:\path\filename. Here is a partial example of what the log may look like: date time comment file filename.ext ContentSignature length (decimal) 1990-11-04 14:29:32.84 Duplicate 2488 J.ZIP zcs = AA99D801 BE2 1990-11-04 14:30:47.65 Redundant 11641 ASDF.ZIP 1990-11-04 14:33:16.88 Failed Zip 2488 FOOEY.ZIP 1990-11-04 14:37:41.35 Accessioned 17040 QWERTY.ZIP zcs = 00A4266D 45AF What is done with the file in the various cases depends on options you have set up; option "s" saves rejected files, otherwise rejects are typically either deleted or sequestered. --- Date and time are in the format year-mo-dy hr:mn:sc.nn which is Year - Month - Day Hours(24) - Minutes - Seconds.Hundredths This can easily be sorted in chronological order. If you have more than one uploads log, you can simply copy them together into one file and sort the combined file to put them in order. -- Comment column: Entries which can appear in the upload log comment column include Accessioned the (zip)file was accepted and entered into the system. for zipfile, its zipfile_contents_signature (zcs) is given. for non_zip file, its contents_signature (cs) is given. Age> mo failed age test; the number of months specified for the test in inserted; its zcs is not given; if NOT s, then IF OLD directory is specified, file is put there. ELSE file is deleted. Banned AV the zipfile had an Authenticity Verification stamp which is believed to have on occasion been attached to creative work without the true author(s)'s consent; its zcs is not given; if NOT s, then IF BAV directory is specified, file is put there. ELSE file is deleted. CsList >512 the uploaded zipfile was a remote inquiry for a contents_signature search; it was rejected because the cs list which it contained had more than 512 entries listed as input for the search; the zipfile's zcs is not given; if NOT s, then the zipfile is deleted. CsList Bad the uploaded zipfile was a remote inquiry for a contents_signature search; it was rejected because the cs list which it contained was defective as to format; the zipfile's zcs is not given; if NOT s, then the zipfile is deleted. CsLookup OK the uploaded zipfile was a remote inquiry for a contents_signature search; it was processed successfully; the zipfile's zcs is not given; if NOT s, then the zipfile is deleted. Duplicate a duplicate zipfile was found; its zcs is given. if NOT s, then IF TRASHOLD directory is specified, file is put there. ELSE file is deleted. Encrypted a zipfile contained apparently encrypted file(s) when kN option was set to, or defaulted to, k0; its zcs is not given. if NOT s, then IF ENC directory is specified, file is put there. ELSE file is deleted. Excluded a lower_case "x" was found in column 17 of CSLIST.SRT or CSLIST1.SRT next to an entry with matching contents_signature, flagging that item for exclusion; the cs for the x_flagged file is given; if NOT s, then (zip)file deleted. Failed Zip failed when tested using PKUNZIP; its zcs is not given; if NOT s, then IF FZT directory is specified, file is put there. ELSE file is deleted. FailedVirus if non_zip, then it failed on test by SCAN; if a zipfile, it failed either on the first test by SCAN (e.g., as a self_extracting zipfile) or on testing of its contained files after unzipping. its cs or zcs is not given; if NOT s, then IF POTVIRUS directory is specified, file is put there. ELSE file is deleted. Obs/DfcntAV the zipfile had an obsolete or deficient Authenticity Verification stamp; its zcs is not given; if NOT s, then IF BAV directory is specified, file is put there. ELSE file is deleted. Redundant it was a zipfile containing only files which could be found on the system either in a larger zipfile or spread among more than one zipfiles; its zcs is not given; if NOT s, then IF TRASHOLD directory is specified, file is put there. ELSE file is deleted. RedundantNZ it was non_zip, and its contents_signature matched a cs in CSLIST.SRT or CSLIST1.SRT; its cs is given; if NOT s, then IF TRASHOLD directory is specified, file is put there. ELSE file is deleted. Zip in Zip it was a zipfile which contained another zipfile; its zcs is not given; depending on how /a5 or /a6 suboption e was set up, if NOT s, then IF ZIZ directory is specified, file is put there. ELSE file is deleted. n/ ttt it was a zipfile, and did not pass the novelty test; the numerical fraction is (not_cs_match)/(total_entries); its zcs is not given; depending on how the novelty_test option was set up, if NOT s, then IF QSN directory is specified, file is put there. ELSE file is deleted. Note: filenames used to illustrate the target subdirectories correspond to the \CSA installation provided in CSAMACS (see SETUP.112). If you change the name of a subdirectory, be sure to change the corresponding macro installed in FWKCS. (If you used KEEPMACS to transfer existing macros from an operating FWKCS Contents_ Signature System, your subdirectory names may be different from the above examples.) For instructions about CSAMACS, type CSAMACS /* . When the zipfile is a duplicate or is accessioned, its zipfile_contents_signature is well_defined, and the zcs is provided. In the case of a defective zipfile, its zcs is not generated. In the case of a redundant zipfile, the material would not necessarily fit on one line, so, for convenience, it is not provided in the log. m (see suboptions under o (/a5)) nN,NN(NN)(,qsn); novelty test, as described above for /a3. If subdirectory (here listed as QSN; the subdirectory can have full d:\path\dirname) is provided to receive the uploaded file (for further consideration), then, if the file does not pass this test, the file is put into that subdirectory and upload credit is given; otherwise, an uploaded file which does not pass this test is deleted and upload credit is not given. o run deletions m,w on non_AV zipfile, or p Purge non_AV file(s) from Authenticity Verification Zipfile. (when running under /a5 or /a6, "o" uses a temporary file which is automatically generated and deleted) Note: To avoid accidentally calling /a5m (which makes a .BAT file), if you use the m suboption, it can be put after the o. This does not reduce your number of available options, because o is required to enable deletions selected with m. These suboptions, j m p w, are used with o : j,Junked log for file(s) removed from within zipfile(s), with the file's contents_signature, the name of the file, as listed for it inside the zipfile, name of zipfile, AV status of file and of zipfile, any zipped path information for the file, as listed for it inside the zipfile. m delete file from non_AV zipfile if any cs_Match is found for it in CSLIST.SRT or CSLIST1.SRT. (see Note above re position of m relative to o: o before m.) subsequent action is as for w , below. p Purge non_AV files from AV zipfiles. w Wipe if w_flagged and file is non_AV: if a file is contained in a zipfile which does not have Authenticity Verification, and the contents_signature of that file matches a contents_signature in either CSLIST.SRT or CSLIST1.SRT which has a lower_case "w" in column 17, and the o option is in effect, then that file is deleted from within the zipfile, a specified age test (see option a, above) for the latest date appearing within the zipfile is reevaluated, the contents_signatures are revised, and the correction to the zipfile's size is made available for use in revising the text description (see option g, above). To mark material for wiping, you can use WIPE, which is provided in FWKCS112.ZIP. For instructions on how to use WIPE, type WIPE /* . p (see suboptions under o (/a5)) q,potvir; Quarantine, subdirectory for Potential virus; subdirectory can have full d:\path\dirname. CSAMACS sets this up as d:\CSA\POTVIRUS. r,refs (see r,refs cross_referencing under /a3) s Save rejects, else delete or sequester (various special subdirectories are for sequestering files, according to why they were rejected. s blocks that). t,trashold; optional subdirectory into which is put an uploaded zipfile rejected as redundant or duplicate (errorlevels 8, 9, 10). Subdirectory can be specified with full d:\path\dirname. v(,vdat;) call SCAN for Viruses (use external virus_data_file). This directs FWKCS to call McAfee's SCAN for testing files. For zipfiles, this test is applied to both the outer file (which might be a contaminated self_extracting file) and to the unzipped contents of the zipfile. The ability to use an external virus_data_file provides a way to quickly add a new recognition pattern, for example, for a recently analyzed "virus". This external file is prepared for use with McAfee's SCAN (remember to end the last line with a carriage_return line_feed). See v1, below. v1,telsysop; d:\path\filename for warning about "virus". If v (above) is not specified, then this option activates v without any external virus_data_file. w (see suboptions under o (/a5)) --- ELSE (if not upload), for file integrity test and/or contents_signature(s). This carries out the not_upload test function under PCBTEST.BAT. This function is called under PCBoard when a user requests TEST. If the file under consideration is a zipfile, it is tested for file integrity using PKUNZIP -t . Note: zipfiles which contain a combination of encrypted and non_encrypted files will appear to fail on this test. You can look at the detailed message to see if this is due only to skipping encrypted file(s), or if there is some other problem which might affect your use of the zipfile. Under option y (below), their contents_signatures are still available. If the z (zipfiles only) option is set, then a test using PKUNZIP -t is forced, whether or not the file is a zipfile. In the case of a non_zip file, this produces an error message from PKUNZIP which is used in constructing a message to the user. If the z option is not used, then FWKCS examines the structure of the file, and calls for a PKUNZIP -t test only if the file is recognized as a zipfile. This does not depend on the file extension, so you are free to use extensions other than .ZIP (or no extension) on the files on your system. If the file under consideration is non_zip, it is processed to produce an FWKCS contents_signature, which is provided to the user. y list contents_signatures for all the files in a zipfile, followed by its zipfile_contents_signature. For example, these contents_signatures can then be compared with those for files already on the user's system, to avoid downloading duplicates. To run option /a5, type FWKCS /a5m which makes PCBTEST.BAT = FWKCS %1 %2 %3 /a5[cdef] If you used automatic setup or used CSAMACS to set up a \CSA area, macros [c], [d], [e], and [f] were installed to support operation of a BBS. To see those macros, type FWKCS /d . If you did not use the automatic setup or otherwise use CSAMACS, and did not use SETFWKCS.BAT -- or if you wish to change the macros -- you can choose which options you are going to use, and prepare the macros [cdef], by running FWKCS ContiguousStringOfOptionsUpTo73BytesLong /dc FWKCS ContiguousStringOfOptionsUpTo73BytesLong /dd FWKCS ContiguousStringOfOptionsUpTo73BytesLong /de FWKCS ContiguousStringOfOptionsUpTo73BytesLong /df If you wish, you can make a .BAT program to do this. Or, you can read material from single_line text files [+file] into the command line to load each macro separately (see example(s) near the beginning this .REF). When these macros are called by using [cdef] (the square brackets "[]" are literal) on the command line, they are run together to make a (contiguous) string. That gives you a total of 292 bytes to work with directly; more is available, if needed, by using macro [b] and/or by reading material from a file into the command line at time of execution. To check the macros, type FWKCS /d A macro can be replaced with another macro, using the same procedure described above. New material can be added to the end of a macro. For example, to add g1,3 to the end of the [c] macro, type FWKCS [c]g1,3 /dc The new material is capitalized before entry onto screen FWKCS /d. A macro can be deleted by providing a blank input for it. For example, to delete macro [b], type FWKCS /db To reset all of the defaults, type FWKCS /revert Or, you can run REVERT.BAT, which first tells you what it does and gives you a chance to quit, and then performs the reset for all the FWKCS defaults. (REVERT is provided in FWKCS112.ZIP. For on_line help, type REVERT /* .) The /a5 auxiliary function is also available for local use, via LUT.BAT, which provides on_line help, can synthesize a template for upload description, and provides all but the @code_ generation functions performed by /a5 under PCBTEST.BAT. If you used GO, 2_SETUP, S_SETUP, or CSAMACS to install FWKCS, a copy of LUT.BAT should be in your \CSA subdirectory. /a5 automates the evaluation and testing of uploads onto an electronic bulletin board system operating PCBoard 14.n, including preparation of messages to users, testing new (zip)files, updating the contents_signature system, keeping a log, and deleting or sequestering rejects. It is able to create a contents_signature analysis of a new (zip)file and compare that with the contents_signatures for all the files in all the available zipfiles, and with all the non_zip files, on the system. It can recognize that a new non_zip file is redundant; and recognize as redundant a zipfile made of pieces which appear as separate non_zip files, or as pieces within one or more zipfiles, on the system. It does this quickly enough to make a timely response to the user and to play a role in controlling the disposition of the incoming file. These file_comparisons carry far better resolution than is provided solely by the older 32_bit CRCs. Auxiliary Function 6 ~~~~~~~~~~~~~~~~~~~~ This is a more restricted version of /a5, which treats the input file as an upload and makes ASCII output file PASS.TXT or FAIL.TXT. It accepts only two input positions on the command line. The first one is for the file to be processed, which must be present. The second is for a text description of the uploaded file (as in the third position for /a5): that file must exist to be called by FWKCS, but can be of zero length. (Note that one can explicitly designate a description file within the g subcommand. That description file must exist to be opened by FWKCS.) /a6 is carried for back_compatibility. It does not provide any function not available under /a5 used with the x0 option. Auxiliary Function 7 ~~~~~~~~~~~~~~~~~~~~ Format: FWKCS (infile) /a7.n(flags) (outfile1 (outfile2)) To make it more convenient to use flags in column 17 of CSLIST.SRT and of CSLIST1.SRT, FWKCS has an auxiliary function to strip flags, insert a flag, select all entries which have a flag in a given set, and split a file according to whether each line has a flag in a given set. if INFILE is not specified, "standard input" is assumed. if OUTFILE is not specified, "standard output" is assumed. Because INFILE and OUTFILE are separated by the /option, it is possible to omit either without affecting the other. For example, this lets FWKCS open an output file at the end of a pipeline. 7.0 /a7.0 strip all flags: copies lines from INFILE or from redirectable standard input, replacing the flags in column 17 with blank spaces, write lines to OUTFILE or to redirectable standard output. 7.1 /a7.1flag insert (lower_case) flag. The flag can be put into the command line in upper or lower case; the flag as written into the output file in column 17 is in lower case. copies lines from INFILE or redirectable standard input, places lower_case flag into decimal column 17, writes flagged lines to OUTFILE or to redirectable standard output. 7.2 /a7.2flags "flags" is a specified set of flags. In /a7.2, flags are compared independent of case: copies lines with matching flag from INFILE or redirectable standard input to OUTFILE1 or redirectable standard output. if no specified flag, default is all flags. if no OUTFILE2, output only lines with a matching flag; else put lines without matching flag in OUTFILE2. ------------------ Suppose, for example, you had a set of commercial .EXE files which you wanted to exclude from your system. You could put just those files all together into one subdirectory, and then enter them for automatic exclusion by running this command pipeline: fwkcs /1fp *.exe | qsort | fwkcs /a7.1x | fwkcs /u Version 1.12 of FWKCS allows you to use wildcards, or a directory name, to specify the file(s) to be processed. If you wish, you could do these steps one at a time. This prepares a list of exclusions, XCSLIST.SRT, which you can also send to others: fwkcs /1fp *.exe cslist.tmp qsort cslist.tmp cslistmp.srt del cslist.tmp fwkcs cslistmp.srt /a7.1x xcslist.srt del cslistmp.srt fwkcs /u xcslist.srt You can make the XCSLIST.SRT file available to others, and they can perform the single (last) step to load it into their systems: FWKCS /u XCSLIST.SRT Similarly, if there are ads which you wish to automatically delete from non_AV zipfiles when processing uploads, you can use /a7.1w instead of /a7.1x , and thus mark their contents_ signatures for automatic Wipe (for example, calling the file WCSLIST.SRT). If FWKCS is on your PATH, you can install such a marked file on your system by typing, for example, fwkcs /u wcslist.srt from anywhere on your system from which you can call FWKCS -- the rest is handled automatically. ------------------ 7.3 FWKCS (CSLIST1.NDX CSLIST1.SRT) /a7.3(blt) zipfile.ext to process a zipfile Authenticity Verification stamp: Phil Katz provided an Authenticity Verification ("AV") system for registered users of his PKZIP(R), to help protect the integrity of the ZIPfile system. This desirable feature of PK(UN)ZIP has run into two problems. First, some people have taken to adding their AV stamp to the creative work of other people, without the true author(s)'s consent. Second, although I am not aware of anyone yet fully cracking the AV stamp system, software has been created which facilitates making up unauthorized AV stamps for malicious purposes. This auxiliary function /a7.3 is intended as a partial measure against such mischief. The ban against a specific AV stamp, is in the form of information which can be rapidly disseminated in the BBS community. In this way, one BBS which is set up using the FWKCS Contents_Signature System, Version 1.10 - 1.12, can provide "immunization" information which can be rapidly put to use protecting other BBS's which use FWKCS 1.10 - 1.12. Auxiliary function /a7.3 provides 3 options. For convenience, each of these options is available as a special program. b Ban AV_stamp. To ban a specific AV stamp from your system, take a zipfile which has that AV stamp, and do this fwkcs /a7.3b bannedav.zip to create a special line entry in CSLIST1.SRT with a lower case "b" in column 17. This function is provided in BANAV, together with additional features to simplify use. For instructions on how to use BANAV to ban a specific AV_stamp from your BBS, type BANAV /* . (BANAV is provided in FWKCS112.ZIP) When you do FWKCS /v (the revise command, described below), that information will be moved into your master cslist. This "immunization" information can (then) be collected, using /a7.2 to collect all the lines in which one of a set of specified flags is found. For example, you could collect all the appearances of b, w, or x in column 17 by first doing fwkcs /v to put all the entries in sorted form into the master list, CSLIST.SRT, and then doing fwkcs cslist.srt /a7.2bwx cswarn.srt The "immunization" information in the resulting file, CSWARN.SRT, can then be added to another FWKCS Contents_ Signature System, Version 1.10 - 1.12, by using the /u command, fwkcs /u cswarn.srt from anywhere on that system. The rest is handled automatically. l List AV_stamp as OK. This creates an entry in CSLIST1.SRT with a column 17 flag of lower_case "l", for the system to recognize that AV stamp if it encounters it again. This function is provided in LISTAV. For instructions on how to use LISTAV, type LISTAV /* . (LISTAV is provided in FWKCS112.ZIP) t Test AV_stamp, set errorlevel. This uses information already stored in CSLIST.SRT and/or CSLIST1.SRT for recognizing an AV_stamp on an AV_zipfile, and sets the errorlevel accordingly. These errorlevels can be used for controlling a .BAT program. This function is provided in TESTAV, with visual output to the screen. For instructions on how to use TESTAV, type TESTAV /* . (TESTAV is provided in FWKCS112.ZIP) Here are errorlevels /a7.3 can set, with their meanings: 5 - apparently non_zip file, when only ZIPfile(s) accepted. 7 - failed PKUNZIP(R) zipfile test. 15 - zipfile has a Banned Authenticity_Verification_stamp. 16 - zipfile did not have an Authenticity_Verification_stamp. 17 - zipfile had an AV_stamp, match not found. 18 - zipfile has an AV_stamp which is recognized, and which is not banned from the system. 23 - zipfile has an obsolete or deficient AV_stamp 101-188 - "DOS extended error" + 100 decimal; see also FWKCS.ERR. Three .BAT programs are provided, for carrying out the three /a7.3 suboptions: BANAV.BAT to make an entry in CSLIST1.SRT to ban that Authenticity Verification stamp from acceptance on the system. Usage: BANAV abusedAV.zip LISTAV.BAT to make an entry in CSLIST1.SRT for future recognition of that Authenticity Verification stamp. Usage: LISTAV AV.zip TESTAV.BAT to compare the Authenticity Verification stamp of a zipfile with CSLIST.SRT and CSLIST1.SRT, and report the results. Usage: TESTAV AV.zip 7.4 FWKCS infile /a7.4 outfile For use in reformatting contents_signature cross_reference(s) found in a reply received from a BBS in response to a remote_ inquiry Lookup, for preparing the next inquiry. This is used in Rcrosref (see LOOKUP.DOC). For the structure of a contents_ signature cross_reference line in local use, see Auxiliary Function 3, option r,refs; . Note: One way of organizing a cross_reference system is to use "nodes" to which many related files are linked. In that case, the first reply to a Lookup inquiry may include a small number of references to one or more such nodes. The first Rcrosref inquiry based on that reply may bring back a large number of cross_references from those nodes which were cited. The second Rcrosref inquiry may then bring back the names of the files (and, if they are contained in zipfiles, the names of zipfiles in which they can be found) for downloading. (The maximum number of input items for a single remote contents_signature search is 512. See Auxiliary Function 5, option i, above.) (This ends the discussion of Auxiliary Functions.) Special Column_17 Flags: ~~~~~~~~~~~~~~~~~~~~~~~~ Column 17 of CSLIST.SRT and CSLIST1.SRT is used for flags. These lower_case letters have special meanings as column 17 flags: b Banned AV_stamp. marks that line for use in recognizing a banned zipfile Authenticity Verification stamp. (for use, see suboption b for /a3, /a4, and /a5 (includes use in /a6); see auxiliary function /a7.3, above; and see BANAV, above) e Extend /vs comparison to include filename + AV_status. k Keep whole line. (up to 253 ASCII characters >=" " + carriage_return + line_feed) l keep + compare Line_text. r cross_Reference. (see discussion of suboption r in /a3) w Wipe. if a file in a non_AV zipfile has a contents_ signature which matches a cs which has this flag, and both the o and w options are active, then delete (Wipe) that file from that zipfile. (see discussion of option w in /a3 and /a5) x eXclude. to exclude from the system any file, or any zipfile which contains any file, whose contents_signature matches one marked with this flag. ------------------------- d - Default names, paths, macros (discussed further, below) To change any one of the entries, use its single hexadecimal digit on the left. For example, to specify a path D:\TIMBER\ for the uploads log, item A, type FWKCS d:\timber\fwkupld.log /da To change it back to what it is on the left (the initial default), type FWKCS /da . To put in a path C:\DOS\ for finding PKUNZIP.COM, item 7, type FWKCS c:\dos\pkunzip.com /d7 and so on. (See also macros, below.) For safety, the location of the specific copy of FWKCS which you are working with is presented at the bottom of the display. If you type FWKCS /d just before you make the change, the place on the screen where you will see your entry as you make it will be directly below a display of this specific d:\path\name.ext information, to help you be aware of which copy you will be making a change in. This can be particularly important on networks. For speed of execution, please use the /dn option to specify efficient paths to reach the programs called by FWKCS, if you can. If you have several servers on a network, you may wish to leave out that path information, provided that the programs are available on the path. In that case, FWKCS uses the path to find the programs, and may find them in different places on the different servers. There are two special cases of using /dn to suppress an operation: FWKCS has an internal scrolling capability which it can use if it does not find LIST. If you wish to use this scrolling instead of LIST, you can prevent a search for LIST by putting a "." (period) in place of that filename, by typing FWKCS . /d8 Similarly, if you do not wish to keep an uploads log, you can suppress this option by putting a period in place of item A, by typing FWKCS . /da macros: these provide a way for you to use longer command lines than are supported by DOS. Each of the macros can be a contiguous string up to 73 bytes long. They can be loaded in just the same way as the paths, etc, above are put in, using FWKCS /dn, where n designates the macro. So, you would use /db, /dc, /dd, /de, or /df. Macros can be loaded from files prepared separately, using the ability of FWKCS to accept a file as part of the command line. For example, suppose you used a word processor to prepare a file called NEWMACRO. To put the contents of that file into macro [b], you could type FWKCS [+newmacro] /db New material can be added to the end of macro [b], [c], [d], [e], or [f], so long as the total length does not exceed 73 bytes. For example, to add g1,3 to the end of the [c] macro, type FWKCS [c]g1,3 /dc The new material is capitalized before entry onto screen FWKCS /d. To write the contents of macros [cde] together to a single file DID, you could type FWKCS [cde] [-did] (see the discussion near the beginning of this .REF, on how to use the FWKCS command line.) ------------------------- e - Errorlevels + messages To see this screen, type FWKCS /e . A copy is provided, in the section on FWKCS.COM screens, below. This screen specifies the meanings for various errorlevel values which FWKCS can provide. Various of these errorlevels can be used to control, for example, a .BAT program, providing considerable versatility in applications. (see also "Customizing messages to users", under /a5 option h, above.) The statements of meanings for various of the errorlevels are themselves character_strings, which are used in creating messages delivered by FWKCS under the various circumstances to which they refer. Thus, they serve two purposes. ------------------------- f - Find if contents_signature(s) in CSLIST.SRT match the input, set errorlevel. If you do not give a specific CSLIST and index to use, then FWKCS uses both of the lists and both of the index files, as listed in entries 1 - 4 on the defaults (/d) screen. FWKCS is able to carry 16,000 different matches from files contained in a single zipfile, regardless of which of the two lists each of the entries was found in. This f option provides a way to incorporate FWKCS into .BAT or other programs, to provide greater flexibility as to the response which the system makes under the specific conditions reported via the errorlevel. The f command does not gather material into a general matching_report; its output is via the errorlevel setting. See also f,g,u format and f,g suboptions, below. g - Get all contents_signature(s) in CSLIST.SRT which match the input, set errorlevel. This provides the same errorlevel information as f to control .BAT or other execution. If you do not give a specific CSLIST and index to use, then FWKCS uses both of the lists and both of the index files, as listed in entries 1 - 4 on the defaults (/d) screen. g gathers (sometimes a lot of) matches, delivered either to a file named in the command line after the options, or via the 'standard output'. This output can be used for a variety of purposes, such as sending reports to users, or preparing reports, or looking for waste, etc. The format of the report of matches found, is the same as the one used in auxiliary function /a2, which creates a file called MATCHING.RPT (you can choose to keep or to delete that report when you use /a2). See also f,g,u format and f,g suboptions, below. f,g,u format: The contents_signature format used in FWKCS is a 16_character structure composed of a 32_bit CRC in 8 hexadecimal characters concatenated with an 8_hexadecimal_character expression of the uncompressed file_length, with the leading zeros of the file_length blanked (see FWKCS112.DOC). All of the hexadecimal letters are capitalized. For convenience, FWKCS Ver. 1.12 will accept as input, for /f, /g, and /u, a contents_signature expressed in upper and/or lower case, and reformat it for further internal use. IBM/MS DOS permits use of replaceable parameters in a .BAT program. In order to permit a contents_signature to be inserted in this way, FWKCS Ver. 1.12 will also accept a contents_signature in which all the blank spaces are filled in with zeros, forming a single, 16_character, hexadecimal string, upper and/or lower case. For example, GET.BAT (provided in FWKCS112.ZIP) can be used to get, to your screen or into an output file, all the contents_ signatures on the system which match the contents_signature which you enter, or which match the contents_signature(s) of a plain file or of all the files in a zipfile, including the zipfile_contents_signature. To use this to find matches to cs= 14ABCDEF 3EB2 and send them to your screen, you would type get 14abcdef00003eb2 To send the same material to the file CSFOUND, you would type get 14abcdef00003eb2 csfound If you look at the output, you can see that the letters are capitalized and that the leading zeros were removed from the uncompressed file length. For on_line help in using GET.BAT, type GET /* . f,g suboptions: j,junked;mnN,NN(NN);o,oust;pr,refs;w j,junked keep a log of what is proposed for deletion. (see j under option o in /a5) m collect for deletion from a non_AV zipfile every file in that zipfile for which a contents_signature Match was found in CSLIST.SRT or CSLIST1.SRT. nN,NN(NN) Novelty test, see discussion under /a3, above. o,oust list of names (with zipped path information, if any) as they appear inside the zipfile, to use with PKZIP zipfile.zip -d @oust to carry out the deletions. The o option enables the deletion options. The resulting oust file can have full d:\path\filename.ext. If there are no entries to list in the oust file, then the oust file is deleted. A calling program, for example, a .BAT program, can test for the existence of the oust file, and branch accordingly. This provides a control capability separate from the errorlevel. p include in the oust list every non_AV file found within an AV_zipfile. r,refs cross_Referencing, see discussion under /a3, above. w collect for deletion all files found in non_AV file for which a matching contents_signature in CSLIST.SRT or CSLIST1.SRT was marked in decimal_column 17 with a lower_case "w". (See discussion of w under /a3, and of w under o in /a5, above. For instructions on how to mark material for Wipe, type WIPE /* . (WIPE is provided in FWKCS112.ZIP.) These are the same as discussed for /a3. They provide the ability to remove unwanted material from a zipfile, and to collect cross_references against the entire system. ------------------------- h - Help (or FWKCS sans option; or FWKCS /* ); see also /a , /d , /e , /l, /1h. i - make Index CSLIST.NDX for use with CSLIST.SRT. The reason why the program can search for matches as quickly as it does is because it does not have to search sequentially. It uses information properties of a contents_signature to find where to look in the files, and the index_files are generated according to that system. l - License and dedication. Reformatted (see the /l screen): - - - - - - - - - - - - FWKCS(TM) Ver 1.12: Provided that no fee is charged for the use, copying, or distribution; and provided that they are not modified in any way; then, EXCEPT FOR USE IN A PUBLIC, COMMERCIAL, SCHOOL, OR GOVERNMENTAL ENVIRONMENT, you are free to use and copy FWKCS(TM) and FWKDG(TM), and to distribute them solely in their original Authenticity Verification Zipfile named FWKCS112.ZIP with AV code = "# OFT466 Frederick W. Kantor (founder/information mechanics)" Frederick W. Kantor disclaims any and all warranties as to this software, whether express or implied, including by way of example and without limitation any implied warranties of merchantability, fitness for a particular purpose, functionality, data protection, or data integrity. Dedication: The file format of the cs_lists created by this program, which file format is original with the first release of this software; and the related nomenclature, "contents signature", "zipfile contents signature", "file contents signature", "cs", "zcs", and "fcs"; are all hereby dedicated to the public domain. If you have any questions or comments re FWKCS(TM) or FWKDG(TM), please send them to Fred Kantor via The Invention Factory BBS, 212-431-1194. If you find FWKCS(TM) quick and useful in private, personal use, your payment of $25 by check or money order to Fred Kantor, 523 West 112 Street, New York, NY 10025-1614, would be appreciated. Licensing available. - - - - - - - - - - - - Discussion: First, PLEASE TAKE NOTICE, that use of this software in a public, commercial, school, or governmental environment, without payment and registration, is not permitted. To all who wish to use this software in a public, commercial, school, or governmental environment: contact me in writing. To contact me in writing and/or to send payment, please use either my address as it appears above, or, if you prefer, Frederick W Kantor Founder Information Mechanics 523 W 112 St New York, NY 10025 That is my address as used by The American Physical Society and, with small variation in punctuation, elsewhere; by using the same address, you may help keep down the number of ways my name is included in numerous time_wasting listings. See REGISTER.DOC. - - - - - - - - - - - - m - Merge unique full lines from sorted files: FILE1 into FILE2, or FILE1 with FILE2 make FILE3. This uses Format 2: FWKCS file1 file2 (file3) /m This is used internally for options /u and /v, and for some auxiliary functions. For example, the two_file form, f1 into f2, lets you merge a 100k sorted file into a 9.5 Meg sorted file to produce a 9.6 Meg sorted file, all done within a total space of less than 10 Meg. This is especially convenient if you wish to run the whole system up in memory, e.g., using a ram_disk. Another example: if you are writing a program, with 16 portions which are kept in a zipfile, and revise one of the parts, only that modified file is added to the zipfile by the PKZIP -f command. When you accession that revised zipfile into the contents_signature system, only two lines finally get added: one carrying the new contents_signature for that modified file, and one carrying the new zipfile_contents_signature for the whole zipfile as a collection. This may take two steps: if you have used the /v option between the two entries, then final consolidation would not take place until you ran the /v option again, for reasons which are discussed below. (The probability that you would only add 1 line would typically be rather small: the probability of accidental match between two contents_ signatures is typically less than one part per trillion. The statistical resolution provided by the contents_signatures is better than that of the older 32_bit CRCs. See FWKCS112.DOC) Another example: in S_NEWCSL.BAT and S_REVCSL.BAT, a sorted list of selected d:\path\file entries is merged with NUL to produce an output file which contains only unique lines. This provides greater freedom in designing string searches. (See SETUP.112.) ------------------------- r /1r - Redirect screen a, d, e, h, l, 1h to "standard output". e.g., FWKCS/rd >filename, FWKCS/1rh >>filename. This provides a convenient way to redirect the screen messages into a file. The default is to send them to the "standard error output", which usually means that they are sent to the screen. The reason for making the default go to the screen rather than be redirected to a file was to help keep the output file, or output pipeline, relatively clean. This is convenient for documentation. For example, before and after you use the /dn option to change the defaults, you can send a copy of the screen to a file or to a printer, to make a record for future reference. For example, FWKCS /rd >before FWKCS s:\logs\fwkupld.log /da FWKCS /rd >after ------------------------- t,nnnn.nn decimal 0 - 9999.99 seconds of retry (networks) This is related to network systems, to provide some control of how long the program will retry. If, after retrying for that time, it still does not succeed, then typically FWKCS will prepare an error message which is sent to the screen and to FWKCS.ERR in the default directory. Depending on the command being executed, FWKCS will either continue to process another file, or will stop. Different network protocols have different "collision handling" procedures, and it is possible that certain network protocols may preclude such repeated tries. ------------------------- u Update FWKCS (CSLIST1.NDX CSLIST1.SRT) /u (<) CSNEW.SRT , 1 2 can create full (). In executing this command, FWKCS can accept a variety of contents_signature formats, and reformat them for further internal use (see discussion of f,g,u format, above). This accepts a single_line, or sorted list, of contents_ signature(s). If a filename is provided for each of the two files at the first two command_line locations before the /u, then the left one (1) is used as the index and the right one (2) is used to hold the list of contents_signature(s). If the files named at locations 1 and 2 do not exist, they are created. If there is no filename given on the command line between FWKCS and the /u, then FWKCS adds the input contents_signature(s) to the default sorted contents_signatures file specified in entry 4 on the FWKCS /d screen, and makes a new index for that file, as specified in entry 3 on the FWKCS /d screen. If the default file and its index are missing, it creates them. For example, this is used in various setup procedures to create CSLIST1.SRT and CSLIST1.NDX. (See SETUP.112.) up restricts Update with tests as in vp, below. For example, /up is used in Rcrosref.bat, to restrict the remote cross_reference inquiry to unique contents_ signatures. ------------------------- v - reVise default files #1-4; vp add only unique cs(+flag+ref)'s with Path; vs Sans path. Revise default files #1-4, consolidate into masterlist, restart CSLIST1. This makes use of the default filenames for the two contents_signatures files and their respective indexes. This system accomodates rapid updating by maintaining two sorted files of contents_signatures. One of them is used as a master_file, and is updated only occasionally. The other is kept conveniently short, so that it can be quickly added to and re_indexed. Redundant lines are dropped on the transfer of lines from the rapid_update file to the master_file. The program is able to keep track concurrently of whether a match was found for each of 16000 (sixteen thousand) contents_signatures from the zipfile being evaluated, regardless of which of the two lists the match was found in. Thus, a zipfile containing no more than 16000 files, some of which are in one list and the rest in the other, can be recognized as redundant. vp add only unique cs(+flag+ref)'s with Path; This accepts inputs from two possibly large files, one having the d:\path\filename given in the default_screen entry 2 (the location for CSLIST.SRT), the other having the d:\path\filename given in the default_screen entry 4 (the location for CSLIST1.SRT) (see below). In practice, one renames the old CSLIST.SRT as CSLIST1.SRT. The /vp option makes a new, combined CSLIST.SRT, by scanning both CSLIST.SRT and CSLIST1.SRT; lines in CSLIST1.SRT which start with contents_signatures (+flag etc. as noted below) not found in CSLIST.SRT, are transferred into CSLIST.SRT complete with their paths. It then makes an index CSLIST.NDX for CSLIST.SRT, and restarts CSLIST1.SRT and CSLIST1.NDX. If a flag is found in decimal column 17, the comparison is made according to the requirements for that flag: For comparison includes b entire line e contents_signature + flag + filename k entire line l entire line r contents_signature + flag + 4_byte_note + contents_signature w contents_signature + flag x contents_signature + flag vs Sans path. This is like vp, but, except in case of flags b, l, and k, it leaves off the d:\path part of the line when it copies it into the new CSLIST.SRT. This is intended to provide the ability to carry forward the contents_signatures of files which have been retired from the system, e.g., because they were obsolete. The presence of these cs's protects against accidental uploads of the older files being accepted again as if they were new. Because the files would in fact not be present, there would seem relatively little reason to keep the d:\path as part of the new cslist. The absence of that part of the line then provides an easy way to identify which lines they are: those lines do not have the ":" which separates the drive letter from the path. Thus, a simple search, using the IBM/MS DOS FIND function, FIND /v ":" < CSLIST.SRT > CARRIED , would collect all of those entries into the file CARRIED. It is worth noting that the only part of CSLIST.SRT which cannot be reconstructed by scanning the system, is the part which would be copied into CARRIED by this command line. If you are pressed for time or cost concerning backing up your system on a regular basis, this might help. If you wish to be able to use the master_list as part of checking to see that you have restored all your files, then a back_up of that master_list would appear in order. A set of procedures for a complete revision, carrying forward cs's from retired files, can be found in SETUP.112. The programs used, Z_REVCSL.BAT, A_REVCSL.BAT, and S_REVCSL.BAT, are included in FWKCS112.ZIP. From the /d screen, with labels added: FWKCS /d - display this screen, showing current defaults on right of "=". 1. C:\CS\CSLIST.NDX = C:\CS\CSLIST.NDX <---<< index for master_list 2. C:\CS\CSLIST.SRT = C:\CS\CSLIST.SRT <---<< sorted master_list 3. C:\CS\CSLIST1.NDX = C:\CS\CSLIST1.NDX <---<< index for quick_update_list 4. C:\CS\CSLIST1.SRT = C:\CS\CSLIST1.SRT <---<< sorted quick_update_list For example, you might choose to put both index files, with the quick_update_list, into a ram_disk. If you have the space, all four files can go into ram_disk. If you are using hard_disk or slower, try to avoid having the index files all the way across the radial traverse from where you keep their associated lists. And, if you are able to, "dress" your disk to make each of the files contiguous, to reduce head travel time. ------------------------- FWKCS.COM /1options ~~~~~~~~~~~~~~~~~~~ Usage: FWKCS(/1aNNNNcdefmprsvwxyz) file1 file2 /1 default is to do fpxz. Option h suppresses acdefmpsvwxyz. /1r Redirect screen to "standard output". The single "/1" with option letter(s) are all contiguous. aNNNN = if latest file in zipfile is older than NNNN months, set errorlevel=2. NNNN is decimal. This /1 option makes available, via the errorlevel, information used internally in support of various of the /a auxiliary functions discussed above. It can be used, for example, to control a .BAT program in testing the "age" of zipfiles. c = expect a Control_list; if its filename is not given, use "standard input". The Control_list is an ASCII file. It contains one entry on each line. Each entry can be a d:\path\filename.ext or can be d:\path\subdir. DOS wildcards ? and * may be used in filenames, but not in directory names. You can make a list of files in root and sub directories for drives C:-Z: by using FWKDG/l (see item 1 in UTILS.112), and feed it in as a pipelined input (see example in /1 option f, below). You can use the directory command DIR with no switch settings (using IBM/MS DOS 3.30 - 5.00), with wild-characters ? and * , to feed a pipeline, and use the e option (below) together with the c option, so as to parse the DIR output and accept it as a list of files to process. With IBM/MS DOS 5.00, you can use the /b ("bare") switch with the DIR command, to provide a list of d:\path\files to process. DOS 5.00 DIR also has the ability (/a) to select files according to their attributes. For example, this command line DIR ??xyz*.z* /a-d-h-s/b | fwkcs /1cfpxz >cslist would use the DIR /a and /b switches to provide to FWKCS a bare listing of all non_directory non_hidden non_system d:\path\filename.ext in the current directory which matched the DOS DIR identification given (in this example, with wildcards). Note: in DOS 5.00, if you use DIR with the /b switch and do not use the /s switch, it does not provide path information for each file, and does not provide a listing for whether the file is a subdirectory. FWKCS /1 is able to accept and process such listings in the current directory, because it is able to get the path information once it finds a file. But, a DOS 5.00 DIR /b listing without /s, taken of a directory other than the default directory, does not provide the d:\path needed to find and open the file(s). In that case, FWKCS returns an error message for each missed file. If you use the DIR listing with no switches (DOS 3.30 - 5.00), then the FWKCS /1 option e, below, edits the directory to put the filename.ext into proper format, and extracts the needed d:\path information from the directory_listing header. In this way, it is able to process directory listings whether or not they are from the default directory. DOS 5.00 DIR also has the ability (/s) to search all subdirectories below the one called (which does not have to be the current directory). For example, this command line DIR \*.* /a-d-h-s/b/s | fwkcs /1cfpxz >cslist would generate a list of contents_signatures for all the non_directory non_hidden non_system files on the default drive. For other ways to make selections using the DOS 5.00 DIR command, see the DOS 5.00 User's Guide and Reference. You can use the /1 option c without any pipeline or filelist. For example, you can run this command line, FWKCS /1cfpxz and FWKCS will wait for you to type in one or more [d:\][\path\]name's. Filenames can include wildcards * and ?, and directories can be explicitly named; multiple entries can be separated by ^E ; after you have keyed them in, press . d = reDate zipfile to latest file in it. This is handy when you have zipfiles respectively with dates different from the last date of the files contained, as often happens when you get programs from an electronic bulletin board. e = Edit input from control_list. This is for use together with the /1 option c, above, so that you can accept the unmodified output of the IBM/MS DOS 3.30-5.00 DIR command, with no switch settings on the DIR command. In that case, the DIR command makes filenames which may have a gap between the filename and the extension, and provides a header. This option gets the d:\path\ from the directory_listing header, and parses the file to use the d:\path\filename.ext for each non_directory file. As long as you leave the directory_listing header exactly unchanged, you can edit the directory listing with a word processor to delete entire lines. For example, there may be files which you know are kept open. Depending on how your system handles access to those open files, you may want to delete those lines before processing the remaining files (for instance, using FWKCS /1cefpxz). Note: with the increasing variety of headers for directory listings, this /1 option e has been largely replaced by the ability of FWKCS /1 to accept subdirectory names and to accept filenames with wildcards; by the use of the /1c option with the IBM/MS DOS 5.00 DIR /a/b/s switches (see /1 option c, above); and by the powerful string_search capabilities provided in S_SETUP, S_NEWCSL, and S_REVCSL (provided in FWKCS112.ZIP; see SETUP.112). f = make a "File contents signature" for (each) File in zipfile(s). This looks inside the zipfile named on the command line (wildcards * ? are permitted), or inside each file in the control list (wildcards can be used inside the control list); tests to see if the file is a zipfile (including both storage zipfiles and self-extracting zipfiles); and then uses information stored in the ZIP format to make a contents_ signature for each of the files in the zipfile; the zipfile format already provides the 32-bit CRC, so this is a relatively quick operation. The output includes the contents_signature for the file inside the zipfile (using the 32_bit CRC and the uncompressed length of that file), the name of that file, the name of the zipfile in which it was found, the drive\path where that zipfile was found, and any path information which was zipped into the zipfile with that file. For example, on a system which has no overlapping paths and no files held open, you can use /1 options c and f with FWKDG /l to get the contents_signatures for all the files in all the zipfiles on all the drives C:-Z:, by using a single pipelined command line: fwkdg /l | fwkcs /1cf >outfile You can run the /1f option on all the files in all the zipfiles (independant of their extensions) in a subdirectory, by specifying the subdirectory, and send the output to a different subdirectory to avoid trying to open your output file. For example, to do subdirectory c:\ZIPS\*.* and send the result to C:\WORK\OUTFILE, FWKCS /1f c:\zips c:\work\outfile Note that this did not require redirection, although the output could be allowed to go to the screen or could be redirected elsewhere. m = Make MULTCNT.RPT; cancels acdefpvwxyz This accepts as input a sorted contents_signature file, and makes a report of how many files had the same contents_ signature(s), and therefore probably had the same contents. An example of such an output report is shown in item 5 in EXAMPLES.112. p = make a "file contents signature" for (each) Plain file (non_zip). In this case, unlike the case of a file in a zipfile, FWKCS can't just look up the 32_bit CRC: it calculates it. This is written in assembly language, and, even with an 8088, it carries the running variables in the CPU chip. Then, it makes a contents_signature. The output contains the contents_signature; then the filename; then a designation "f cs", to mark that it is an independent file contents_signature rather than the contents_signature for a file contained in a zipfile; and the drive\path for that file. For an example of the output, see FWKCS112.DOC. r = Redirect messages to standard output. s = Scan a sorted contents_signature file for contents_signature multiple(s); cancels cdefpyz This is for use after the contents_signature file has been sorted. It looks for non_unique contents_signatures. These usually mean that there is the same contents in the two (or more) files. The probability of statistical error in doing this is very much smaller than would be provided by using a 32_bit CRC. One should bear in mind that a storage zipfile which has the same file contents as a self extracting zipfile will also have the same contents_signature; and that a zipfile which contains only one file will have the same contents_signature as the single file which it contains. This scan is written in assembly language, for speed. The input can come from the pipeline. For example, on a system which has no overlapping paths and no files held open, this single, pipelined command line: fwkdg /l | fwkcs /1cz | qsort | fwkcs /1s > zcs_dups is sufficient to find all the duplicate zipfiles on drives C:-Z:, and list them, grouped together, in ZCS_DUPS. The /1s output contains spacer lines made of hyphens, and line entries for only those input lines which have matching contents_ signature. When applied to CSLIST.SRT, this can tell you, for example, if the same file contents is present in files located in different zipfiles, or tell you that an ordinary file has the same contents as a file with a different name and date, even if that file is stored in a zipfile. (For a discussion of the statistical resolution provided in this matching process, see FWKCS112.DOC) v = if any zipfile, set errorlevel = 1 (sans cs output). This brings outside, via setting the errorlevel, part of the interpretation of files which is used in controlling internal procedures in FWKCS. For example, this function is used in testing whether any file unzipped into a subdirectory is itself a zipfile. The /1v errorlevel output can be used, for example, for controlling a .BAT program. w = set errorlevel = 1 if apparently nonzip file. The /1 default for FWKCS is to do options f, p, x, and z. To do this, the program distinguishes between zipfiles and non_zip files, independent of file extensions, and then generates the contents_signature(s) accordingly. If the w option is set as part of an explicit command string, such as FWKCS /1fpwxz file and so on, then this internal decision is made available to the calling program by setting the errorlevel. ------------------------- x = eXpress encryption and/or Authenticity Verification for file and zipfile. This option causes notations to be made as to the availability of AV for the file and for the zipfile, by using lower_case letters "a" and/or "v"; and labels apparently encrypted files "p" if they do not have AV, and "u" if they have AV: /1x Output Format ~~~~~~~~~~~~~~~~~ Interletter spacing for zcs and fcs chosen for visual clarity; left_16_characters output in hexadecimal, with filelength_ leading_zeros blank: Col.1 | | | | | | | || | 1 |2| 3 |4| 5 |6| 7 || 8 | | | | | | | || 16_character_cs| |cs_owner | |in, or has| |where... ||,zipped_path...| 32bitCRC32bitLen FILENAME.EXT ZIPFILE.EXT [D:][\PATH][,INNERPATH] CRCsum32LenSum32 ZIPFILE.EXT z cs [D:][\PATH] 32bitCRC32bitLen FILENAME.EXTpZIPFILE.EXT [D:][\PATH][,INNERPATH] CRCsum32LenSum32 ZIPFILE.EXT p z cs [D:][\PATH] 32bitCRC32bitLen FILENAME.EXTaZIPFILE.EXT v[D:][\PATH][,INNERPATH] CRCsum32LenSum32 ZIPFILE.EXT v z cs [D:][\PATH] 32bitCRC32bitLen FILENAME.EXTuZIPFILE.EXT v[D:][\PATH][,INNERPATH] CRCsum32LenSum32 ZIPFILE.EXT u z cs [D:][\PATH] 32bitCRC32bitLen FILENAME.EXTaZIPFILE.EXT v[D:][\PATH][,INNERPATH] 32bitCRC32bitLen FILENAME.EXTpZIPFILE.EXT v[D:][\PATH][,INNERPATH] CRCsum32LenSum32 ZIPFILE.EXT u z cs [D:][\PATH] 32bitCRC32bitLen FILENAME.EXT f cs [D:][\PATH] (The above examples are grouped to show what happens when the file entry in location 5 appears in location 3.) 1. 16 character hexadecimal contents_signature. see FWKCS112.DOC. 2. column 17, used for lower_case flags. (see Special Column_17 Flags, above.) 3. the file which has that contents_signature. It can be a file inside a zipfile, or it can be a zipfile which has one or more files in it, or it can be a non_zip file. - for "file contents signature" of a file in a zipfile, the name of the zipfile which contains that file is listed in location 5. 4. for certain types of files in location 3, location 4 (column 30) may contain a lower case "a", "p", "u", or "v": - "a" designates a file with Authenticity Verification available in a zipfile, and which does not appear to be an encrypted file. - "p" designates a file or zipfile which contains encrypted material and which does not have Authenticity Verification. If the entry in location 3 is a zipfile, then location 5 will contain "z cs". - "u" designates a file or zipfile which contains encrypted material and which contains material with Authenticity Verification: If the entry in location 5 is a filename(.ext), then the entry in location 3 is the name of a file contained in the zipfile named in location 5. In that case, the Authenticity Verification belongs to the encrypted file in location 3. If the entry in location 5 is "z cs", then the entry in location 3 is a zipfile which contains at least one encrypted file and at least one file with AV, but the encrypted file might not have AV. - "v" designates a zipfile which contains at least one AV file, and which did not appear to contain any encrypted file. 5. if the owner of the contents signature is a file in a zipfile, then the zipfile which contains that file is listed here. If the owner of the cs is not a file inside a zipfile, then the nature of the contents_signature is specified here: - "z cs" designates a "zipfile contents signature", made by summing mod 2^32 all the 32_bit CRCs for the files in the zipfile, and summing mod 2^32 their uncompressed filelengths; zipfile_name is listed in location 3 as contents_signature_ owner. - "f cs" designates "file contents signature" for plain file; filename is listed in location 3 as signature_owner; in this case, FWKCS calculates the file's 32_bit CRC. 6. If the zipfile named in column 5 contains at least one file which has Authenticity Verification, then the /1x ("eXpress the encryption and AV status") command puts a lower case "v" into location 6 (column 43). 7. This is the d:\path for reaching the file. 8. This is path information which was PKZIPped into a zipfile along with the file. The following example is available as a single program, AVONLY.BAT, provided in FWKCS112.ZIP: - - - - - - - - Example: because the AV status of the zipfile and the AV status of the file in it are separately indicated, if you scan your system to make a CSLIST.SRT using this x option (there are .BAT files in FWKCS112.ZIP to do this; see SETUP.112), you can use this command line (provided in FINDUNAV.BAT, in FWKCS112.ZIP) find "v" < cslist.srt | find /v "a" | find /v "z" | find /v "u" >toreject.tmp to collect into TOREJECT.TMP the identifications, paths, etc., of all the non_AV files which have been inserted into AV zipfiles on your entire system. For on_line help re FINDUNAV.BAT, type FINDUNAV /* . If you find any, then you can run a short .BAS program, basica toreject (or another BASIC for your system) to use the entries in TOREJECT.TMP to make TOREJECT.BAT, which calls PKZIP to delete those non_AV files from the respective AV zipfiles. This can delete files which were zipped with or without path information. (TOREJECT.BAT is provided in FWKCS112.ZIP.) Then, you can run TOREJECT to clear out such non_AV files without changing the zipfile date(s). - - - - - - - - Thanks to what Phil Katz has provided, this use of the /1x option in FWKCS provides a way to remove a certain kind of pollution from your files. After you do that, if you find only a few, you may want to test the resulting zipfiles one by one using the Y.BAT_file described in /a2, above, and look at the display to see if every remaining entry in the laundered zipfile is a duplicate of some other file. If you have a lot of files which were laundered, or prefer to handle it more nearly automatically, then you may want to run a full preliminary screening for zcs_duplicates (see section 4 in SETUP.112; see also RECIPES.112.) ------------------------- y = make "file contents signature" for both zip and plain files as if for plain files. This option is different from options f and z in that it is deliberately made to be sensitive to details which f and z are designed to avoid. It can be used for looking for exact duplicate (zip)files: unlike z, the contents_signature made using option y on a zipfile depends on compression, names, dates, and order of files in the zipfile, and on comment(s). In this case, FWKCS reads every byte in the zipfile and calculates its 32_bit CRC, rather than look up CRC value(s) in the zipfile. Options p and y can be used in looking for change, e.g., in monitoring equipment performance, etc. z = make a "Zipfile contents signature" for (each) Zipfile. This is done by reading the 32_bit CRC's for all the files in the zipfile, adding them together mod 2^32, reading all the uncompressed_file_lengths listed in the zipfile, adding them together mod 2^32, and then putting the two pieces together to make a special zipfile_contents_signature (zcs). This zcs does not depend on the names, dates, compression ratios, order of appearance, zipped paths, nor comments, of files appearing in the zipfile, nor on the zipfile's name, date, nor zipfile comment. It is a powerful tool for finding duplication. This has special features which are discussed in FWKCS112.DOC. The output line for each zipfile is like that for the option /1p, except that the line is marked with a "z cs" to indicate a zipfile_contents_signature (see /1x output formats, above). ------------------------- 4. FWKCS screens ~~~~~~~~~~~~~~~~~~~~ FWKCS provides six (sets of) help and information screens, which can be generated using these commands: fwkcs(/h) the default with no command_tail input is the /h help screen. /* is also accepted as a request for on_line help. fwkcs/a auxiliary functions, including making .BATs to call some, and providing a list of credits. fwkcs/d default names, paths, macros, and how to change them. fwkcs/e errorlevels and messages. Various of these messages are used by fwkcs in preparing text output. fwkcs/1h subset of functions for generating and manipulating contents_signatures, with some special functions. fwkcs/l license and dedication. Here is roughly what the screen displays may look like: this command line: fwkcs/h may give approximately this screen: FWKCS(TM) Contents_Signature System, Ver. 1.12. 1991 Jul 18. FWKCS/h for help. (C)Copyright Frederick W. Kantor 1989-1991. All rights reserved. Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS ........................ C A V E A T O P E R A T O R ........................ Format 1: FWKCS (CSLIST.NDX CSLIST.SRT) /option ((<)in ((>)out)) where CSLIST.NDX is index for CSLIST.SRT sorted contents_signature_list. 2: FWKCS f1 f2 (f3) /m |Format 1,2,3 grammar: /(r)(t)other_options. 3: FWKCS (files)([b])([+file1])[c](options)([def])([+file2]) (files) ([-file3]) where "[]" are literal square brackets; [b][c][d][e][f] are macros (see /d); [+file] includes file's contents (/d macros OK) in command line; [-file] prevents execution, writes (expanded) command line to file. Options: (upper or lower case OK) | Error messages are logged to FWKCS.ERR. a Auxiliary functions. d Default names/paths/macros. e Errorlevels + messages. f Find if cs_match. g Get cs_match(es). h (*) Help (see also /a,/d,/e,/l,/1h). f,g subopts: j,Junked;mnN,NN(NN);o,Oust;pr,Refs;w. (see FWKCS112.REF) i make Index CSLIST.NDX for use with CSLIST.SRT. (i m u v may take a while) m Merge unique full lines from sorted files: f1 into f2, or f1 f2 make f3. r Redirect screen a,d,e,h,l to "standard output"; e.g., FWKCS/rd >filename. tNNNN.NN 0-9999.99 seconds of retry (networks). l License and dedication. u Update FWKCS (CSLIST1.NDX CSLIST1.SRT) /u CSNEW.SRT , can make (); up see vp. v reVise /d files #1-4; vp add unique cs(+flag+ref)'s with Path; vs Sans path. If you find FWKCS(TM) quick and useful in private, personal use, your payment of $25 by check or money order to Fred Kantor, 523 West 112 Street, New York, NY 10025-1614, would be appreciated. Licensing available. this command line: fwkcs/a may give approximately this sequence of screens, with pause between them indicated below as "(pause)": FWKCS(TM) Contents_Signature System, Ver. 1.12. 1991 Jul 18. FWKCS/h for help. (C)Copyright Frederick W. Kantor 1989-1991. All rights reserved. Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS ........................ C A V E A T O P E R A T O R ........................ Auxiliary Functions, FWKCS ... /an(...) ..., "n" at left of item, below. AF2-6,7.3 use /d #1-4. For AF2-4, *? OK. Mem swap #();u,tmp; see FWKCS112.REF. 1: FWKCS filelist /a1(.N)#(); (>)nonmatch ; to find novel: 1 filenames. 2 cs's. may use local DIRGUIDE.TXT(TM) (from FWKDG) and call QSORT.| 3 both tests. 2:{FWKCS (d:)(\path\)file(.ext)} /a2#();du,tmp; ;to make contents_signature(s), get matching, add new signatures, make new index, make report; d reDate zipfile to latest file in it; may call QSORT, LIST. format: FWKCS /a2m - make YT.BAT = " FWKCS %1 /a2 " to call #2. 3:{} /a3#();aNNNNbc(,zcomb;)dj,junkd;nN,NN(NN);o,oust;mpr,refs;su,tmp;v(,vdat;) v1,tsys;w |options contiguous | to accession files. see FWKCS112.REF format: FWKCS /a3m - make ACCESN.BAT = " FWKCS %1 /a3[b] " to call #3. 4:{} /a4#();aNNNNbc(,zcomb;)df,free;j,jnkd;nN,NN(NN);o,oust;mpr,refs;su,tmp;v() v1,tsys;wz | like #3, but PKUNZIPs|f,free; subdir dflt=##CELLAR|z ZIPs_only. format: FWKCS /a4m - make ACCZIP.BAT = " FWKCS %1 /a4[b] " to call #4. 5: FWKCS file (UPLOAD (UPDESC.nnn)) /a5(opts) (out) ; test + accession; opts: #(lsuvz);aNNNN(,(2),old;)b((2),bav;)c(,zcom;)de((2),ziz;)f,free;gN,N(,...;) h,msg;ij,junk;kNke,enc;kf,fzt;l,uplog;nN,NN(NN)(,qsn);ompq,potvirus;r,refs; st,trashold;u,tmp;v(,vdat;)v1,telsysop;wx(0)yz | options contiguous, macros OK. x ascii, else makes PCBoard v14.n @codes; may call QSORT, PKZIP, PKUNZIP, SCAN. #; swap all. #lsuvz; swap if List Sort Unzip scanV Zip.| see FWKCS112.REF. (pause) (5) format: FWKCS /a5m - make PCBTEST.BAT = " FWKCS %1 %2 %3 /a5[cdef] ". 6: FWKCS file (desc) /a6() | like #5 UPLOAD with option x0 | see FWKCS112.REF. 7: FWKCS (infile) /a7.n(flags) (outfile1 (outfile2)) |re CSLIST column_17 flags if no INFILE, use "standard input". if no OUTFILE, use "standard output". /a7.0 strip all flags. /a7.1flag insert (lower_case) flag. /a7.2flags (in /a7.2, flags are compared independent of case): if no specified flag, default is all flags. if no outfile2, output only lines with a matching flag; else put lines without matching flag in outfile2. FWKCS (CSLIST1.NDX CSLIST1.SRT) /a7.3(blt) zipfile.ext ; process AV_stamp, set errorlevel: b Ban AV_stamp. l List AV_stamp as OK. t Test AV_stamp. FWKCS infile /a7.4 outfile ; reformats remote_inquiry cs_cross_refs for search. Special Column_17 Flags (CSLIST): b Banned AV_stamp. e Extend /vs comparison to include filename + AV_status. k Keep whole line. l keep+compare Line_text. r cross_Reference. w Wipe. x eXclude. ---- FWKDG(TM) Ver. 1.08 is copyright 1988,1989,1990 by Frederick W. Kantor. LIST Ver. 7.5i is copyright 1983-91 by Vernon D. Buerg. PCBoard v14.5a is copyright 1985-91 by Clark Development Company, Inc. PKZIP, PKUNZIP Reg.U.S.Pat. and Tm.Off., Copr.1989-1990 PKWARE Inc., Ver.1.10. QSORT Ver. 3.20 is copyright 1985,86,87,88 by Ben Baker. SCAN 7.6V80 is copyright 1989-1991 by McAfee Associates. -- please compensate properly. this command line: fwkcs/d may give approximately this screen (note that new defaults which you set can be viewed in this way, and note that the program which you would be changing is explicitly identified at the bottom of this display, by whatever name it was called, just above where your keyboard entries would be displayed on the screen): FWKCS(TM) Contents_Signature System, Ver. 1.12. 1991 Jul 18. FWKCS/h for help. FWKCS /d - display this screen, showing current defaults on right of "=". 1. C:\CS\CSLIST.NDX = C:\CS\CSLIST.NDX 2. C:\CS\CSLIST.SRT = C:\CS\CSLIST.SRT 3. C:\CS\CSLIST1.NDX = C:\CS\CSLIST1.NDX 4. C:\CS\CSLIST1.SRT = C:\CS\CSLIST1.SRT ---- for speed of execution, please set paths to reach these programs: 5. QSORT.EXE = QSORT.EXE 6. PKZIP.EXE = PKZIP.EXE 7. PKUNZIP.EXE = PKUNZIP.EXE 8. LIST.COM = LIST.COM 9. SCAN.EXE = SCAN.EXE ---- log for uploads; command_line macros B-F: cite using "[]", e.g., [c],[cd] A. FWKUPLD.LOG = FWKUPLD.LOG B. = C. = D. = E. = F. = Where "n" is single hexadecimal digit at left of item above, |Note: resulting FWKCS newentry /dn - make this the new Default for item n. |.COM file is FWKCS /dn - reset n as on left of "=". /revert - reset all. |uncompressed. Before using /dn, please note: this copy of FWKCS you would be changing is at C:\DOS\FWKCS.COM this command line: fwkcs/e may give approximately this sequence of screens, with pauses between them indicated below as "(pause)" (these errorlevels with these meanings are available externally, for example, for controlling .BAT programs): FWKCS(TM) Contents_Signature System, Ver. 1.12. 1991 Jul 18. FWKCS/h for help. Options /a2-6,/a7.3,/f,/g set errorlevel; some errorlevels + messages are: 0 - did not find any contents_signature match. 1 - found contents_signature_match for only some of the input files. 3 - file(s) would have been listed for deletion were this not an AV_zipfile. 4 - file(s) for deletion appear in deletion list... 5 - apparently non_zip file, when only zipfile(s) accepted. 6 - found matching contents_signature(s) with exclusion_flag. 7 - failed PKUNZIP(R) zipfile test. 8 - a contents_signature match was found for every contained file, but not for zipfile_contents_signature; the contents may have appeared in a larger prior zipfile, or been spread among more than one prior (zip)files. 9 - a contents_signature match was found for each input file; the input file(s) might be not in zipfile format. 10 - a contents_signature match was found for every file, including zipfile_contents_signature; this material probably all appeared in a prior (zip)file. 11 - the file appears more than months old. 12 - a zipfile was found inside this zipfile. 13 - this (zip)file may contain a "virus". 14 - a question as to sufficient novelty has arisen: of ..... entries, found matching contents_signature(s) for all but ..... . 15 - zipfile has a Banned Authenticity_Verification_stamp. 16 - /a7.3 zipfile sans AV_stamp. (pause) 17 - /a7.3 zipfile with AV_stamp, match not found. 18 - /a7.3 zipfile with OK AV_stamp. 19 - re your request for a contents_signature search: 20 - the contents_signature list was defective as to format. 21 - too many entries: maximum per search is 512 entries. 22 - zipfile contains apparently_encrypted file(s). 23 - zipfile has an obsolete or deficient AV_stamp. 99 - help_screen provides further information. 101-188 - "DOS extended error" + 100 decimal; see also FWKCS.ERR. this command line: fwkcs/1h may give approximately this screen: FWKCS(TM) Contents_Signature System, Ver. 1.12. 1991 Jul 18. FWKCS/h for help. (C)Copyright Frederick W. Kantor 1989-1991. All rights reserved. Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS ........................ C A V E A T O P E R A T O R ........................ For: files in Phil Katz's ZIP format, both storage and self_extracting; and : plain files. Usage: FWKCS(/1aNNNNcdefmprsvwxyz) file1 file2 (/1 default is to do fpxz). The single "/1" with option letter(s) are all contiguous. h (*) = Help, suppresses acdefmpsvwxyz. r = Redirect screen to "standard output". c = expect a Control_list; if its filename is not given, use "standard input". d = reDate zipfile to latest file in it. | if NOT c, * ? OK in naming file1. e = Edit input from control_list. | * ? OK inside Control_list. f = make a "File contents signature" for (each) File in zipfile(s). p = make a "file contents signature" for (each) Plain file (non_zip). x = eXpress encryption and/or Authenticity Verification for file and zipfile. z = make a "Zipfile contents signature" for (each) Zipfile. Special /1 functions: aNNNN = if latest file in zipfile is older than NNNN months, set errorlevel=2. m = Make MULTCNT.RPT re multiplicity : s,m null acdefpvwxyz. s = Scan sorted cslist for multiple(s): s,m may be used together. v = if any zip, set errorlevel=1 (sans cs output). w = if apparently nonzip file, set errorlevel=1 (allows cs text output). y = make "file contents signature" for zip or plain file as if plain file(s). this command line: fwkcs/l may give approximately this screen: FWKCS(TM) Contents_Signature System, Ver. 1.12. 1991 Jul 18. FWKCS/h for help. (C)Copyright Frederick W. Kantor 1989-1991. All rights reserved. Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS ........................ C A V E A T O P E R A T O R ........................ FWKCS(TM) Ver 1.12: Provided that no fee is charged for the use, copying, or distribution; and provided that they are not modified in any way; then, EXCEPT FOR USE IN A PUBLIC, COMMERCIAL, SCHOOL, OR GOVERNMENTAL ENVIRONMENT, you are free to use and copy FWKCS(TM) and FWKDG(TM), and to distribute them solely in their original Authenticity Verification Zipfile named FWKCS112.ZIP with AV code = "# OFT466 Frederick W. Kantor (founder/information mechanics)" Frederick W. Kantor disclaims any and all warranties as to this software, whether express or implied, including by way of example and without limitation any implied warranties of merchantability, fitness for a particular purpose, functionality, data protection, or data integrity. Dedication: The file format of the cs_lists created by this program, which file format is original with the first release of this software; and the related nomenclature, "contents signature", "zipfile contents signature", "file contents signature", "cs", "zcs", and "fcs"; are all hereby dedicated to the public domain. If you have any questions or comments re FWKCS(TM) or FWKDG(TM), please send them to Fred Kantor via The Invention Factory BBS, 212-431-1194. If you find FWKCS(TM) quick and useful in private, personal use, your payment of $25 by check or money order to Fred Kantor, 523 West 112 Street, New York, NY 10025-1614, would be appreciated. Licensing available.