FWKCS(TM) Contents_Signature System, Ver. 1.12, 1991 Jul 18. (C)Copyright Frederick W. Kantor 1988-1991. All rights reserved. Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS ........................ C A V E A T O P E R A T O R ........................ SELECTED EXAMPLES ~~~~~~~~~~~~~~~~~ To the reader: These examples are intended for use with FWKCS112.REF. Additional cases may be found in SETUP.112, RECIPES.112, and UTILS.112. (The brief examples in EXAMPLES.112 about installing FWKCS are intended for illustrative purposes. The most recent procedures for installing FWKCS are provided under the GO installation program, and described in SETUP.112. Detailed recipes for certain previous cases appear in RECIPES.112.) 1. Finding files and adding new files 2. Running FWKCS with an electronic bulletin board system: PCBoard 14.n 2.1 Broader use under EBBS with plain ascii output for sending 3. Accessioning zipfiles 4. Accessioning general files 5. FWKCS/1m multiplicity report MULTCNT.RPT What are some uses? (with some working examples and .BAT files) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Suppose you have a set of 20,000 zipfiles in your system, and have a text or executable file of uncertain ancestry and name. Just type: Y filename.ext answer y when asked if you wish to see the report, and read the display on the screen. If the file is unfamiliar, it's added and indexed. Y.BAT, as furnished in FWKCS112.ZIP, provides on_line help. To see that help, type Y /* . A terse form of Y.BAT, called YT.BAT, which does not contain on_line help (for example, for use as a subroutine), can be generated directly from FWKCS.COM. To make terse_form YT.BAT, just type FWKCS /a2m . To run Y.BAT recursively, you can use "wildcards" *, ?, in specifying the file(s) to process, and just press the y or n key(s) when asked, to control whether you see a particular report. The y and n keys can be pushed ahead of time up to the DOS buffer capacity. A .BAT program can append or rename the reports so that they are kept for later. In order to keep the update response time short, occasionally (perhaps every few days or if CSLIST1.SRT gets over about 50k) run FWKCS /v to revise the allocation of information for storage and retrieval, consolidating the material into master_list CSLIST.SRT, and restarting CSLIST1.SRT, with a new index for each. Suboption: d reDate zipfile to that of latest file in zipfile. If you wish to use suboption d routinely as part of Y.BAT, change Y.BAT FWKCS %1 /a2 to FWKCS %1 /a2d 2. Suppose you have an electronic bulletin board system with more than ten thousand zipfiles, and have an ongoing problem of people accidentally uploading zipfiles which contain the same files as ones which you already have, although possibly with different names, different comments, different dates, and/or different ordering of the files in the respective zipfiles, and even made of pieces from several different zipfiles. (see also FWKLOCAL.BAS, item 6 in UTILS.112) If you have not yet installed FWKCS, then... First, configure your system so that there are no duplicate paths to the same files (large systems with networks). Then, to do a partial clean up, you could run fwkdg/l|fwkcs/1cz|qsort|fwkcs/1sm >multis - - - - - - - - - - - - - - - NOTE Note: if your system keeps one or more files open and not shared for reading, you can put a pipeline filter between fwkdg/l and fwkcs/1cz. Suppose your system keeps open files EX.1 and XCL.TOO. You could exclude these two filenames by inserting a two_stage pipeline filter, for example, fwkdg/l|find /v "EX.1"|find /v "XCL.TOO"|fwkcs/1cz|qsort|fwkcs/1sm >multis The S_SETUP.BAT setup procedure (also available via GO.BAT), and the cslist_generation program S_NEWCSL.BAT, contain the ability to exclude specified strings, making it easy to avoid such unwanted files. - - - - - - - - - - - - - - - NOTE ENDS to put all the duplicate zipfiles together in groups, in the file MULTIS; and to make a report, MULTCNT.RPT, showing number of probably duplicate zipfiles. Then, you could use a word processing program to look at the MULTIS file, and to put a lower case "d" in column 17 on each line containing a file you wished to delete. For example, here are 9 lines copied from the beginning of the IFBB MULTIS list (see discussion, below): ---------------------------------------------------------------- 014FF56D 158AC LAWN2.ZIP z cs J:\RBBS 014FF56D 158ACdLAWN200.ZIP z cs M:\RBBS ---------------------------------------------------------------- 0249E9AB 486D EQIPVIEW.ZIP z cs H:\RBBS 0249E9AB 486DdVIS.ZIP z cs K:\RBBS ---------------------------------------------------------------- 02FF6937 522E7dCHEMIC20.ZIP z cs E:\RBBS 02FF6937 522E7 CHEMICAL.ZIP z cs E:\RBBS Then, at solely your own risk, you could run basica fwkc17d (or use a different BASIC with your system) to delete all of those marked files, and to save those lines (without the "d") in a file named DELETED.LOG . For example, here are the corresponding 3 lines copied from the beginning of the IFBB DELETED.LOG (see discussion, below): 014FF56D 158AC LAWN200.ZIP z cs M:\RBBS 0249E9AB 486D VIS.ZIP z cs K:\RBBS 02FF6937 522E7 CHEMIC20.ZIP z cs E:\RBBS Then, you could choose what drive you will keep your cslist files on. For what follows here please have all the files from this AV Zipfile FWKCS112.ZIP in a single subdirectory. There are three ways to proceed: NULSTART.BAT fastest to set up, slow to use, sets up empty files to which contents_signatures can be added 2_SETUP.BAT scans any drives C:-Z:. For instructions on how to use 2_SETUP, type 2_SETUP S_SETUP.BAT scans selected drives A:-Z: and selected file extensions and/or strings. For instructions on how to use S_SETUP, type S_SETUP Note: if you are using a system which keeps one or more files open, you can use the exclusion capability of S_SETUP to exclude those filenames from the list of files to be processed. If you wish to, you can ignore the cross_redundancy in which all the pieces of a file may be present in others, and skip to use 2_SETUP or S_SETUP as set forth in SETUP.112. Or, you can use NULSTART, and then recursively accession all the files in your system one by one. This is (much) slower, but catches some of the cross_redundancy. The order in which you accession the files affects the result, and you may wish to give some thought to which kinds of files you wish to have left (e.g., large, or narrowly defined). After you have run S_SETUP (or whichever you choose), and run CHKDSK *.*, and done such file defragmentation as is appropriate, then.... If you do not wish to use the installation program GO, probably the second_easiest way to install a coordinated set of macros and their target subdirectories is to run CSAMACS, automatically called by 2_SETUP and S_SETUP. CSAMACS is provided in FWKCS112.ZIP and described in SETUP.112. Study the FWKCS/d screen, and choose paths which will quickly reach the programs listed there. If you do not use a path on FWKUPLD.LOG, then it can be kept separately in each of the multiple defaults for the respective nodes on your system; if you prefer to keep a single, central log, you can specify its path. Using the l,uplog; option, you can override this default via the command line. For multiple nodes, one can group an array of nodes to a specific log for that array, for example, by putting a different copy of PCBTEST.BAT to serve each specific node, calling respectively one of several different configuration files which are then respectively incorporated into the command line. A specific array of nodes can be set up with its own contents_signatures system, with its own lists and indexes, to serve a specific group of users, by using the settable defaults in FWKCS displayed using the /d option. If you wish to use macros different from those installed by the CSAMACS program, then... Choose the options to run your EBBS, and make a .BAT file to load them into defaults /dc, /dd, /de, and /df (if you need more space, you can use all of b,c,d,e,f, and change PCBTEST.BAT so that the square brackets "[]" include the macros in their proper sequence). If you did not run CSAMACS, then, after you have your cslists set up, type FWKCS /a5m to make PCBTEST.BAT (CSAMACS does that for you). In either case, make any needed changes in the PCBTEST.BAT macro calls "[]", and put PCBTEST.BAT where your system can call it properly (e.g., put d:\csa on your PATH). To see that it's working properly, take a piece out of each of several different zipfiles on your system, put the pieces together to make a single zipfile, and upload it. The system should catch it as being redundant. You can do this directly by typing FWKCS d:\path\filename.ext upload /a5[cdef]. The word "upload" is necessary for the file to be treated as an upload. You can read the resulting PCBFAIL.TXT file, and look in FWKUPLD.LOG. The redundant file is automatically deleted, unless there is specified a subdirectory to put it in, or the "s" suboption was used. CSAMACS sets up that subdirectory for you, called d:\CSA\TRASHOLD, and installs the necessary macro option. If you used GO, or directly used 2_SETUP or S_SETUP (both run CSAMACS during installation), just look in \CSA\TRASHOLD for that rejected file. (At this point, you may want to take a look at FWKLOCAL.BAS, item 6 in UTILS.112.) You can use a recursive DOS command line to have a multiplicity of files be treated as uploads, e.g., for %x in (*.zip) do fwkcs %x upload /a5[cdef] If you are going to use an option which changes the zipfile comment or makes any deletions from a zipfile, then you should use a two_ .BAT procedure (discussed below) rather than this single command line. This recursive upload operation would cause entries in the upload log, and, unless you have used the "s" suboption, would delete or sequester the files which were not accepted. - - - - - - - - - - - - - - Note: On some systems, you may find it more effective to make a second .BAT program which does one file at a time and is called repeatedly, along these general lines: File 1: ------- rem BAT1.BAT. rem This repeatedly calls BAT2.BAT. for %%x in (%1) do call bat2 %%x File 2: ------- @echo off rem BAT2.BAT fwkcs %1 upload /a5[cdef] if not exist %1 goto end copy %1 accepted del %1 :end ACCEPTED is a subdirectory under the default directory. If you have a move program or command, you can replace the two_line COPY and DELete with a single_line MOV %1 ACCEPTED . Either way, this serves to place the processed file, if it is still there, into a separate directory, so that the recursive call won't find it again. (When PKZIP(R) processes a zipfile, it makes a new copy and deletes the old one. The new copy could have been placed at the end of the already_existing set of files in the default directory. If it is not removed from there, then the recursive call in BAT1 would find it there. But, it would no longer be a new file, so it would be treated as a duplicate. By moving each file after it is successfully processed, one prevents this from happening. The removal is performed in BAT2, rather than in BAT1, because the recursive call in BAT1 would run all the way through before going on to another line in BAT1.) For example, to use this combination on a set of .zip files, type bat1 *.zip or specify the files to process in some other way, wildcards OK. >-->A more recent procedure, FWKLOCAL, also processes the descriptions (in PCBoard directory format) which go with each file, and is especially well suited for the batch processing of files transferred from one PCBoard bulletin board system to another. It can also be used to help prepare a PCBoard_format directory, starting with a list of files, and making a template into which descriptions can be entered. See item 6 in UTILS.112. - - - - - - - - - - - - - - If you have used default "/da" to set a path to the uploads log (see FWKCS/d), and have not overridden it with a command_line entry, then you can modify the call to FWKCS so as to create or add to a local upload log, e.g., fwkcs %1 upload /a5[cdef]l,uplog to make or add to local upload log UPLOG. Check to make sure that the punctuation is correct at the end of default [f], so that you can run it into the "l" without syntax problems. If the last entry in the last non_blank macro is a filename, subdirectory name, or other variable_length field, then it must be followed by a semicolon ";". One easy way to see the full, expanded command line, without carrying out the rest of its operation, is to intercept execution and write the expanded line into a text file, and then look at that. To do this, you could type fwkcs filename upload /a5[cdef]l,uplog [-zzz] and then type TYPE zzz Here is a partial example of what the log may look like: 1990-11-04 14:29:32.84 Duplicate 2488 J.ZIP zcs = AA99D801 BE2 1990-11-04 14:30:47.65 Redundant 11641 ASDF.ZIP 1990-11-04 14:33:16.88 Failed Zip 2488 FOOEY.ZIP 1990-11-04 14:37:41.35 Accessioned 17040 QWERTY.ZIP zcs = 00A4266D 45AF You can use the zcs or cs to find which file in your system was duplicated by or matched with the rejected upload. When you type the contents_signature, you can fill in all the blank spaces with zeros. To use an example from above, you can type echo AA99D80100000BE2|fwkcs /g The spacing is important: the single space after "echo" causes the zcs or cs to be read starting with the first character and to be correctly alligned; the five zeros or blank spaces are used as part of the 8_character hexadecimal representation of the uncompressed file_length. When they are blank (which is the way FWKCS generates and uses them), they come from blanking the leading zeros in the uncompressed_file_length. The whole structure of 16 characters, in the correct geometric relationship, is required to form the contents_signature. Starting with version 1.10, you can enter the contents_signature as a single string in which all the blank spaces are filled with zeros, and FWKCS will reformat it for you. If you have a large collection of zipfiles, you may have a file with the cs of 00000000 0 on your system. You can try this and see, by filling the blanks with zeros to make a single string of sixteen zeros, or you can use the 8 zeros, 7 spaces, and 1 zero. If you use a string of sixteen zeros, you can use GET.BAT for this. (GET.BAT is included in FWKCS112.ZIP. For instructions on how to use GET.BAT, just type GET /* .) 2.1 Suppose you have an electronic bulletin board system using software which is able to shell out to a batch program, and which provides as %1 the full d:\path\filename.ext for the zipfile to be tested, and provides as %2 the name of a file which serves as an error_flag; but does not provide the "UPLOAD" instruction used by PCBoard, and is only able to process pure ascii files created by FWKCS. Then, you can make a .BAT program something like this: ------------ if exist %2 del %2 if exist sendme.txt del sendme.txt fwkcs %1 /a6[cdef] if exist pass.txt goto good echo ERROR >%2 ren fail.txt sendme.txt goto sendit :good ren pass.txt sendme.txt :sendit (here you put your program to send SENDME.TXT ) del sendme.txt ------------ The /a6 option can perform all the tests etc. that are provided under #5 UPLOAD, but it generates PASS.TXT or FAIL.TXT, both of which are pure ascii files without any special control flags. The sample .BAT program listed above then uses the existence of the PASS.TXT file to control whether it creates an error_file. Whichever name was used for the output text file, it is renamed to SENDME.TXT, which you then send to your COM port. After that, SENDME.TXT is deleted, and the .BAT program returns control to the system. 3. Suppose you want to evaluate and possibly incorporate, e.g., fifty new zipfiles which you have downloaded from various other boards. You can put them all in their own subdirectory, go there, and type ACCESZIP *.zip The ones which pass the tests retain their .ZIP extension, and the new contents_signatures are added and indexed. Those which do not pass the various tests you have set up are renamed according to the results of the tests, unless you have used the "s" option - in which case, the corresponding ERRORLEVEL is displayed if the filename would otherwise have been changed. You can then dispose of them appropriately. This does not get automatically entered in the uploads log. To make a short form of ACCESZIP.BAT, called ACCZIP.BAT, type FWKCS /a4m Then, choose the options and load them into the macro [b]: FWKCS option_string /db GO, 2_SETUP.BAT, S_SETUP.BAT, and CSAMACS.BAT automatically install those options in macro [b]. 4. Suppose you have some files, some of which are not zipfiles, to add to your contents_signatures system. You can choose a way to call all of them, such as by isolating them in a subdirectory or by using suitably chosen "wildcards", and type, for example, ACCESION n?me*.b* If the program runs into a relabelling name_conflict, it reports it to the screen. Otherwise, with a statistical error rate per pairwise file comparison usually better that one part per trillion, those zipfiles which pass the tests retain their respective extensions. According to the options used, and following a set of relative priorities, redundant zipfiles are labeled ".RED"; duplicate zipfiles are labeled ".DUP"; redundant non_zip files are labeled ".RNZ"; too_old files are labeled ".OLD"; exclusion_flagged (zip)files are labeled ".XFL"; viruses are labeled ".VIR"; apparently encrypted are labeled ".ENC"; etc., as discussed in more detail in FWKCS112.REF. To make a short form of ACCESION.BAT, called ACCESN.BAT, type FWKCS /a3m Then, set the options in a macro, as above. GO, 2_SETUP.BAT, S_SETUP.BAT, and CSAMACS.BAT automatically install those options in macro [b]. The SYSOP (Michael Sussell) of The Invention Factory BBS did a file purge, of the kind discussed above as preliminary clean_up, processing more than 11,000 (eleven thousand) zipfiles and deleting more than 600 (six hundred) duplicate or redundant zipfiles. It helped protect users against wasting time and telephone money. The MULTIS and DELETED.LOG from that cleanup operation can be seen in FWK-IFBB.ZIP, made by and available on The Invention Factory BBS. That's part of why the IFBB uses ZIP. If you are on the IFBB, BULLETIN.02 reports their experience and policy on this. The IFBB has since grown, with a current collection of more than 30,000 zipfiles, and uses FWKCS to process its uploads automatically. I wrote some special software to help clean up the text directories after a lot of deletions were made. To help those of you who face this problem, I have included two programs, FWKTDR19.BAS and DSA.COM, in this AV_zipfile. (See section 3 and items 3.1 and 3.2 in UTILS.112.) 5. FWKCS/1m multiplicity report MULTCNT.RPT: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is a special use of FWKCS. It produces a report of how files are probably grouped by contents. This is discussed in examples, above. Here is an example of such a listing: 'matched': number of files which had the same particular contents signature 'cases' : number of times in which the set of matchups was that big matched cases 2 278 3 59 4 16 5 19 6 5 7 5 9 1 totals: signature_lines read: 3510 non_unique matches: 966 probably redundant: 583