FWKCS(TM) Contents_Signature System, Ver. 1.12, 1991 Jul 18. (C)Copyright Frederick W. Kantor 1988-1991. All rights reserved. Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS ........................ C A V E A T O P E R A T O R ........................ Selected FWKCS_related UTILITIES: (See also SETUP.112, EXAMPLES.112, FWKCS112.REF) 1. FWKDG.COM options 1.1 FWKDG screen 2. FWKC17D.BAS "column 17 delete" : to delete designated files 3. Cleaning up text directories FWKTDR19.BAS DSA.COM 4. Locally processing batches of files as uploads FWKLOCAL.BAS 5. Programs specially about zipfile Authenticity Verification BANAV.BAT LISTAV.BAT TESTAV.BAT 6. Protecting against abuse BANAV.BAT EXCLUDE.BAT WIPE.BAT 1. FWKDG.COM options: ~~~~~~~~~~~~~~~~~~~~~ (This program is heavily used in installing the FWKCS(TM) Contents_Signature System, is used to support Auxiliary Function /a1, and also finds use in a "daily event" for convenience in finding files quickly on a large BBS.) Part of the following is adapted from the DOCs for FWKDG Ver. 1.08: FWKDG(tm) Copyright (C) 1988,89,90 by Frederick W. Kantor. All rights reserved. Version 1.08 by Frederick W. Kantor circa 1990 Feb 21. For help: FWKDG Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS C A V E A T O P E R A T O R This can be run in at least three ways: 1.01. via a menu program which calls it; 1.02. using a 'pipelined' command_line of two programs, sending the output into the final file: FWKDG [d] [d] /g | QSORT > DIRGUIDE.TXT : : :: : : : : : : :: : : : final file : : :: : : : : : :: : : to redirect the output : : :: : : : : :: : program to sort the file; your choice : : :: : : : :: this '|' is to pipeline the work to the sorting program \ : : :: : : :option g to prepare material for use in DIRGUIDE.TXT (*) : : : : : the '/' is an 'option flag'; it tells fwkdg to look for option : : : end of range of drives for FWKDG to scan (not needed for /t or /r) : start of range of drives for FWKDG to scan (not needed for /t or /r) (*) option g is for making DIRGUIDE.TXT; l is for making a list of filenames complete with drive and path; t is for reading filenames from text directories without scanning other parts of the system; r redirects the help screen to the redirectable standard output, which can be sent to a file, e.g., FWKDG/r > helped.txt; it defaults to the screen, to not have it clutter the output stream. (if you use option /r, all you get is the help screen, which gets a little hard to read if you send it though a sorting program first) I have indicated Ben Baker's copyrighted "QSORT" as an example of a sorting program. To use the command line as presented above, have both FWKDG.COM and QSORT.EXE on your path or in your default drive and directory. This way, the hard parts are done by using the programs one at a time, so as to leave room in memory. 1.03. You can call a small batch program, which uses "qsort" in a command line like case 1, and prints a moderately informative, but, hopefully, courteous, final screen. It calls the two programs "externally". To run it this way, please have FWKDG108.BAT, FWKDG.COM, and QSORT.EXE on your path or in your default drive and directory, and please have FWKDG108.SCR in your default drive and directory. To start, FWKDG108 [d] [d] , where the two d's are the FIRST_DRIVE and LAST_DRIVE to scan; this runs as a batch program; the hard parts are done by using two external programs. Here is a full listing of FWKDG108.BAT: -------------------------------------------- @echo off cls echo - For use with FWKCS(TM) Contents_Signature System, Version 1.12. echo - Copyright (C) 1988-91 by Frederick W. Kantor. All rights reserved. echo - echo - FWKDG.BAT to make DIRGUIDE.TXT(TM). echo - echo - For use in same subdirectory with PCBoard text directories, echo - on (a set of) drive(s) which do not have any duplicate paths. echo - echo - Usage: echo - echo - FWKDG108 d d "" echo - ^ ^last drive (single letter) echo - ^ echo - ^ first drive (single letter) echo - echo - if only one letter is listed, then only that one drive is included echo - echo - default is to do drives c: - z: For more information, see UTILS.112. echo - echo - To quit, press Ctrl+C, then y echo - pause cls fwkdg %1 %2 /g|qsort >dirguide.txt cls type fwkdg108.scr -------------------------------------------- The first of these external programs is FWKDG.COM. It presents this screen: 1.1 FWKDG screen: ~~~~~~~~~~~~~~~~~~~~~ " FWKDG(tm) Copyright (C) 1988,89,90 by Frederick W. Kantor. All rights reserved. Version 1.08 by Frederick W. Kantor circa 1990 Feb 21. For help: FWKDG Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS C A V E A T O P E R A T O R Format: FWKDG [d] [d] /option initial drive ^ ^ final drive (each is a single letter; default: C Z) if only one drive is designated, then only that drive is scanned. Options: (using only one; upper or lower case OK) g make source-file which when sorted becomes DIRGUIDE.TXT(tm) l list files in this output format: d:\path\filename.ext t read only from text directories for making DIRSONLY NOTE: if not g, l, or t option, then you get this screen WITHOUT a full run. r send this help screen to the redirectable "standard output" Input: if option g or l, then read from drives as specified above, scanning all their root_ and sub_directories; if option g, then also read from the text directories; if option t, then read only from the text directories; text directories: DIR0 - DIR999 on default drive[\path\] (please see docs) Output is via the "standard output", which is redirectable. " In processing the /g option, FWKDG.COM first searches the drives for files, and delivers its findings via the "standard output", as indicated on the screen illustrated above. Then, FWKDG.COM searches through text directories DIR0 - DIR999; this is the text directory format which I designed it to read: filename.ext ............X-XX-XX[ . . . ] ^location 00 ^location 019 (hexadecimal) The FWKDG.COM program looks for X-XX-XX in each text line, where the 'X' means that the character in that space is > ASCII 020 (decimal 32) (the 'blank space' ' ') and is not '-'; starting at location 019 from the start of the text line, where the first location is labeled zero (decimal 25 if the first location is labeled position 1). If it finds that, then it takes the left 0D characters from the text directory line, appends another blank space, and then appends the directory name. These lines created from the respective text directory(s) are appended to the end of the sequence of lines delivered to the 'standard output'. Because an extra blank space is inserted before the text directory's name, sorting this output puts each filename.ext's directory entry above its d:\path\ entry. Output format after sorting: filename.ext DIRnnn filename.ext d:\path\ . . The second program called by the batch program is for sorting. The name for the external sorting program is set on the command line or in the .BAT file, and is presently set as QSORT. QSORT is NOT included with this program: you may obtain it from the Invention Factory electronic bulletin board, noted above. The name of each external_program's author, respectively, is displayed on the screen during execution of that external program. Please remember to provide proper reward and thanks to each author, respectively (or, in the case where you use a different sorting program, to the author of FWKDG (that's me) and to whichever author whose sorting program you use). 2. FWKC17D.BAS "column 17 delete" : to delete designated files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is a special-purpose program, intended for use with a marked version of the special file produced by FWKCS/1s when that program finds redundant signature lines. This program reads a special annotated file called MULTIS, looking for a lower_case "d" in column 17 (decimal). (An example of such a specially marked copy of a MULTIS can be found in FWK-IFBB.ZIP from and on The Invention Factory BBS.) If it finds the "d", then it rearranges parts of the line to put together the drive\path\filename and then deletes that file at that location. It makes a copy of the line it was working on, but without the "d", and appends that edited line to a file, DELETED.LOG. (there's an example from IFBB in FWK-IFBB.ZIP) For an example of how to use it, see 4.2 in SETUP.112. 3. Cleaning up text directories ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After you have removed hundreds of duplicate files from your disc, what do you do with the entries in the text directories? When Michael Sussell converted The Invention Factory BBS to ZIP format, and ran an early form of the FWKCS contents_signature system, he had that problem. I wrote a program in BASIC to help with that chore. I do not know how hard or easy it will be for other people to use it. 3.1 FWKTDR19.BAS ~~~~~~~~~~~~~~~~~ This is in interpretive BASIC. It is spread out and annotated, so that it serves as its own documentation. Questions may be left for me on the SmartNet SYSOPS conference or the Ilink Shareware conference. It works with text directories named DIRnnn, where nnn are decimal digits. All but the uploads directory are single_line_per_entry files in alphabetical order; the uploads directory, DIR99, is not required to be in alphabetical order. This .BAS program is supposed to be run in the same subdirectory that contains the text directories. It calls QSORT, may call DSA, and calls FWKDG. In order to work with a complicated system, it shells out to call a .BAT program named MAKEDIRG.BAT in order to make a new copy of DIRGUIDE.TXT, as its last step just before it exits to the system. That subroutine is called from line 2110; the line that does the SHELL command is 2730. If this poses a problem, you can put an apostrophe (') immediately after the line number in line 2110, to turn that line into a non-executing comment. 3.2 DSA.COM ~~~~~~~~~~~~ This is an assembly_language "pipeline filter", Directory Spacing Adjust. It reads from the "standard input" and writes to the "standard output", both of which are redirectable. It adjusts the spacing and punctuation of the filename and extension in a listing made by the DOS DIR command. For on_line help, type DSA /* . For an example of what it does, type DIR | DSA in a directory which has several entries, and see what it sends to the screen. ------------------------------------------------------------------------------- 4. Locally processing batches of files as uploads ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Suppose you have a text directory TEXTDIR of files which you have downloaded from an EBBS which is operating under PCBoard 14.n. Put all the new files in a subdirectory for processing, e.g., C:\CSA, with a subdirectory ##CELLAR under it, e.g., C:\CSA\##CELLAR. If you wish to alter the macros controlling options used locally, capture the working macros from your main copy of FWKCS, by running fwkcs [cdef] [-localmac] Any g option which you may have set up in a macro in FWKCS, is overridden by the special use of the g option here, described below. At this point, you should have FWKCS.COM on your path, a version of BASIC on your path, optionally, a LOCALMAC, FWKLOCAL.BAS, the text directory TEXTDIR, and the files you wish to process, in your default directory, a ##CELLAR (or whatever you have specified for the f,free; option if it is different from ##CELLAR) under your default directory. Then, run BASICA FWKLOCAL (or whatever version of BASIC is appropriate for your system) When your are asked for: list of filenames with descriptions = type TEXTDIR When you are asked for the format to use for the date, and whether the contents_signature is to be in UPPER_CASE or lower_case, and whether the contents_signature line is to be suppressed in graphics mode, press the key for the single digit which appears on the screen at the left of the format you select, 0...7. FWKLOCAL makes WORKBAT.BAT, and reports created WORKBAT.BAT number of file descriptions found (with files) = number of file descriptions, defective or no file = FWKLOCAL also makes a set of files which have the extension ".DSC", one for each description it found which had a corresponding file it found. If FWKLOCAL does not find a corresponding file, it appends the description to MISSING.LST. If MISSING.LST does not have any entry when all the descriptions have been processed, FWKLOCAL deletes MISSING.LST. Existence of MISSING.LST can be tested in a .BAT program and used to control branching. WORKBAT.BAT is a plain ASCII file. It tells FWKCS how to process each file with its description. Then you type WORKBAT and watch it go... When it has finished, all the files should have been processed: The files which were accepted should be in the same default directory as before. The files which were rejected should have been sent off to wherever you have specified in LOCALMAC, together with copies of their respective descriptions. Example: IGLYPOO.ZIP has description IGLYPOO.DSC A copy of the description for each rejected file remains, with the new extension ".REJ". The descriptions for files which were accepted have been revised to show date, true filesize after processing, the contents_signature, and latest date in a zipfile, in the format you specified. If you have specified processing for encrypted zipfiles, the revised description may contain a statement reporting the presence of encrypted file(s). These revised descriptions were appended to the end of a file called JOINLIST and then deleted from the default directory. You can then run UPSYSOP.BAS (which is included in FWKCS112.ZIP), to add a line which says | Uploaded by: SYSOP If you wish, you can edit line 120, and replace the word SYSOP with your name, provided that you do not distribute the modified copy. UPSYSOP.BAS also offers the options of suppressing the "Uploaded by" line in PCBoard graphics mode (using @X80), and of inserting a 'blank' line with the PCBoard format after the last line of text, to break the vertical display of descriptions and guide the reader's eye. When this is all done, you can append JOINLIST to your uploads directory. A record of what was done is kept in the uploads log specified in item A on the default screen, which you can see by typing FWKCS /d . If you wish, you can override that specification by including the l suboption in your LOCALMAC. For example, to direct the "local upload" information into a file UPLOG in the default directory, you can use your word processor to add l,uplog; to the end of LOCALMAC (be sure to include the semicolon ^ after the filename). A collection of the reports which would have been sent to a user, one report for each "uploaded" file, appears in the file PROCESS.RPT in the default directory. If you wish, you can simply make a list of the names of files to process, and use that instead of TEXTDIR. When you are all done running it, each accepted file will be listed in JOINLIST with its true size and date in the PCBoard U.S.A. format, and then (if you have set kN for processing encrypted files, the next line may be used for that purpose, and then, below that) on the next line will be the contents signature, etc., leaving you with a blank space to use for entering a description. One easy way to see what this does is to try it. FWKLOCAL orchestrates a combination which is easy to use, powerful, and carried out with the statistical resolution provided by the contents_signature system. 5. Programs specially about zipfile Authenticity Verification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ FWKCS contains special assembly_language coding for working with zipfile Authenticity Verification. Three utilities are provided, which bring these special assembly_ language features out for convenient use: BANAV to ban an abused AV_stamp from your BBS. After you run this on a zipfile which has that specific AV stamp, any zipfile uploaded to your BBS which has that same AV stamp is automatically rejected. LISTAV to list a good AV_stamp for future recognition. TESTAV to test the status of an AV_stamp. All three of these programs have built_in explanations of how to use them. To see how to use each of these three programs, type its name, then a /* , and then press . 6. Protecting against abuse ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The FWKCS(TM) Contents_Signature System is able to recognize files it has seen before, even if their names have been changed. And, it is able to single out zipfile Authenticity Verification stamps which you have told it to avoid. These recognition functions are based on contents_signatures, and are written in assembly_language. Three utilities are provided, which bring these special assembly_ language features out for convenient use: BANAV to ban an abused AV_stamp from your BBS. After you run this on a zipfile which has that specific AV stamp, any zipfile uploaded to your BBS which has that same AV stamp is automatically rejected. EXCLUDE to exclude specific files from your BBS. After you run this on a specific file, or run it on a zipfile which contains one or more files, every one of those files is automatically rejected, even if its name has been changed, and even if its inside a zipfile. If the excluded file is inside a zipfile, then that whole zipfile is rejected. WIPE to remove specific file(s) from inside an uploaded zipfile which does not have Authenticity Verification. For example, another BBS may have been putting ads into other people's zipfiles. After you use WIPE on an example of that ad, the o and w options for processing uploads work together to automatically remove that ad from uploaded non_AV zipfiles received by your BBS. (There is a p Purge option for processing uploaded files. It works with option o, to automatically purge non_AV files from inside of AV_stamped zipfiles. It recognizes that those non_AV files are to be removed, without your having to point out a specific example first.)