FWKCS(TM) Contents_Signature System, Ver. 1.12. (C)Copyright Frederick W. Kantor 1988-1991. All rights reserved. Your use of these programs and procedures is at solely your own risk: PLEASE HAVE PROPER BACKUPS (Note: see SETUP.112 and FWKCS112.REF for recent procedures and commands.) Recipes for some uses of the FWKCS(TM) Contents_Signature System ---------------------------------------------------------------- This collection is based mostly on explanations and step by step recipes which have been provided to people who asked questions about FWKCS, expressed here in terms of FWKCS Ver. 1.12. Support is available for registered users, in the FWKCS conference on The Invention Factory BBS, 212 431 1194, 2400 N 8 1 (other speeds are available, up to 19,200 bits/sec; Hayes, USRobotics V.32, and CompuCom high speed modems are supported). See lines 1334-1399. Line 1. Setting up to handle only .ZIP files ........................ 41 2. Setting up for both .ZIP and .GIF files ..................... 465 3. Setting up options for automatically processing uploads ..... 782 4. How to fine-tune your options for uploads ................... 957 5. Discussion of automatic deletion ............................ 1060 6. Setup summary ............................................... 1154 7. Registration and Support .................................... 1209 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Setting up to handle .ZIP files: **ÝFred, I got the earlier versions from CDC to look at. It sounds like Ýa great way to clear some space on the system, but after looking at Ýthe docs, I'm a little apprehensive! Can you provide a "quickstart" Ýdoc file to point out the steps to take to run the program? I'm not Ýsure of just where to start, or which program to start with. Does Ýit take a looong time for the first run on existing files? ****, I can well appreciate that you are "a little apprehensive!". Here is a step by step explanation. I have broken apart the pipelined command structure so that you can see what is happening at each step, and can stop if you see any problem. As for speed, all of the FWK*.COM programs in this system are written in assembly language, and the conceptual structure which makes it work gives it rather more speed... First, I suggest you get FWKCS112.ZIP from The Invention Factory BBS (212 431 1194 where the current version is a special no_fee download), for two reasons: 1. people bring strangely damaged zipfiles to my attention, and I have been making sure that each revised version of FWKCS can handle all of them, before making it publicly available. 2. there are new features which it is more efficient for you to use from the beginning. Then, you should do a preliminary clean_up, to remove all duplicate zipfiles. A duplicate zipfile is one which contains a set of files which have the same contents as the files contained in another zipfile, on a one_to_one basis, although their names, dates, comments, compression ratios, and order may all be different. With Ver. 1.05, I added a special ability which you can use to first remove every non_AV file (such as BBS ad) which has been placed inside an Authenticity Verification zipfile anywhere on your system. If you think any such files might be on your system, then you should use this feature as your first step in the preliminary clean_up; otherwise, you might wind up with two files on your system which had not been recognized as duplicating each other because one of them had been altered by the inclusion of such an ad. Here is a sequence of steps for starting out, designed so that you can see what is happening at the intermediate steps: 1. The first step is to configure your system so that there are no duplicate paths to the same file (I don't know what your system is, and this may not be present on your system). 2. Next, make a list, FILELIST, of all the (non_directory) files on your system ("non_directory" means that files which are themselves sub_directories are not included). If they are on drives c:- z:, then you can run FWKDG /L >FILELIST If they are on a few different drives, you can add them, one drive at a time, to FILELIST. Suppose they were on drives A:, C:, and Q:. Then, you could do FWKDG A /L >FILELIST FWKDG C /L >>FILELIST FWKDG Q /L >>FILELIST where the ">" creates or truncates FILELIST, and the ">>" adds material to the existing file. 3. I will assume here that you are interested in processing only zipfiles which would be found by a directory listing for *.Z*, whether to save time or to avoid accidentally stumbling over a hidden file kept open by your system. To select these files, you can do FIND ".Z" WORKLIST where "FIND" is the IBM/MS DOS FIND filter. You are now done with FILELIST, so you can do DEL FILELIST to free some disk space. 4. You are going to be working with some files which could be relatively large: a line will be used for the contents_signature plus other information, for each file contained in each zipfile, plus one for each zipfile. To make this contents_signature_list, or "cslist", FWKZCS will read WORKLIST as a command_file, using this DOS command line: FWKCS /1cfxz WORKLIST CSLIST where the "1" tells FWKCS how to interpret the commands (see help screen FWKCS /1h) the control characters "cfxz" tell FWKCS c - expect a Command list f - make a contents_signature for each File in each zipfile x - eXpress encryption and/or Authenticity Verification for a file in a zipfile and for zipfile z - make a zipfile_contents_signature for each zipfile To give you some feel for how long this takes, on The Invention Factory BBS, with somewhat over 30,000 zipfiles, containing together over 175,000 unique files, this takes about an hour, and produces a cslist about 12_Megabytes long; execution appears to be network_limited rather than cpu_limited. 5. Having made CSLIST, you can then do a sequence of selective extractions, choosing the order of steps so as to rapidly shrink the file_size. First, find all the lines pertaining to zipfiles which have Authenticity Verification: FIND "v" CSLIST.V (The lower_case "v" must be used, because it is a literal character being searched for.) Then, find those lines which are not related to specific Authenticated files inside the zipfile: FIND /V "a" CSLIST.Z (The lower_case "a" and "u" must be used, because they are literal characters being excluded by the /V commands.) Then, find those lines which are not the zipfile_contents_signature lines: FIND /V "z" TOREJECT.TMP (The lower_case "z" must be used, because it is a literal character being excluded by the /V command.) The lines in TOREJECT.TMP should show all the non_AV files which have been inserted into any AV_zipfile which ended with .Z*, anywhere on your system. You can now delete CSLIST.V and CSLIST.Z to free disk space. 6. If TOREJECT.TMP is empty, then you can delete TOREJECT.TMP and go to step 7, below. If TOREJECT.TMP is not empty, then you can run TOREJECT.BAS, which is included in FWKCS112.ZIP: BASICA TOREJECT (or whichever version of BASIC is appropriate for your system) TOREJECT.BAS reads TOREJECT.TMP, and puts together a batch file, TOREJECT.BAT, which calls Phil Katz's PKZIP(R) to delete each and every one of those unwanted non_AV files, while keeping the date on the zipfile. You can type that batch file to your screen, to see what it will do before you run it. Then, with your system mapped the same way as it was when you made CSLIST, run TOREJECT to clean out that pollution. 7. If TOREJECT.TMP is empty, then you can use the zipfile_contents_signature lines from CSLIST, which you can extract by using the DOS FIND: FIND "z" ZCSLIST If TOREJECT.TMP is not empty, then the pollution_removal process will result in changing one or more zipfile_contents_signature(s), so the entries in CSLIST no longer correctly describe the status of your system. There are some moderately complicated things you can do to replace part of CSLIST, but it makes sense to simply delete CSLIST, and let the machinery do the work for you, by making a new list which contains only the zipfile_contents_signature lines, "ZCSLIST". To make that more restricted list, using WORKLIST, run FWKCS /1cz WORKLIST ZCSLIST 8. The next operation requires that ZCSLIST be sorted. I suggest that you use Ben Baker's QSORT, version 3.20: QSORT ZCSLIST ZCSLIST.SRT After looking to see that ZCSLIST.SRT came out OK, you can delete the unsorted ZCSLIST. 9. To find all the duplicate zipfiles on your system (that end with .Z*), now run FWKCS /1sm ZCSLIST.SRT MULTIS This creates two files: MULTIS which contains, grouped together, the duplicate zipfiles MULTCNT.RPT which tabulates the duplications found If MULTIS does not contain any files, then, if TOREJECT.TMP was empty, you can delete ZCSLIST.SRT and save CSLIST to use below in #11, and skip the rest of the preliminary clean_up. 10. If MULTIS contained files, then, using a word processor, you can put a lower_case "d" into column 17 of each line which you want to delete, leaving one line of each group unmarked. The reason why this is not automated is because you may have your own good reasons for choosing one filename or another. Examples of this are given in item 2 in EXAMPLES.112. Before you run FWKC17D.BAS, make sure that your system is mapped the same way as when you created CSLIST and ZCSLIST: FWKC17D.BAS uses the d:\path\filename.ext information from each of the lines which was marked in column 17 with a lower_case "d" to delete each of those files, respectively, and logs the result to DELETED.LOG. Then, at solely your own risk, you can run BASICA FWKC17D (or whichever version of BASIC is appropriate for your system) FWKC17D.BAS is included in FWKCS112.ZIP. A report on such a clean_up by The Invention Factory BBS can be found in FWK-IFBB.ZIP on IFBB. ======================================================================= On a large system, this preliminary clean_up may free many megabytes of file_storage space. ======================================================================= 11. Now comes setting up the automatic contents_signature system: For your first set_up: I do not recommend doing it one file at a time, unless you have a ~~ ~~~ special reason for wanting to remove zipfiles which are redundant, that is, which are made up out of pieces which can be found among the other files in the system: the results of such a screening process depend on the sequence in which you do them, and it's sssllllooooooowww. Here, I will assume that you are interested in getting it up and running relatively quickly: First, if you removed any file in the preliminary clean_up, do steps 1-4 again to make a new CSLIST. Then, find a place with plenty of room to work in, preferably at least four times as long as CSLIST, and move there. Once there, to make a sorted copy of CSLIST, run QSORT CSLIST CSLIST.SRT After looking to see that CSLIST.SRT came out OK, you can delete CSLIST to free storage_space. 12. To make life easier for me, I will assume for this discussion that you are going to set up your system using the defaults which I put into FWKCS. That means that you make a subdirectory C:\CS, with room to hold CSLIST.SRT plus at least 20% to spare. Put CSLIST.SRT into C:\CS. Change to that subdirectory, so that you are now in C:\CS. Bring FWKCS.COM with you. Then, to make an index CSLIST.NDX, run FWKCS CSLIST.NDX CSLIST.SRT /I Then, to protect the master_list, run ATTRIB +R CSLIST.* to mark it read_only. On a Novell system, this corresponds to Shareable_Read_Only. 13. To make the rapid_update list and index, run FWKCS /U This creates an empty CSLIST1.SRT, with 16388_byte index CSLIST1.NDX. 14. There are specific paths which you can insert into the defaults in FWKCS.COM. One reason for doing this is so that you can have faster access to programs which are called by FWKCS. Another reason is so that, under automatic operation of an EBBS with a multi_node system, you can send all the entries in FWKUPLD.LOG to a single location. To do this, first do FWKCS /d and look at what kinds of things you may wish to make more specific. After you make a change, it will show on the right side of the "=" sign. At the left edge of the screen is a single hexadecimal digit for each of the defaults. There are detailed instructions at the bottom of the display, explaining how to make these changes and how to make them revert back to the default which is provided on the left side of the "=" sign. You can undo all the changes, by typing FWKCS /revert . Or, if you prefer, there is a batch file included in FWKCS112.ZIP, REVERT.BAT, which will undo all the changes you make in the FWKCS.COM defaults. So, don't worry about trying it. Something you might notice at the bottom of the display: the true d:\path\filename.ext of the file you actually called is presented directly above where you would be entering your change, even if you gave the program a different name. This is to help protect against making accidental changes in files which aren't where you thought they were. It also lets you see where a file came from, when there might be more than one copy on the system. 15. After you set paths for conviently reaching the programs called by FWKCS.COM (and provided d:\path\ to make d:\path\FWKUPLD.LOG (or other name) as a target for logging upload_data from all your nodes) put FWKCS.COM and FWKDG.COM on your PATH, either by moving them to a (sub)directory which is already on your PATH or by adding C:\CS to your path. 16. Go to a place on your PATH you would like terse versions of the Auxiliary Function batch programs to be, and make them by running FWKCS /a2m (makes YT.BAT) FWKCS /a3m (makes ACCESN.BAT) FWKCS /a4m (makes ACCZIP.BAT) FWKCS /a5m (makes PCBTEST.BAT) (This PCBTEST.BAT is for operation under PCBoard Ver 14.x, and I do not offer any advice here about operation under ProDoor. My software is running on the IFBB under PCBoard Ver 14.5a.) ======================================================================== There are many options available for use with /a3, /a4, /a5, and /a6. Please see examples in this Recipes file, and see FWKCS112.REF. ======================================================================== 17. Try this: go somewhere on your system away from the c:\cs subdirectory, make a new subdirectory, put in it pieces from several different zipfiles on your system, and then zip them together into a new zipfile XXX.ZIP. Then do ACCESN XXX.ZIP It should rename the file XXX.RED, for "redundant", meaning that it found all the component parts but not the aggregate zipfile_contents_signature. Take a single file from inside a zipfile, and try ACCESN on that. It should rename it ".RNZ", meaning that it is "redundant non_zip". To carry out these operations, it had to figure out whether the file was a zipfile or not; and, when the file was not a zipfile, it had to calculate the 32_bit CRC and create the contents_signature directly. That is all done in assembly language. To see what it finds, you can rename the resulting XXX.RED XYZ.POP, and then do YT XYZ.POP and when asked if you would like to see the report, type y . This would have it do the search again, but this time make a report. After you leave the display, it asks you whether to delete the report. You can save it if you wish, and simply overwrite it later with another report; or you can delete it. 18. To finish setting up, first change to c:\cs, where you have cslist.srt, and run the DOS CHKDSK command, CHKDSK CSLIST*.* and see if any of the four files is broken into non_contiguous parts. If any of them is, then I suggest that you run a suitable file defragmentation program, such as Norton Utilities SPEEDISK or equivalent. File fragmentation can lead to substantial amounts of head travel, with resulting increase in response time. 19. As part of operation, if your system supports ramdisk, I suggest that you keep at least CSLIST1.SRT and CSLIST1.NDX on ramdisk, for speed in updating and to avoid some head travel during update. I deliberately wrote the update procedure in such way that when you update CSLIST1.SRT, you don't need any more space than it takes to hold the final result. This is intended for operation on a ramdisk. The reason why I haven't emphasized this in the DOCs is because the system where I have been doing "burn_in" before public release (the IFBB) is running under Novell which does not support ramdisk, and the DOCs are based on testing. 20. As part of your daily "event", you can run FWKCSF /v to revise CSLIST.SRT, CSLIST.NDX, CSLIST1.SRT, and CSLIST1.NDX, so that the CSLIST1 part is kept small and the response for updating the files to include new material is kept fast. 21. For operation under other than PCBoard Ver 14.x, see FWKCS112.REF and EXAMPLES.112 about auxiliary function /a6. ======================================================================= This solves a problem which faces almost the entire electronic BBS community, and does so with such speed and precision as to be able to operate unobtrusively in real time as part of the control system, while learning from its input. The probability of statistical error for a single match between these contents_signatures is typically below one part per trillion. ~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ I am not aware of any other software which does the job this does. ======================================================================= ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Setting up for both .ZIP and .GIF files: **ÝI was surprised to find that after its first run it located close Ýto 300 duplicate ZIP files. I ran the basic program that you Ýsupplied against the file MULTIS and then had PCBFILER scan for Ýmissing files. I hope you left the lower_case "d" out of one line in each of the groups of duplicate zipfiles. A group of duplicate zipfiles includes all the zipfiles which share the same zipfile_contents_signature -- you want to keep one of them. ***, with 300 Megs of GIF files and 20,000 ZIPfiles, it would seem likely that some of those GIFs are duplicates of each other, and that some of those GIFs are duplicates of files which are contained inside ZIPs. So, I'm going to break this up into three problems: 1. How to get rid of non_zipped GIF files which are duplicates of other non_zipped GIF files; 2. How to get rid of non_zipped GIF files which are duplicates of GIF files inside ZIP file(s); 3. How to make a CSLIST for the remaining ZIPs and GIFs together; The order in which you carry out these steps is important. They are numbered in sequence: first do all the 1.n steps, then do the 2.n steps, then do the 3.n steps. 1.1 First, configure your system so that there are not any cases where two paths lead to the same file (this may already not be present for your system). This is like the first step you use for making a list of ZIP files, but now you are doing it for GIF files. 1.2 If all your files are on drives C: - Z:, you can then run FWKDG /L >FILELIST to make a list of all the files on your system. If your files are on a few drives, say A:, Q:, T:, you can add them to FILELIST by doing, for example, FWKDG A /L >FILELIST FWKDG Q /L >>FILELIST FWKDG T /L >>FILELIST where ">" creates or truncates FILELIST, and ">>" then adds to an existing file. I suggest you make a .BAT program to generate FILELIST, because you're probably going to want to do it again after deleting some files. 1.3 Using the IBM/MS DOS FIND command, extract from FILELIST a list of all the uncompressed GIF files on your system that end with ".GIF", by running FIND ".GIF" WORKLIST Check to see if WORKLIST came out OK. If it did, then you can delete FILELIST to save some space. 1.4 FWKCS is able to generate the contents_signature for a file directly, by calculating the 32_bit CRC and preparing the 16_character contents_signature. It's written in assembly language, and it carries this calculation almost completely up in the CPU chip, even if you are using an 8088. You may find that the speed is network_limited rather than cpu_limited. To do this, run FWKCS /1cp WORKLIST CSGIFS where the options /1cp tell FWKCS c expect a Command_list p prepare a contents_signature for a Plain (non_zip) file Note that all the contents_signatures produced in this step are marked with a lower case "f" in the expression "f cs" (for "file_contents_signature"). 1.5 Next, we need to sort CSGIFS. Find a work area at least four times as big as CSGIFS, and move the file there. I suggest you use Ben Baker's QSORT (version 3.20), to run QSORT CSGIFS CSGIFS.SRT Check to see that CSGIFS.SRT came out OK. If it did, then you can delete the unsorted version CSGIFS to free disk space. 1.6 To find duplicate GIFs, run FWKCS /1sm CSGIFS.SRT MULTIS Where options /1sm tell FWKCS s Scan sorted cs_list for duplicate contents_signatures. m make MULTCNT.RPT This makes MULTIS containing groups of files which have the same contents; all the contents_signatures are the same within each group; these groups are separated by dashed lines MULTCNT.RPT tabulating how many of those groups were found 1.7 Then, using a word processor to process MULTIS, put a lower_case "d" in column 17 of each line that contains a file you want to delete. IMPORTANT: Be sure to leave one, and only one, line in each group without the "d" -- otherwise you would delete all of the ones which have that contents, or cause duplicates to be found later. 1.8 Make sure that your system is in the same configuration as when you made CSGIFS. Then, at solely your own risk, run BASICA FWKC17D (or whichever BASIC is appropriate for your machine) to read your annotated copy of MULTIS and delete all the unwanted files. (A copy of FWKC17D.BAS is included in FWKCS112.ZIP.) 1.9 If you have deleted any GIF file in step 1.8, then CSGIFS.SRT is no longer a correct description of the status of your system, so delete CSGIFS.SRT. 2. Next, is any uncompressed GIF a duplicate of a GIF file which is inside a ZIP file somewhere on your system? 2.1 Do steps 1.1 and 1.2, to make a new FILELIST. 2.2 Using the IBM/MS DOS FIND command, extract from FILELIST a list of all the ZIP files that end with .Z* plus all the uncompressed GIF files on your system that end with ".GIF", by running FIND ".Z" WORKLIST FIND ".GIF" >WORKLIST where the ">" creates or truncates WORKLIST, and the ">>" adds to an existing file. Check to see if WORKLIST came out OK. If it did, then you can delete FILELIST to save some space. 2.3 Make a contents_signature list for both the ZIPs and the GIFs. To do this, run FWKCS /1cfp WORKLIST CSLIST.TMP where the options /1cfp tell FWKCS c expect a Command_list f find a contents_signature for each file in each zipfile p prepare a contents_signature for a Plain (non_zip) file Note that all the contents_signatures prepared for plain files via the "p" option are marked with a lower case "f" in the expression "f cs" ("file_contents_signature"), as distinct from the cs's for files contained in zipfiles. Check to see if CSLIST.TMP came out OK. If it did, then you can delete WORKLIST to save some space. 2.4 This time, only GIF files are to be considered. So, to make the sorting easier, first use the IBM/MS DOS FIND command to extract only those files, by running FIND ".GIF" CSGIFS Check to see if CSGIFS came out OK. If it did, then you can delete CSLIST.TMP to save some space. 2.5 The CSGIFS that you now have includes GIFs which are in ZIPs and GIFs which are not in ZIPs. But the sorting step is the same as in 1.5: Find a work area at least four times as big as CSGIFS, and move the file there. I suggest you use Ben Baker's QSORT, Version 3.20, to run QSORT CSGIFS CSGIFS.SRT Check to see that CSGIFS.SRT came out OK. If it did, then you can delete the unsorted version CSGIFS to free disk space. 2.6 To find duplicate GIFs, run FWKCS /1s CSGIFS.SRT MULTIS.TMP Where option /1s tells FWKCS s Scan sorted cs_list for duplicate contents_signatures. This makes MULTIS.TMP containing groups of files which have the same contents Note that this time we are not making MULTCNT.RPT, because this is just intended as an intermediate step to make a deletion list. You can use /1sm if you would like to see MULTCNT.RPT. 2.7 In step 1.7, when you put the lower_case "d" in column 17 of each line which you wanted to delete, it was important to leave one, and only one, contents_signature line from each group. The result is that the non_zipped GIFs have no duplicates among themselves. So, for a non_zipped GIF to show up in this new MULTIS.TMP, its contents_signature matched the contents_signature of a GIF which was contained in a ZIP. But the line for a non_zipped GIF has a lower_case "f" in it, and that's how we're going to find them. To do this, using the IBM/MS DOS FIND command, run FIND "f" MULTIS (the lower_case "f" is a literal character used in the search) Check to see that MULTIS came out OK. If it did, then you can delete MULTIS.TMP to save space. 2.8 Every file in this new MULTIS is a duplicate of a file in some zipfile on your system. Your Decision: it's up to you to decide whether you want to keep the duplicate GIFs. I suggest that, before you do, think through what purpose they may serve for you, and consider such things as whether they can be extracted from a zipfile on the system by a user, whether they may waste a user's time with duplicate downloads, etc. This new ability to find duplication provides an opportunity to make some practical decisions as to how you run your BBS. If you decide not to delete the duplicate non_zipped GIFs, then you can clean up and skip to 3.1. If you decide to delete them, then, rather than write another deletion program, I suggest that you use a word processor to put a lower_case "d" in column 17 of every line in this new MULTIS to delete all of them. 2.9 To do this deletion, make sure that your system is in the same configuration as when you made CSGIFS. Then, at solely your own risk, run BASICA FWKC17D (or whichever BASIC is appropriate for your machine) to read your "d" marked copy of MULTIS and delete all the unwanted files. 3.1 To make a combined CSLIST for both non_zipped GIFs and for ZIPs, first do steps 2.1 and 2.2, to make a new WORKLIST. 3.2 Then, to make a contents_signature list for both the ZIPs and the GIFs, run FWKCS /1cfpxz WORKLIST CSLIST where the options /1cfpxz tell FWKCS c expect a Command_list f find a contents_signature for each file in each zipfile p prepare a contents_signature for a Plain (non_zip) file x eXpress encryption and/or Authenticity Verification for a file in a zipfile and for a zipfile z make a Zipfile_contents_signature for each zipfile 3.3 Find a work area at least four times as big as CSLIST, and move the file there. I suggest you use Ben Baker's QSORT Ver. 3.20, to run QSORT CSLIST CSLIST.SRT Check to see that CSLIST.SRT came out OK. If it did, then you can delete the unsorted version CSLIST to free disk space. 3.4 Make or clear a subdirectory, C:\CS, with at least enough space for CSLIST.SRT + about 1 Megabyte working space for adding new material. You can use more space as you need it, but the revision process does not require large empty disk_space. 3.5 Move CSLIST.SRT to C:\CS, and change directory to make that the default directory. 3.6 Run FWKCS CSLIST.NDX CSLIST.SRT /I to create an index for master_list CSLIST.SRT. 3.7 To protect the master_list and its index, do ATTRIB +R CSLIST.* 3.8 Make the rapid_update CSLIST1.SRT and its index CSLIST1.NDX, by running FWKCS /u ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Setting up options for automatically processing uploads: **ÝYes, age_testing, virus_testing (via McAffee's Scan) and comment Ýswapping is important. ****, First, get FWKCS112.ZIP. It's available as a no_fee download on The Invention Factory BBS. In FWKCS(TM) Ver. 1.12, the /a5 option is able to handle both zip and plain files, and treats all non_zip files as plain files. A restriction has been added for security reasons: a zipfile which contains another zipfile is not automatically released onto the board. There is a suboption that lets you set up a special subdirectory into which that upload can be put, and, if you set up such a subdirectory, you can choose whether or not immediate upload credit will be given. This is discussed further, below. After you unzip FWKCS112.ZIP, to generate the new PCBTEST.BAT, type FWKCS/a5m The resulting .BAT contains one line, "FWKCS %1 %2 %3 /a5[cdef]". The "[cdef]" incorporates four macros, which you can see by typing FWKCS /d . They start out blank, and each of the macros can be up to 73 characters long. What we will do is set up macros for the different options you want to include. In the examples below, I have used filenames related to the purpose of the respective file -- FWKCS can use whatever DOS filenames you put in those places. --- The age test is aNNNN where NNNN is (decimal) number of months for maximum acceptable age. For example, one year maximum age would be a12 --- To replace the zipfile comment with your own, the option is c,d:\path\comment; (note the semicolon after the filename), where the last line in the COMMENT file ends with the carriage_return__line_feed expected by PKZIP. --- The option for saving zip_in_zip uploads is e(2),d:\path\zipinzip; The zip_in_zip upload is put into subdirectory ZIPINZIP, which has to exist (FWKCS does not create subdirectories). To give no upload credit when you save the zip_in_zip upload, e,d:\path\zipinzip; To give immediate upload credit for zip_in_zip, e2,d:\path\zipinzip; Because upload credit is handled by PCBoard, the way immediate upload credit is given is to pass to PCBoard a file called PCBPASS.TXT. For this reason, giving immediate upload credit for zip_in_zip files has the undesirable effect that the uploaded file is listed in the directory as if it were accessible to users, when for security reasons it has been set aside for further study. I included the ability to give immediate upload credit, but I advise against using it unless you are going to monitor and promptly check all those files which were set aside. For security reasons, all uploads need to be tested for viruses etc. before being released onto the board. As a practical matter, it is highly desirable that such testing be automatic. And, the contents_ signatures have to be added in order for the screening process to catch duplicate and redundant files. I regard not giving immediate upload credit for zip_in_zip as a gentle way of reminding users to let the automatic screening do its job. --- To process zipfile uploads, FWKCS Ver. 1.12 must have a subdirectory to work in. The usual procedure is to put a subdirectory under the default directory of each node, respectively, using the same name in each case. The default name for the workarea subdirectory is ##CELLAR. If you wish to use a different name, you can specify it by f,freearea; where FREEAREA is replaced by whatever subdirectory name you choose. Note that this does not have d:\path\, so that each defaults to its own separate workarea, to avoid accidental overlap in the workareas when processing uploads on different nodes at the same time. --- To retain for review files rejected as redundant or duplicate, you can use t,d:\path\trashold; to put them in subdirectory TRASHOLD. --- To test for viruses, v(,d:\path\virdata;) where the optional part in parentheses is for use of an external virus_data_file, made using the format for McAfee's SCAN. That way, a sysop who finds a new virus can send out a message containing an entry for recognition of a new virus, and other sysops can quickly set up such an external file to help protect their systems against that virus. You can set up a special file, to which a notice will be added if a virus is detected. The option for this is v1,d:\path\telsysop; . If you wish to keep a potential_virus upload (one which failed the virus test) for further study, you can use the q option to set up a quarantine subdirectory; otherwise, the file is deleted (unless the s option is used to Save all rejects). q,d:\path\potvirus; NOTE: this virus test does not test inside various non_zip forms of compressed libraries. --- There are some options for removing unwanted material from zipfiles: the o option (for oust), which enables such deletions, the p option, which says to Purge from an Authenticity Verification zipfile every non_AV file which has been inserted into it (typically ads from bulletin boards which have handled it), the w option, which says to Wipe from a non_AV zipfile every file for which a matching contents_signature has a column_17 "w" flag, and, in case you want it, the m option, which says to delete from a non_AV zipfile every file which matches one already on your system (!). if you choose to use these deletion options, you may want to keep track of what has been deleted from uploaded zipfiles. To do that, you can use the j option, j,d:\path\junked; --------------- There are other options which you can explore later, if you wish, including three novelty tests. But now, let's put together some of the options listed above to make macros for automatic use. The three lines between the dashed lines, below, can be lifted out of this message, edited on your word processor to put in the drives, paths, and filenames, and then run as a .BAT program to install the macros: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - FWKCS a12c,d:\path\comment;e,d:\path\zipinzip;f,##cellar; /dc FWKCS t,d:\path\trashold;vq,d:\path\potvirus;v1,d:\path\telsysop; /dd FWKCS opwj,d:\path\junked; /de - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - These macros, incorporated in the command line by [cde] (macro [f] is not used in this instance, but may be included in [cdef]), tell FWKCS to automatically: test uploaded zipfiles for age, and accept a zipfile if the latest file in it not more than 12 months old ("12" is just an example; you can use a different number of months), install your zipfile comment, save zip_in_zip uploads, without misleading users into thinking that the file is available for downloading, use a subdirectory ##CELLAR under each node's respective default subdirectory as a workarea, put redundant or duplicate uploads into TRASHOLD for review, use McAfee's SCAN to test plain files and files inside zipfiles for viruses, report in TELSYSOP if a virus is found, and quarantine potential viruses in POTVIRUS, and Purge and Wipe unwanted material from zipfiles, keeping a log of such deletions in JUNKED. After you install the macros, you can check them by typing FWKCS /d Note that the macros are capitalized during installation, so it doesn't matter whether you use upper or lower case in your input. It's OK to mix lower and upper case, for example, to distinguish options from such things as drive letters, subdirectories, and the like. >>===> Be sure to create all the subdirectories. <===<< ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ How to fine-tune your options for uploads: **ÝFred, I typed in those manually rather than capturing them from the Ýscreen. ****, You can write a macro from FWKCS(TM) to a file. For example, to write macro [c] to a file named MACRO_C, just do FWKCS [c] [-macro_c] The minus sign in front of the filename tells FWKCS to write the fully expanded command line from its internal interpretation region of memory into the output file, and then return without executing the rest of the command line. This is explained on the first help screen, as format number 3. You can load a macro from a file. To continue with the above example, suppose you wanted to edit macro [c]. You could write it to an output file, and edit that file. To reload the edited version of the file named MACRO_C into FWKCS macro [c], just do FWKCS [+macro_c] /dc The plus sign tells FWKCS to read the contents of the file as part of the command line, as explained on the first help screen, under format number 3. How to load a macro from the command line is explained near the bottom of the FWKCS/d screen. **ÝI understood that if the "s" was specified, it would place the Ýfiles into the defined subdirectories rather than delete them, if Ýthey did not pass -- I would rather review a file personally before Ýkilling it. is there another way? the s suboption saves all rejected files right where they are; it does not place them into subdirectories. If the s suboption is not used, then there are specific suboptions to move specific kinds of rejects to their respectively specified subdirectories: if you set up q with a Quarantine subdirectory, viruses are put there. if you set up t with a Trashold subdirectory, duplicate or redundant files are put there. if you set up e with a Zipinzip subdirectory, zip in zip files are put there. if you set up n with a Qsn (Question as to sufficient novelty) subdirectory, files which fail the novelty test but are not fully duplicate nor fully redundant are put there. there is no backup on the x eXclusion flag rejection. to save files rejected because they failed the age test, see FWKCS112.REF. If you want backups of all uploads just in case, just put a copy command into the PCBTEST.BAT file, to copy whatever is uploaded into a backup directory. You can then let FWKCS carry out your instructions, and fine_tune the instructions by comparing the results with the backup copies of uploaded files. For example, - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @echo off rem PCBTEST.BAT with full backup of UPLOAD files to d:\path if not "%2" == "UPLOAD" goto notup copy %1 d:\path :notup FWKCS %1 %2 %3 /a5[cdef] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **Ýi wanted M in it's own macro so I could call it or not call it at Ýwill. If you wish, you can just tack the m on at the end of the command line, or not. When the command line is interpreted internally, the macros are inserted wherever cited, forming a contiguous string with whatever comes directly before and directly after, preserving intervening blank spaces as spaces. So, if you add a letter directly after the "]", it is read as part of the command line directly following the last macro cited. You can write the whole thing to a file to see what the assembled string looks like, by doing this: FWKCS /5[cdef]m [-zzz] That would write the string to a file named ZZZ . Then, to see it, just type TYPE ZZZ This is a convenient way to make special purpose configuration files. For example, when you run PCBTEST.BAT, you might ordinarily work in the individual subdirectories of the respective nodes, by leaving out the drive and path from the f,free; suboption. But, when you are working locally on one of your servers, you could make a configuration file ZZZ and edit it to insert a path to a particular free workarea elsewhere on your system. Then, instead of /a5[cdef]m , you would put [+zzz] to use that configuration file. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Discussion of automatic deletion: ***, you have raised some important points about automatic deletion. The FWKCS(TM) Contents_Signature System Ver. 1.12 contains hard_coded logical constraints on deletion, to help protect against various kinds of harm. These are complicated. (Those who do not wish to study the rest of this message should please see the item marked with arrows below.) Here are some of the steps involved: 1. To do a system_wide cleanup of zipfiles, execution is under FWKCS /1xz option. This creates a zipfile_contents_signature ("zcs") for each zipfile. The zcs is unaffected by the names of the files in the zipfile, their extensions, their compression ratios, dates, comments, zipped paths, and the order in which they appear. The deletion is based on the output from FWKCS /1s, which groups together only those files which have the same signature. The probability of statistical error in comparing two such FWKCS Contents_Signatures is typically less than one part in a trillion. A person marks which files to delete, by inserting a "d" in column 17, and then runs FWKC17D.BAS to perform the deletions. This is only able to delete zipfiles which have fully duplicate file contents. Each such deletion removes a complete zipfile in its entirety. 2. When processing a single file, FWKCS looks at the internal structure of the file in order to find whether it is a zipfile, and does not rely on the file's extension. 2.1 If the file is not a zipfile, then it is treated as a plain file and a single FWKCS plain file_contents_signature ("fcs") is generated for it. 2.2 If the file is a zipfile, FWKCS looks to see if the zipfile carries Authenticity Verification ("AV") (this can be made externally available, via /1xz options, for an entire zipfile (see FWKCS /1h help screen)). --- The performance of deletions from within a zipfile requires that the o (oust) command be given. 2.2.1 If the zipfile is an AV_zipfile, then the only deletions which FWKCS will do on files in that zipfile is to remove non_AV files which have been inserted into that AV_zipfile. This is done using the p Purge suboption. FWKCS ignores the Wipe flag and wipe suboption, and m (all matches) suboption, when processing AV_zipfiles. --- Presumably, only the person creating the AV_zipfile had a right to put files in it, and that right is carefully respected. --- Further, the AV_stamp is carefully preserved, so as to protect the rights of the person creating the AV_zipfile and to protect the user. --- I regard it as reasonable that any person offering commercial software packaged in zipfiles should be using AV_zipfiles. 2.2.2 If the zipfile is not an AV_zipfile, then the p suboption does not act. IF ((any file within the non_AV zipfile is found to match a FWKCS Contents_Signature which carries a "w" flag in column 17 (see second help screen provided by FWKCS /a ) AND the Wipe suboption has been invoked) OR (a cs_match was found AND the m (all matches) suboption has been invoked)) AND (deletion has been enabled by the o,Oust; suboption) THEN under FWKCS /f, /g, /a3, or /a4, FWKCS prepares an OUST file specifying what to delete, but does not perform the deletions. Under /a5 in processing an UPLOAD, or under /a6, if the o and w suboptions are given and a column_17 w_flag is found, or the m suboption is given and a cs_match is found, then the >>--> file is automatically deleted. If you are going to perform >>--> *any* automatic deletions from uploads, I suggest that you >>--> automatically copy each upload to a backup subdirectory first. I have reservations as to a need for a list of filenames to not delete: 3 novelty tests are provided. Novelty test n2,NNNN,QSN; might serve your needs, and allows review. Also, names are not as good as FWKCS Contents_Signatures to identify files whose removal you wish to control. And, unless you use m, a file in a non_AV zipfile is not going to be deleted from within that non_AV zipfile by FWKCS unless FWKCS finds that that file's cs_match has a w_flag AND Wipe is enabled. Don't mark it, and you won't Wipe it! And FWKCS won't Wipe a file from inside an Authenticity Verification Zipfile. (FWKCS options /a3 and /a4 preserve this recognition of AV_zipfiles in changing the file_extension and, as do /f and /g, in deciding whether to include w_flagged internal files in the OUST file for your consideration for deletion.) FWKCS also uses a column_17 "x" eXclusion flag. If any file within a zipfile has a cs_match which carries an x_flag, that entire zipfile is excluded. This is to protect against such things as illegal uploads of commercial software, uploads of known trojans, and the like. This may play a significant role in protecting the BBS community against abuse. FWKCS performs further steps to protect the integrity of the AV_stamp and to preserve the internal structure of a zipfile. For example, under logical control using contents_signatures and zipfile internal path information, FWKCS is able to test down to the full depth of a DOS tree of subdirectory levels, using only one subdirectory as a workarea. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Setup summary: Setting up has four major parts: 1. Setting up the subdirectory for FWKCS to use for its contents_ signature list; setting internal defaults 1-4 in FWKCS /d to point to that subdirectory; building the first list of contents_signatures; and using FWKCS to find and remove duplicates already on your system. At this stage, you can remove non_AV files which have been inserted into AV_zipfiles. Also, you can use a list of contents_signatures to remove ads often found in zipfiles. By using FWKCS to remove those non_AV files from AV_zipfiles, and to remove those ads, you may find that there were zipfiles which looked different from each other only because of superfluous material inserted into them. If you make such removals, then you need to make a new contents_signature list to use in searching for duplicates The clean-up process can take time, because it involves your judgment, on a case by case basis, as to which of the duplicate files to keep and which to delete. Programs to help do this are packaged along with FWKCS in AV_zipfile FWKCS112.ZIP. Often, on big BBSs, this initial clean_up has recovered many megabytes of disk space. 2. After the initial clean_up, building the working list of contents_ signatures. FWKCS112.ZIP includes .BAT programs to help do this. 3. Choosing options; installing them as macros in FWKCS; making PCBTEST.BAT (making PCBTEST.BAT is easy -- just go where you want to put it, and type FWKCS/a5m ); setting paths so that FWKCS can quickly reach Ben Baker's QSORT, Phil Katz's PKUNZIP(R) and PKZIP(R), Vernon Buerg's LIST, and McAfee's SCAN; setting the path for a central uploads log; making the subdirectories used for performing some of the options selected; making one subdirectory under each node for use as a free workarea. 4. Sending to Dr. Kantor a check in the amount specified below. :) If you are under time pressure and want to put off the clean_up, it is possible to get FWKCS up and running in a few hours (using GO.BAT, with only a few keystrokes), and later remove duplicates and rebuild your main contents_signature list. Even if you skip the initial clean_up, FWKCS would still protect your BBS against new duplicate or redundant uploads. But try to get to it later if you put it off. From what people have said in the Ilink and SmartNet SYSOPS conferences, that clean_up can be quite an eye_opener! ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ R E G I S T R A T I O N - - - - - - - - - - - - FWKCS Registration Prices effective 1991 Mar 07 ----------------------------------------------- Application Registration ----------- ------------ Personal private use $ 25.00 Electronic Bulletin Board 1 Node $ 25.00 1-3 Nodes $ 48.00 1-6 Nodes $ 70.00 1-9 Nodes $ 92.00 1-99 Nodes $ 194.00 For Electronic Bulletin Board use, the registration fee depends on the capacity of the BBS software installed, such as may be seen on the welcome screen or otherwise remotely noted. The registration upgrade price on going to a larger capacity is just the current new registration price minus the current registration price for the smaller system. Registration is by Firt Class U.S. Mail addressed to Dr. Frederick W. Kantor Founder Information Mechanics 523 West 112 Street New York, NY 10025-1614 Payments -------- Preferred forms of payment are Money Order, Cashier's Check, or Personal/Company Check, made payable to "Fred Kantor" or to "Frederick W. Kantor". For safety, Please mark all checks, "For deposit only." Please do not send cash through the U.S. Mail. Support ------- Persons in good standing who subscribe to The Invention Factory BBS, and who have full current registration for the FWKCS(TM) Contents_Signature System, may obtain support in the FWKCS conference on The IFBB, 212 431 1194, 2400 N 8 1 (other speeds are available, up to 19,200 bits/sec; Hayes, USRobotics V.32, and CompuCom high speed modems are supported; 44 nodes, 5.8 Gigabytes of storage). To register FWKCS, see above. Subscription to The IFBB is between the user and The IFBB. The FWKCS conference is private. Limited questions may be discussed also in SmartNet SYSOP conference and in Ilink Shareware conference.