Subj : mutil MassUpload adding duplicates?
To   : g00r00
From : Bj”rn Wiberg
Date : Mon Feb 07 2022 02:06 pm

Hello g00r00!

I've noticed that since the A48 prealpha I'm running (2022/02/02), mutil MassUpload appears to add duplicates to the file areas every now and then.

It doesn't happen all the time, and the files and areas appear to differ from time to time when this happens, which makes it hard to spot any pattern.

Example:


---------------------  MUTIL v1.12 A48 2022/02/02 Mon, Feb 07 2022 (loglevel 3)
+ 2022-02-07 00:00:02  Startup using maint.ini
- 2022-02-07 00:00:02  EXEC MassUpload
- 2022-02-07 00:00:02  EXEC GenerateTopLists
- 2022-02-07 00:00:02  EXEC PackUserBase
- 2022-02-07 00:00:02  EXEC PurgeMessageBases
- 2022-02-07 00:00:02  EXEC PackMessageBases
- 2022-02-07 00:00:02  EXEC LinkMessages
- 2022-02-07 00:00:02  EXEC MergeNodeLists
- 2022-02-07 00:00:02  EXEC PackFileBases
- 2022-02-07 00:00:02  EXEC FileSort
- 2022-02-07 00:00:02  EXEC GenerateAllFiles
- 2022-02-07 00:00:02  EXEC Export_FILEBONE.NA
- 2022-02-07 00:00:02  EXEC Export_AREAS.BBS
- 2022-02-07 00:00:02  EXEC Export_Golded
+ 2022-02-07 00:00:02  Process: Mass Upload Files
+ 2022-02-07 00:00:02    Uploads
+ 2022-02-07 00:00:02    Allocating 128
+ 2022-02-07 00:00:02    Free 128
+ 2022-02-07 00:00:02    FSX_ARTS: ANSI Art - Groups, Individuals, Etc.
+ 2022-02-07 00:00:02    Allocating 2,472
+ 2022-02-07 00:00:03    Free 2,472
+ 2022-02-07 00:00:03    FSX_DOOR: BBS Doors, Games, Etc.
+ 2022-02-07 00:00:03    Allocating 888
+ 2022-02-07 00:00:04    Free 888
+ 2022-02-07 00:00:04    FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:04    Allocating 8,856
+ 2022-02-07 00:00:04      Add: asicbrwp.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:04      Add: astronwp.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:05      Add: ap220202.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:05      Add: ap220203.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:05      Add: ap220204.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:05    Free 8,856
+ 2022-02-07 00:00:05    FSX_INFO: Weekly Infopacks (fsxNet, Etc.)


(Cut off -- all OK in what follows; Allocating and Free with matching values all the time there as well.)

The file area in question now has e.g. a duplicate ap220203.zip file:


ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿
³ ± File Listing (00 tagged)                  FSX_IMGE: Image Files (Various) ³
ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ
 01 ap220203.zip                                                430KB 02/07/22
                             NASA Astronomy Picture of the
                             Day (plus published report)
                             À Uploader: Mystic BBS
 02 ap220203.zip                                                430KB 02/03/22
                             NASA Astronomy Picture of the
                             Day (plus published report)
                             À Uploader: Mystic TIC


The maint.ini contains (among other things):


[MassUpload]

        ; this function searches all configured file directories for new
        ; files and will upload them into the BBS.  It will attempt to
        ; import FILE_ID.DIZ using the configured archivers if the option
        ; is enabled.

        ; Name to save has the uploader

        uploader_name = Mystic BBS

        ; Import FILE_ID.DIZ?  1=yes

        import_fileid = 1

        ; No description string used when no FILE_ID.DIZ is imported.

        no_description = No Description

        ; Ignore list one file mask per line (allows * and ? wildcards)

        ignore = .htaccess
        ignore = .htpasswd

        ignore = *.readme
        ignore = *.README

        ignore = file_id.*
        ignore = FILE_ID.*
        ignore = files.bbs
        ignore = FILES.BBS
        ignore = lost+found
        ignore = readme
        ignore = README
        ignore = readme.txt
        ignore = README.txt
        ignore = README.TXT

[PackFileBases]

        ; If true, MUTIL will check to make sure files exist, and will update their
        ; file size in the listing, optionally removing missing files (remove_missing)

        check_files = true

        ; If true, MUTIL will remove any files which do not exist from the file
        ; list database while checking integrity.  Otherwise they will be flagged as
        ; OFFLINE in the file listing.  0 byte files are considered "missing"

        remove_missing = false


Can you think of any way to debug this, and/or anything that could cause
this?

Just a guess but could this change possibly have anything to do with it:

 ! MUTIL FilePack was not properly setting the OFFLINE flag when check_files
   was enabled, remove_missing was disabled, and a file was physically removed

E.g. that the file area data files are being "corrupted" in some way when packing, causing the additions to be done as something doesn't quite match up?

I've collected the data files before and after (from my NAS trashcan) if you would like to take a peek at some of the files to see if they could possibly be "corrupted" in some way:

https://scbbs.nsupdate.info/files/FSX_IMGE.tar.gz

(old/ = before MassUpload; new/ = after MassUpload)

Many thanks in advance, and please let me know if there's anything I can do!

Best regards
Bj”rn

--- Mystic BBS v1.12 A48 2022/02/02 (Linux/64)
 * Origin: Star Collision BBS, Uppsala, Sweden (2:201/137)

.