• file size issues?

    From Roy J. Tellason@1:270/615 to all\ on Wednesday, July 28, 2004 21:07:08
    I have one echo here where I really let things pile up, to something over 4000
    messages, until I decided to do something about it a while ago. Wrote out stuff I wanted to keep, deleted bunches of messages, and then thought I'd have a go at packing it.

    No go.

    First attempt with me trying to use sqpack got me an error message when it tried to open the ".sqd" file. The second attempt was using sqfix, which gave
    me the same error message.

    I finally got past it by creating another echo in my squish.cfg, using TimED to "move" stuff over to that, and then deleting the original file and renaming
    the new ones to what the old ones were.

    The .sqd file in question was a bit over 8M in size. Is this a problem for Squish running under dos?

    ---
    * Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)
  • From Mike Tripp@1:382/61 to Roy J. Tellason on Thursday, July 29, 2004 08:57:38
    Hello Roy!

    28 Jul 04 20:07, Roy J. Tellason wrote to all\:

    The .sqd file in question was a bit over 8M in size. Is this a
    problem for Squish running under dos?

    Nope...I have many that are larger without issue. SQFIX and GOLDED DOS versions start misbehaving when areas hit 5400 msgs or so here, though, so I cap mine at 5000. If they go over and break, then SQFIX32 might work...but not
    always. Even with 5K caps, here's my top 5:

    ALLFIX_F SQD 28,155,429 07-29-04 3:44a
    WEATHER SQD 27,625,285 07-29-04 3:43a
    U-F1O4Q SQD 20,507,829 07-29-04 4:09a
    U-RV6BX SQD 15,777,708 07-29-04 4:09a
    U-RVG8O SQD 15,584,469 07-29-04 3:39a

    If you don't already have SQPACK in a regular maintenance routine (daily here),
    it is probably a good idea to run it both before and after your manual cleanup activity, to insure the integrity of the area. There were probably some pointers that were confused before you started (in the .sqi), which got really confused by the time you were done.

    .\\ike

    --- GoldED 2.50+
    * Origin: -=( The TechnoDrome )=- Austin,TX 512-327-8598 33.6k (1:382/61)
  • From Roy J. Tellason@1:270/615 to Mike Tripp on Saturday, July 31, 2004 13:13:08
    Mike Tripp wrote in a message to Roy J. Tellason:

    Hello Roy!

    28 Jul 04 20:07, Roy J. Tellason wrote to all\:

    The .sqd file in question was a bit over 8M in size. Is this a
    problem for Squish running under dos?

    Nope...I have many that are larger without issue. SQFIX and GOLDED
    DOS versions start misbehaving when areas hit 5400 msgs or so here, though, so I cap mine at 5000. If they go over and break, then
    SQFIX32 might work...but not always. Even with 5K caps, here's my
    top 5:

    ALLFIX_F SQD 28,155,429 07-29-04 3:44a
    WEATHER SQD 27,625,285 07-29-04 3:43a
    U-F1O4Q SQD 20,507,829 07-29-04 4:09a
    U-RV6BX SQD 15,777,708 07-29-04 4:09a
    U-RVG8O SQD 15,584,469 07-29-04 3:39a

    If you don't already have SQPACK in a regular maintenance routine
    (daily here), it is probably a good idea to run it both before and
    after your manual cleanup activity, to insure the integrity of the
    area. There were probably some pointers that were confused before
    you started (in the .sqi), which got really confused by the time
    you were done.

    I do have it running daily, but it doesn't do anything in those cases where I haven't set the message area to any particular limits, like the one in question:

    - 672 - Old=1131352; New=1131352

    This is after I'd trimmed out a bunch of stuff...

    5400 messages? There were something over 8000 of them in there, I forget how many now. And yes, this is the dos version.

    ---
    * Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)
  • From Mike Tripp@1:382/61 to Roy J. Tellason on Saturday, July 31, 2004 15:19:38
    Hello Roy!

    31 Jul 04 12:13, Roy J. Tellason wrote to Mike Tripp:

    - 672 - Old=1131352; New=1131352

    This is after I'd trimmed out a bunch of stuff...

    The file should've gotten smaller the first run after you deleted messages. Even without parameters, SQPACK should purge the "slack space" occupied by the deleted messages. Even if you trim to a specific quantity of messages, the 5K you have today may be fewer bytes than the 5K you had yesterday. If that's not
    the case, then you'll see the same bytes reported for Old/New as above.

    5400 messages?

    That seems to be where GoldEd/DOS starts complaining...GoldEd/2 is still fine beyond there...and your editor may have no trouble. I forget the magic number where SQFIX starts suggesting that you use SQFIX32 instead...but I use the lowest common denominator, which for me happens to be the editor's limit.

    There were something over 8000 of them in there, I
    forget how many now.

    672 according to the line above...and now you have a 1mb file instead of the 8mb you originally asked about.

    And yes, this is the dos version.

    And you will have different luck between the 16- and 32-bit DOS versions of SQFIX, but my experience is that is the message quantity (.SQI index entries) and not the filesize of the .SQD that makes or breaks you. It's not that rare that there is some index cleanup even on areas with low caps and small .SQDs, and both SQFIX and SQPACK have to manage these while doing their thing.

    .\\ike

    --- GoldED 2.50+
    * Origin: -=( The TechnoDrome )=- Austin,TX 512-327-8598 33.6k (1:382/61)
  • From Roy J. Tellason@1:270/615 to Mike Tripp on Sunday, August 01, 2004 05:07:32
    Mike Tripp wrote in a message to Roy J. Tellason:

    Hello Roy!

    31 Jul 04 12:13, Roy J. Tellason wrote to Mike Tripp:

    - 672 - Old=1131352; New=1131352

    This is after I'd trimmed out a bunch of stuff...

    The file should've gotten smaller the first run after you deleted messages.

    One would think so. It had gotten pretty large, something over 4000 msgs, and over 8M in size. I wrote out what I wanted to keep, by month, deleted the rest, and tried to run sqpack manually -- and that's when I got the error.
    A shot at sqinfo -b gave nothing, and sqfix gave me the same error.

    Even without parameters, SQPACK should purge the "slack space"
    occupied by the deleted messages. Even if you trim to a specific quantity of messages, the 5K you have today may be fewer bytes
    than the 5K you had yesterday. If that's not the case, then
    you'll see the same bytes reported for Old/New as above.

    I'm seeing the same numbers in both cases there because I have no settings in the area. Most of them here are set to some specific time limit (commonly 60 days at this point), one or two of the real busy ones are also set to a maximum number of messages, but I try to avoid that because it slows down tossing. That way I don't have to worry about feeding any parameters to sqpack
    in my batch file other than a wildcard pointing to the directory, and can change the settings on the fly here in TimED, as well as with sqset.

    5400 messages?

    That seems to be where GoldEd/DOS starts complaining...GoldEd/2 is
    still fine beyond there...and your editor may have no trouble.

    It didn't seem to. I used to use GoldEd but stopped, I can't recall why just now. I know the last time I tried it out I didn't care for it, or was too used to this, or something. <shrug>

    I forget the magic number where SQFIX starts suggesting that you
    use SQFIX32 instead...but I use the lowest common denominator,
    which for me happens to be the editor's limit.

    There were something over 8000 of them in there, I
    forget how many now.

    672 according to the line above...and now you have a 1mb file
    instead of the 8mb you originally asked about.

    Right. But to get to that point I had to *move* the remaining messages I wanted to keep from one area to another. I created a new area, same name as the other one but with a "2" at the end of the area name and no links listed in
    squish.cfg, fired up TimED again, and it found it. I moved everything over there (and didn't get that "moved" string implanted in the messages thankfully), exited the editor, and in my file manager deleted the old area and renamed the new one, then fixed my squish.cfg.

    In other words the software didn't work for me the way I expected it to, and I
    found a workaround. Which is pretty typical. But also why I posted on this in
    the first place, to find out if there's some known point when things get messed up. I know that this is definitely the case with largish text files under dos!

    And yes, this is the dos version.

    And you will have different luck between the 16- and 32-bit DOS
    versions of SQFIX, but my experience is that is the message
    quantity (.SQI index entries) and not the filesize of the .SQD that
    makes or breaks you. It's not that rare that there is some index
    cleanup even on areas with low caps and small .SQDs, and both SQFIX
    and SQPACK have to manage these while doing their thing.

    That would be the .sqi file? Here's what I'm looking at now for the area in question:

    .sqb 20008
    .sqd 1208034
    .sqi 8628
    .sql 4

    All with today's date, and a timestamp within the past hour when I was reading
    it last. What's that sqi file used for anyhow?

    ---
    * Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)
  • From Mike Tripp@1:382/61 to Roy J. Tellason on Sunday, August 01, 2004 12:08:00
    Hello Roy!

    01 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp:

    I'm seeing the same numbers in both cases there because I have no
    settings in the area.

    Lack of settings alone won't keep the numbers static. If you manually kill one
    message and SQPACK again, Old should be larger than New. If you manually edit one existing message to remove a few lines from the message, Old should be larger than New.

    Right. But to get to that point I had to *move* the remaining
    messages I wanted to keep from one area to another.

    Gotcha...and obviously the stats you're posting are for the new version of the area and not the original hosed version. It is the hosed version that should've been shrinking as messages were moved out of it. The new version was
    only growing to swallow the new messages being appended, so there's no reason for them to be occupying more than the actual number of bytes required to do so.

    All with today's date, and a timestamp within the past hour when I
    was reading it last. What's that sqi file used for anyhow?

    Those that've been into the source up to their elbows can probably find plenty of holes with my analogy, but if you understand the basics of how DOS manages files and space allocated on a FAT hard drive: the .SQI is the FAT table for the files (messages) within Squish's hard drive (.SQD).

    The .SQI basically has the pointers to where the header for each message starts
    and how many blocks it occupies. If you delete a message, Squish just updates the .SQI file that references the deleted data to remove the pointer to its header and updates it's usage map to show that space as available for writing new data again. The size of the .SQD is not necessarily representative of the actual number of bytes required by the currently retained messages. Squish will expand the .SQD if more space is needed to accomodate a new retention peak, but it will not shrink it just because fewer are required today than yesterday.

    Just as bad things can happen to a FAT if operations fail or are interrupted by
    a system lockup or power failure, the same is true for Squish and .SQI/.SQD manipulations. SQFIX does for the Squishbase what CHKDSK and/or Disk Doctor does for the integrity of the FAT. It looks to make sure that a message header
    actually starts where each .SQI entry says it does and looks for space that the
    .SQI says should be in use by a retained message but actually isn't (akin to lost clusters and cross-linked files) and attempts to either fix or remove .SQI
    entries that don't seem to match reality found in the .SQD.

    SQPACK is analagous to DEFRAG. If you have specified parameters, it first marks the messages that fail those parameters as deleted and then "compresses" the .SQD data to the beginning of the .SQD file and shrinks the .SQD file down to the actual size required to store the actual bytes associated with those messages that are still retained, rather than the size being some former worst-case maximum that was required to store messages that met the SQSET parameters at some point in the past since you last SQPACKed.

    So if you use no purge parameters, never manually delete/edit a message in an area, and never have any problem that causes the .SQI to lose synch with the actual contents of the .SQD, it will appear that SQPACK never does anything...just as running DEFRAG wouldn't accomplish anything for your hard drive if you only appended new files one at a time and never deleted or modified an existing file since your last DEFRAG. If, however, you make any of
    those changes or SQFIX frees a pointer to an existing message because of a data
    integrity issue, it is possible/probable that there will be some delta between the old/new bytes when you SQPACK next, even if you haven't assigned specific purge criteria to the area.

    .\\ike

    --- GoldED 2.50+
    * Origin: -=( The TechnoDrome )=- Austin,TX 512-327-8598 33.6k (1:382/61)
  • From Roy J. Tellason@1:270/615 to Mike Tripp on Monday, August 02, 2004 05:07:02
    Mike Tripp wrote in a message to Roy J. Tellason:

    Hello Roy!

    01 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp:

    I'm seeing the same numbers in both cases there because I have no
    settings in the area.

    Lack of settings alone won't keep the numbers static. If you
    manually kill one message and SQPACK again, Old should be larger
    than New. If you manually edit one existing message to remove a few
    lines from the message, Old should be larger than New.

    Sure.

    Right. But to get to that point I had to *move* the remaining
    messages I wanted to keep from one area to another.

    Gotcha...and obviously the stats you're posting are for the new
    version of the area and not the original hosed version.

    Right. That one's gone. Old habit, getting rid of stuff like that, from when disk space was a scarcer resource.

    It is the hosed version that should've been shrinking as messages
    were moved out of it.

    No, the hosed version wouldn't have been shrinking as sqpack wouldn't touch it... Too bad my "pack.log" gets rewritten every day, or I'd pull a line out
    of that one to show it.

    The new version was only growing to swallow the new messages being appended, so there's no reason for them to be occupying more than
    the actual number of bytes required to do so.

    Yup. And the way things looked it wasn't ever gonna get any smaller. I'm looking at something like 38M currently on that drive, not so much that I need
    to worry about an 8M file eating too much space but still...

    All with today's date, and a timestamp within the past hour when I
    was reading it last. What's that sqi file used for anyhow?

    Those that've been into the source up to their elbows can probably
    find plenty of holes with my analogy, but if you understand the
    basics of how DOS manages files and space allocated on a FAT hard
    drive: the .SQI is the FAT table for the files (messages) within
    Squish's hard drive (.SQD).

    Ok.

    The .SQI basically has the pointers to where the header for each
    message starts and how many blocks it occupies. If you delete a
    message, Squish just updates the .SQI file that references the
    deleted data to remove the pointer to its header and updates it's
    usage map to show that space as available for writing new data
    again. The size of the .SQD is not necessarily representative of
    the actual number of bytes required by the currently retained
    messages. Squish will expand the .SQD if more space is needed to accomodate a new retention peak, but it will not shrink it just
    because fewer are required today than yesterday.

    Makes sense. This also explains some why TimED's undelete function seems to grab *all* deleted messages, I guess that was just simpler to code, or something.

    Just as bad things can happen to a FAT if operations fail or are interrupted by a system lockup or power failure, the same is true
    for Squish and .SQI/.SQD manipulations. SQFIX does for the
    Squishbase what CHKDSK and/or Disk Doctor does for the integrity of
    the FAT. It looks to make sure that a message header actually
    starts where each .SQI entry says it does and looks for space that
    the .SQI says should be in use by a retained message but actually
    isn't (akin to lost clusters and cross-linked files) and attempts
    to either fix or remove .SQI entries that don't seem to match
    reality found in the .SQD.

    SQPACK is analagous to DEFRAG. If you have specified parameters,
    it first marks the messages that fail those parameters as deleted
    and then "compresses" the .SQD data to the beginning of the .SQD
    file and shrinks the .SQD file down to the actual size required to
    store the actual bytes associated with those messages that are
    still retained, rather than the size being some former worst-case
    maximum that was required to store messages that met the SQSET
    parameters at some point in the past since you last SQPACKed.

    It also seems to write the results out to a new file, so for example if a single message area is larger than the available space on disk, it will *never* get packed down. I have one such area at present, the .sqd file is something around 66 megs. This was from an internet mailing list I was carrying for a while, back when I had that capability on the system, and I figured that I'd just delete stuff as I read 'em. I somehow haven't found the time to mess with that one lately, and if I don't eventually I may end up just
    deleting it.

    So if you use no purge parameters, never manually delete/edit a
    message in an area, and never have any problem that causes the .SQI
    to lose synch with the actual contents of the .SQD, it will appear
    that SQPACK never does anything...just as running DEFRAG wouldn't accomplish anything for your hard drive if you only appended new
    files one at a time and never deleted or modified an existing file
    since your last DEFRAG. If, however, you make any of those changes
    or SQFIX frees a pointer to an existing message because of a data integrity issue, it is possible/probable that there will be some
    delta between the old/new bytes when you SQPACK next, even if you
    haven't assigned specific purge criteria to the area.

    There are only a few areas like that here, most are set for specified time frames, typically 60 days.

    And there do seem to be times when random errors creep in, I don't know if it's memory glitches, power surges, or what, though for the most part things
    are pretty quiet around here in the small hours when maintenance is done. Every so often I'll scan my pack.log file for "err" and find some, like that one mentioned above, but mostly it won't find it, and I'll figure things are okay at that point as there's plenty of areas and they all get packed down each
    night.

    ---
    * Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)
  • From Mike Tripp@1:382/61 to Roy J. Tellason on Tuesday, August 03, 2004 09:01:38
    Hello Roy!

    02 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp:

    The new version was only growing to swallow the new messages
    being appended, so there's no reason for them to be occupying
    more than the actual number of bytes required to do so.

    Yup. And the way things looked it wasn't ever gonna get any smaller.
    I'm looking at something like 38M currently on that drive, not so
    much that I need to worry about an 8M file eating too much space but still...

    I was referring to your new replacement files here, not the original hosed version. If you don't assign it any purge paramaters either, then there's no reason for SQPACK to ever shrink the file, and it should head straight back for
    8mb or however far it grows until it a data integrity issue stops one utility or another from being able to interact with it correctly any more.

    If you want to keep areas uncapped, it might help to do a monthly SQREIDX or SQFIX event to catch little issues before they become big/fatal issues.

    Makes sense. This also explains some why TimED's undelete
    function seems to grab *all* deleted messages, I guess that was
    just simpler to code, or something.

    Yep...and probably has the same pitfoils as UNDELETE in DOS. "Deleted" data is
    overwritten by new data arriving...so some deleted msgs are unrecoverable. TimEd is probably just grabbing all of the puzzle pieces it can find to start with, figuring out which fit together, and then throwing away the few leftovers
    that don't seem to fit anywhere when done.

    There are only a few areas like that here, most are set for
    specified
    time frames, typically 60 days.

    SQUISH and SQPACK work together to maintain the quantity limit. SQPACK alone does the limit by days. SQPACK should =always= show some shrinkage on each daily run for areas with a days limit, unless the area is newer than the limit or the area has days without mail.

    .\\ike

    --- GoldED 2.50+
    * Origin: -=( The TechnoDrome )=- Austin,TX 512-327-8598 33.6k (1:382/61)
  • From Roy J. Tellason@1:270/615 to Mike Tripp on Tuesday, December 14, 2004 12:09:06
    Mike Tripp wrote in a message to Roy J. Tellason:

    Hello Roy!

    02 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp:

    The new version was only growing to swallow the new messages
    being appended, so there's no reason for them to be occupying
    more than the actual number of bytes required to do so.

    Yup. And the way things looked it wasn't ever gonna get any smaller.
    I'm looking at something like 38M currently on that drive, not so
    much that I need to worry about an 8M file eating too much space but still...

    I was referring to your new replacement files here, not the
    original hosed version. If you don't assign it any purge
    paramaters either, then there's no reason for SQPACK to ever shrink
    the file, and it should head straight back for 8mb or however far
    it grows until it a data integrity issue stops one utility or
    another from being able to interact with it correctly any more.

    A data integrity issue? Hmm.

    If you want to keep areas uncapped, it might help to do a monthly
    SQREIDX or SQFIX event to catch little issues before they become
    big/fatal issues.

    I never thought about sqreidx, missed that one completely. Sure enough, there it is, right in my squish directory where it should be!

    Actually what I plan on doing is archiving the stuff I want to keep about once a month (keeping it in month-sized chunks) and then deleting those from the active message base. What I put off for *way* too long with this particular area, since the archives I did end up creating go all the way back to last November.

    Makes sense. This also explains some why TimED's undelete
    function seems to grab *all* deleted messages, I guess that was
    just simpler to code, or something.

    Yep...and probably has the same pitfoils as UNDELETE in DOS.
    "Deleted" data is overwritten by new data arriving...so some
    deleted msgs are unrecoverable. TimEd is probably just grabbing all
    of the puzzle pieces it can find to start with, figuring out which
    fit together, and then throwing away the few leftovers that don't
    seem to fit anywhere when done.

    Or something. I don't use it that often to be able to guess.

    There are only a few areas like that here, most are set for
    specified time frames, typically 60 days.

    SQUISH and SQPACK work together to maintain the quantity limit.

    Yeah, which is why I use it seldom, since it slows down message tossing.

    SQPACK alone does the limit by days. SQPACK should =always= show
    some shrinkage on each daily run for areas with a days limit,
    unless the area is newer than the limit or the area has days
    without mail.

    Sounds right to me!

    I just wonder what the problem was. Oh well. I won't likely be letting any area get that big again so I probably won't run into the problem again...
    ---
    * Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)