• Echomail import limits?

    From RJ Clay@1:120/419 to All on Thursday, May 09, 2013 11:21:44
    Hi all

    I don't recall if I've brought this up again but are there echomail import limits that may not be explictly configurable? I ask because I noticed again (on my v0.92.0 system at 1:120/544) that although a rescan of an echo to my uplink resulted in a message noting that 400 messages were scanned but my system only ended up importing 47. Need to figure out where that is...


    Jame


    --- GoldED+/LNX 1.1.5--b20111217
    * Origin: RJC eeePC (1:120/419)
  • From Andrew Leary@1:320/119 to RJ Clay on Friday, May 10, 2013 17:05:44
    Hello RJ!

    Thursday May 09 2013 11:21, RJ Clay wrote to All:

    I don't recall if I've brought this up again but are there echomail import limits that may not be explictly configurable? I ask because I noticed again (on my v0.92.0 system at 1:120/544) that although a
    rescan of an echo to my uplink resulted in a message noting that 400 messages were scanned but my system only ended up importing 47. Need
    to figure out where that is...

    Is it possible the others were discarded as duplicates?

    Andrew

    ---
    * Origin: Bits & Bytes BBS * V.Everything! * 860/535-4284 (1:320/119)
  • From mark lewis@1:3634/12.42 to Andrew Leary on Friday, May 10, 2013 20:20:10

    I don't recall if I've brought this up again but are there
    echomail import limits that may not be explictly configurable? I
    ask because I noticed again (on my v0.92.0 system at 1:120/544)
    that although a rescan of an echo to my uplink resulted in a
    message noting that 400 messages were scanned but my system only
    ended up importing 47. Need to figure out where that is...

    Is it possible the others were discarded as duplicates?

    they would have been logged as dupes, wouldn't they? ;)

    )\/(ark

    ---
    * Origin: (1:3634/12.42)
  • From RJ Clay@1:120/419 to Andrew Leary on Saturday, May 11, 2013 05:40:25
    Hi Andrew!

    10 May 13 17:05, you wrote to me:

    Thursday May 09 2013 11:21, RJ Clay wrote to All:

    I don't recall if I've brought this up again but are there
    echomail import limits that may not be explictly configurable? I
    ask because I noticed again (on my v0.92.0 system at 1:120/544)
    that although a rescan of an echo to my uplink resulted in a
    message noting that 400 messages were scanned but my system only
    ended up importing 47. Need to figure out where that is...

    Is it possible the others were discarded as duplicates?

    Shouldn't have been; I'd just added the echo to the configuration...




    Jame



    --- GoldED+/LNX 1.1.5--b20111217
    * Origin: RJC eeePC (1:120/419)
  • From mark lewis@1:3634/12.42 to RJ Clay on Saturday, May 11, 2013 16:58:49

    I don't recall if I've brought this up again but are there
    echomail import limits that may not be explictly configurable? I
    ask because I noticed again (on my v0.92.0 system at 1:120/544)
    that although a rescan of an echo to my uplink resulted in a
    message noting that 400 messages were scanned but my system only
    ended up importing 47. Need to figure out where that is...

    Is it possible the others were discarded as duplicates?

    Shouldn't have been; I'd just added the echo to the
    configuration...

    that depends on the tosser's method of dupe detection... some of them allow the
    exact same message in more than one area because their dupe base is for each area... others use one dupe base for all areas...

    i can't say which i feel is best... both have their pluses and minuses...

    )\/(ark

    ---
    * Origin: (1:3634/12.42)
  • From RJ Clay@1:120/419 to mark lewis on Monday, May 13, 2013 10:19:43
    Hi mark!

    11 May 13 16:58, you wrote to me:


    I don't recall if I've brought this up again but are there
    echomail import limits that may not be explictly configurable?
    I ask because I noticed again (on my v0.92.0 system at
    1:120/544) that although a rescan of an echo to my uplink
    resulted in a message noting that 400 messages were scanned but
    my system only ended up importing 47. Need to figure out where
    that is...

    Is it possible the others were discarded as duplicates?

    Shouldn't have been; I'd just added the echo to the
    configuration...

    that depends on the tosser's method of dupe detection...

    True but in this case none of the messages would be in any other echo...



    Jame


    --- GoldED+/LNX 1.1.5--b20111217
    * Origin: RJC eeePC (1:120/419)
  • From Paul Quinn@3:640/384 to Rj Clay on Tuesday, May 14, 2013 08:32:00
    Hi! Jame,

    In a message to Mark Lewis you wrote:

    Is it possible the others were discarded as duplicates?

    Shouldn't have been; I'd just added the echo to the
    configuration...

    that depends on the tosser's method of dupe detection...

    True but in this case none of the messages would be in any other echo...

    Smack me if I'm wrong... I think this aspect is weak point for MBSE: the handling of dupes/bad mail. IIRC there ain't no option to 'toss from bad', after configuring new areas. So, you are at the mercy of any dupes in the dupe
    database.

    It would be nice to get some advice from Michiel or Vince on this...

    Cheers,
    Paul.

    ... Gravity isn't easy, but it's the law.
    --- Paul's Win98SE VirtualBox
    * Origin: Quinn's Post - Maryborough, Queensland, OZ (3:640/384)
  • From mark lewis@1:3634/12.71 to RJ Clay on Monday, May 13, 2013 15:28:28

    On Mon, 13 May 2013, RJ Clay wrote to mark lewis:

    I don't recall if I've brought this up again but are there
    echomail import limits that may not be explictly configurable? I
    ask because I noticed again (on my v0.92.0 system at 1:120/544)
    that although a rescan of an echo to my uplink resulted in a
    message noting that 400 messages were scanned but my system only
    ended up importing 47. Need to figure out where that is...

    Is it possible the others were discarded as duplicates?

    Shouldn't have been; I'd just added the echo to the
    configuration...

    that depends on the tosser's method of dupe detection...

    True but in this case none of the messages would be in any other echo...

    understandable... but that doesn't mean that the MSGID hasn't been seen before and it is possible that there may be a CRC16 or CRC32 collision if the tosser uses such methods and then that also depends on what data in the message they use for those calculations... some take only the header information... some take the header information plus 40 bytes or so to try to include some of the hidden control lines... they are hoping to catch the MSGID line in there but if
    the line is too far away that that won't always work...

    duplicate detection is definitely an art and generating messages to pass the various duplicate detection schemes can be even more of an art... however, in some cases, the duplicate detection is very brain dead (eg: header only or message body only) and no amount of crafting will get a valid new message by them... sbbsecho is one that insists that a message with an identicle body is a
    duplicate even thought the header plainly shows that it is a new posting... DB is another one with a similar method of duplicate detection...

    )\/(ark

    --- FMail/Win32 1.60
    * Origin: (1:3634/12.71)
  • From mark lewis@1:3634/12.71 to Paul Quinn on Tuesday, May 14, 2013 16:26:04

    On Tue, 14 May 2013, Paul Quinn wrote to Rj Clay:

    that depends on the tosser's method of dupe detection...

    True but in this case none of the messages would be in any other echo...

    Smack me if I'm wrong... I think this aspect is weak point for
    MBSE: the handling of dupes/bad mail. IIRC there ain't no option
    to 'toss from bad', after configuring new areas. So, you are at
    the mercy of any dupes in the dupe database.

    exactly and this isn't the only system aflicted with this problem...

    synchronet (actually the sbbsecho tosser) is another one... it deletes what it considers to be duplicates with no facility for saving them to a special area for human perusal... the only real solution to this is to use a script that deletes the dupe databases before each round of tossing performs the toss and then deletes the dupe bases again after the toss... this second deletion so that other methods of message importing (QWK, news) don't get caught with possibly false dupes... it barely has the ability to save messages for areas not defined and no good way to move them into the proper area once it is defined... i'm ever hopefuly that this will be remedied... i'd contribute if i were a C coder but since i'm not, i can't... all i can do is point out what is desired and how it would be done...

    i'm sure there are other tossers with similar aflictions, as well...

    )\/(ark

    --- FMail/Win32 1.60
    * Origin: (1:3634/12.71)
  • From Vince Coen@2:250/1 to Paul Quinn on Tuesday, May 14, 2013 23:02:15
    Hello Paul!

    14 May 13 08:32, you wrote to Rj Clay:

    that depends on the tosser's method of dupe detection...

    True but in this case none of the messages would be in any
    other echo...

    Smack me if I'm wrong... I think this aspect is weak point for MBSE:
    the handling of dupes/bad mail. IIRC there ain't no option to 'toss
    from bad', after configuring new areas. So, you are at the mercy of
    any dupes in the dupe database.

    It would be nice to get some advice from Michiel or Vince on this...


    For badmail that I have created a echo area for I just use golded to mark them and them move to the new area.

    On dupes I have it set to 16k and have not noticed any issues but I also use age, so that any msgs over 30 days old get junked.

    Hope that helps in some way.


    Vince

    --- Linux/Mbse/GoldED+/LNX 1.1.5-b20120229
    * Origin: Air Applewood, The Linux Gateway to the UK (2:250/1)
  • From Paul Quinn@3:640/384 to Vince Coen on Wednesday, May 15, 2013 15:18:00
    Hi! Vince,

    On Tue, 14 May 13, you wrote to me:

    For badmail that I have created a echo area for I just use golded to
    mark them and them move to the new area.

    Neat, and cunning. I like it! ;-)

    On dupes I have it set to 16k and have not noticed any issues but I
    also use age, so that any msgs over 30 days old get junked.

    I have some large-ish echoes, some going back a decade, so I use 50,000 records
    in FastEcho's dupe database.

    Hope that helps in some way.

    I did. Thank you.

    Cheers,
    Paul.

    ... Hey SysOp! You'd better upgrade me or el%$^&%NO CARRIER
    --- Paul's Win98SE VirtualBox
    * Origin: Quinn's Post - Maryborough, Queensland, OZ (3:640/384)