• Double postings

    From LEE GREEN@1:102/401 to TERRY ROATI on Wednesday, September 11, 2019 18:35:20
    Did you know that your messages are posting twice?





    ...Take my advice, I don't use it anyway.
    ---BapStats Module (bsDBASE v6.1 Build 1)

    * Origin: TECHWARE BBS - Since 1995 - www.techware.dynip.com/ (1:102/401)
  • From Paul Quinn@3:640/1384 to LEE GREEN on Thursday, September 12, 2019 16:39:00
    Hi! Lee,

    On 11 Sep 19 18:35, you wrote to TERRY ROATI:

    Did you know that your messages are posting twice?

    Not 'posting'. Just copied by various mail pushing centers (hubs/nodes).

    For example, your note came to me from Mark Lewis in North Carolina...

    @PATH: 102/401 218/700 261/38 3634/12 640/1384

    Mark's system probably got at a second copy from my system that may have travelled to me via a different route. That would be a duplicate at Mark's joint and was killed by his dupe-detection facility in his software, which is actually a-okay. It's called a GoodThing(tm) in echomail mover huddles.

    All of this is a no-drama consequence, given most traffic is travelling over the internet super highway, by high-speed machines and high-capacity drives.

    Cheers,
    Paul.

    ... Simon says Stand! Simon says sit! Format drive C: ! HA! Gotcha!
    --- GoldED+/LNX 1.1.5-b20130515
    * Origin: Quinn's Rock - Live from Paul's Xubuntu desktop! (3:640/1384)
  • From Terry Roati@3:640/1321 to LEE GREEN on Thursday, September 12, 2019 16:33:22

    Lee,

    Do know if this is this happening every time and if it's in other echos as well?

    Thanks.

    On Sep 11, 2019 06:42pm, LEE GREEN wrote to TERRY ROATI:

    Did you know that your messages are posting twice?




    Terry Roati - 3:640/1321 tfb-bbs.org

    ... Platinum Xpress & Wildcat!..... Nice!!!!
    --- Platinum Xpress/Win/WINServer v7.0
    * Origin: The File Bank BBS! (3:640/1321)
  • From LEE GREEN@1:102/401 to TERRY ROATI on Thursday, September 12, 2019 07:27:06
    Lee,

    Do know if this is this happening every time and if it's in other echos as well?

    Yes every time, I will answer both of these messages received.

    WC should be catching any dupes like it usually does for dupes.


    Thanks.

    On Sep 11, 2019 06:42pm, LEE GREEN wrote to TERRY ROATI:

    Did you know that your messages are posting twice?




    Terry Roati - 3:640/1321 tfb-bbs.org

    ... Platinum Xpress & Wildcat!..... Nice!!!!
    --- Platinum Xpress/Win/WINServer v7.0
    * Origin: The File Bank BBS! (3:640/1321)
    --- Platinum Xpress/Win/WINServer v3.1
    * Origin: TECHWARE BBS - Since 1995 - www.techware.dynip.com/ (1:102/401)
  • From LEE GREEN@1:102/401 to TERRY ROATI on Thursday, September 12, 2019 07:35:40
    @TID: PX/Win v7.0 PX96-0649M
    @MSGID: 3:640/1321 9ADCB6F0
    @REPLY: 1:102/401 bc88f121
    @TZUTC: 1000
    Lee,

    Do know if this is this happening every time and if it's in other echos as ->well?

    Terry:
    It looks like it started happening on 9/9/2019 on message(s) to Greg Youngblood, all messages prior were not duplicated.

    WC should catch any dupes like it does everyday.

    Thanks.

    On Sep 11, 2019 06:42pm, LEE GREEN wrote to TERRY ROATI:

    Did you know that your messages are posting twice?




    Terry Roati - 3:640/1321 tfb-bbs.org

    ... Platinum Xpress & Wildcat!..... Nice!!!!
    --- Platinum Xpress/Win/WINServer v7.0
    * Origin: The File Bank BBS! (3:640/1321)



    ...What has four legs and an arm? A happy pitbull.
    ---BapStats Module (bsDBASE v6.1 Build 1)

    * Origin: TECHWARE BBS - Since 1995 - www.techware.dynip.com/ (1:102/401)
  • From LEE GREEN@1:102/401 to TERRY ROATI on Thursday, September 12, 2019 07:43:46
    @TID: PX/Win v7.0 PX96-0649M
    @MSGID: 3:640/1321 9ADCB6F0
    @REPLY: 1:102/401 bc88f121
    @TZUTC: 1000

    Lee,

    Do know if this is this happening every time and if it's in other echos as ->well?

    Which other conferences do you post in so I can check?

    Thanks.

    On Sep 11, 2019 06:42pm, LEE GREEN wrote to TERRY ROATI:

    Did you know that your messages are posting twice?




    Terry Roati - 3:640/1321 tfb-bbs.org



    ...Suicide is the most sincere form of self criticism.
    ---BapStats Module (bsDBASE v6.1 Build 1)

    * Origin: TECHWARE BBS - Since 1995 - www.techware.dynip.com/ (1:102/401)
  • From LEE GREEN@1:102/401 to Paul Quinn on Thursday, September 12, 2019 07:45:44
    Hi! Lee,

    On 11 Sep 19 18:35, you wrote to TERRY ROATI:

    Did you know that your messages are posting twice?

    Not 'posting'. Just copied by various mail pushing centers (hubs/nodes).

    For example, your note came to me from Mark Lewis in North Carolina...

    @PATH: 102/401 218/700 261/38 3634/12 640/1384

    Mark's system probably got at a second copy from my system that may have travelled to me via a different route. That would be a duplicate at Mark's joint and was killed by his dupe-detection facility in his software, which
    is
    actually a-okay. It's called a GoodThing(tm) in echomail mover huddles.

    WC catches dupes very well too, for some reason they are not being seen
    as dupes.

    All of this is a no-drama consequence, given most traffic is travelling over the internet super highway, by high-speed machines and high-capacity drives.

    Well no drama if you want dupes everywhere in your system.

    Cheers,
    Paul.

    * Origin: Quinn's Rock - Live from Paul's Xubuntu desktop! (3:640/1384)
    --- Platinum Xpress/Win/WINServer v3.1
    * Origin: TECHWARE BBS - Since 1995 - www.techware.dynip.com/ (1:102/401)
  • From Paul Quinn@3:640/1384 to LEE GREEN on Friday, September 13, 2019 08:21:25
    Hi! Lee,

    On 12 Sep 19 07:45, you wrote to me:

    Well no drama if you want dupes everywhere in your system.

    Yes. If nothing else, it means:

    1) people are still writing in Fidonet;
    2) you're still connected; and,
    3) your software still works.

    If all else fails, you could just read your dupe base instead of chopping & changing between echo areas all the time. Neat.

    Cheers,
    Paul.

    ... Whenever I get a grip on reality, the handle falls off.
    --- GoldED+/LNX 1.1.5-b20130515
    * Origin: Quinn's Rock - Live from Paul's Xubuntu desktop! (3:640/1384)
  • From LEE GREEN@1:102/401 to TERRY ROATI on Thursday, September 12, 2019 18:12:32
    @TID: PX/Win v7.0 PX96-0649M
    @MSGID: 3:640/1321 9ADCB6F0
    @REPLY: 1:102/401 bc88f121
    @TZUTC: 1000
    Lee,

    Do know if this is this happening every time and if it's in other echos as ->->well?

    Terry:

    make that July it started, when you started WC ?

    WC should catch any dupes like it does everyday.

    Thanks.

    On Sep 11, 2019 06:42pm, LEE GREEN wrote to TERRY ROATI:

    Did you know that your messages are posting twice?




    Terry Roati - 3:640/1321 tfb-bbs.org

    ... Platinum Xpress & Wildcat!..... Nice!!!!
    --- Platinum Xpress/Win/WINServer v7.0
    * Origin: The File Bank BBS! (3:640/1321)



    ...What has four legs and an arm? A happy pitbull.
    ---BapStats Module (bsDBASE v6.1 Build 1)




    ...You're only young once; you can be immature forever
    ---BapStats Module (bsDBASE v6.1 Build 1)

    * Origin: TECHWARE BBS - Since 1995 - www.techware.dynip.com/ (1:102/401)
  • From LEE GREEN@1:102/401 to Paul Quinn on Thursday, September 12, 2019 18:14:50
    @TID: FMail-lnx32 2.1.0.18-B20170905
    @TZUTC: 1000
    @CHRS: UTF-8 2
    @MSGID: 3:640/1384 5d7ac657
    @REPLY: 1:102/401 3bb50e66
    Hi! Lee,

    On 12 Sep 19 07:45, you wrote to me:

    Well no drama if you want dupes everywhere in your system.

    Yes. If nothing else, it means:

    1) people are still writing in Fidonet;
    2) you're still connected; and,
    3) your software still works.

    If all else fails, you could just read your dupe base instead of chopping & ->changing between echo areas all the time. Neat.

    What does "chopping & changing between echo areas all the time. Neat." mean?


    Cheers,
    Paul.

    ... Whenever I get a grip on reality, the handle falls off.
    --- GoldED+/LNX 1.1.5-b20130515
    * Origin: Quinn's Rock - Live from Paul's Xubuntu desktop! (3:640/1384)



    ...Take my advice, I don't use it anyway.
    ---BapStats Module (bsDBASE v6.1 Build 1)

    * Origin: TECHWARE BBS - Since 1995 - www.techware.dynip.com/ (1:102/401)
  • From Paul Quinn@3:640/1384 to LEE GREEN on Friday, September 13, 2019 12:24:58
    Hi! Lee,

    On 12 Sep 19 18:14, you wrote to me:

    What does "chopping & changing between echo areas all the time.
    Neat." mean?

    Your dupe base could be the -only- echo you need ever read again. Just reply as necessary via netmail. ;)

    Cheers,
    Paul.

    ... Broken pencils are pointless.
    --- GoldED+/LNX 1.1.5-b20130515
    * Origin: Quinn's Rock - Live from Paul's Xubuntu desktop! (3:640/1384)
  • From mark lewis@1:3634/12.73 to Terry Roati on Saturday, September 14, 2019 15:08:20

    On 2019 Sep 12 16:33:22, you wrote to LEE GREEN:

    [TOP POSTING FIXED]

    Did you know that your messages are posting twice?

    Do know if this is this happening every time and if it's in other echos as well?

    it happens in ALL echos that are multiply linked al la fidoweb style... dupes are a fact of FTN life these days and one's mail tosser must be able to detect dupes by control information within the messages and eliminate them... if the necessary control information is not available, multiple copies will be posted into the echos...

    suggest that lee green check his tosser's settings to eliminate duplicate messages when they arrive...

    )\/(ark

    Once men turned their thinking over to machines in the hope that this would set
    them free. But that only permitted other men with machines to enslave them.
    ... SMILE....You're on Candid Camera !!
    ---
    * Origin: (1:3634/12.73)
  • From Joe Martin@1:104/57 to Mark Lewis on Saturday, September 14, 2019 17:02:01
    it happens in ALL echos that are multiply linked al la fidoweb
    style... dupes
    are a fact of FTN life these days and one's mail tosser must be able
    dupes by control information within the messages and eliminate
    them... if the
    necessary control information is not available, multiple copies will
    into the echos...

    Just out of curiousity, what's the recommended way of dupe checking
    these days, MSGID, TID/PID, CRC, combination there of (including Msg
    Header)?

    Regards,

    Joe Martin

    --- ViaMAIL!/WC v2.00
    * Origin: ViaSoft Support BBS - Back online at 303-953-0568 (1:104/57)
  • From mark lewis@1:3634/12.73 to Joe Martin on Saturday, September 14, 2019 20:29:38
    On 2019 Sep 14 17:02:00, you wrote to me:

    Just out of curiousity, what's the recommended way of dupe checking
    these days, MSGID, TID/PID, CRC, combination there of (including Msg Header)?

    MSGID is the main way but older software doesn't generate MSGID so other methods need to be used...

    trying to root back through 20 years of memory, i recall one tosser takes the header + 20 or 40 bytes of the message body and runs that through a CRC32... that gets the header and generally most of the control lines... with this in mind, i specifically wrote my message posting tools to put the MSGID at the top
    of the control lines so it was pretty much guaranteed to be in that block of bytes that tosser grabbed for its CRC32... i do no recall any of the posts made
    by those tools being reported as dupes by any system in the network...

    other tossers take a more involved approach and CRC the header, CRC the control
    lines, and CRC the message body... maybe even MD5 instead of CRC... the problem
    then comes from those systems that mistakenly reformat the messages as they process them and write the reformatted messages to new PKTs... now the message body is different and will fail CRC/MD5 calcs...

    some systems specifically CRC the message body to avoid dupes that are identical even with different headers and control lines... this is apparent on systems that only get, for example, one posting of an echos rules each month and only accept new postings of those rules IF the message body is different than the last... i can understand this reasoning but it thwarts the idea of monthly rules postings... it also has problems when the message is purged for age but the CRC database retains the CRC of the purged message... so that results in there being no area rules in the area even though they are posted monthly... the only workaround is to add the posting date to the message body which is kinda rediculous because the header already contains that information... plus, in my case, there was a new MSGID in each posting as well... but those monthly posts were still duped out because of the body being the exact same...

    what i would do would be to ask other tosser devs what they use in their code...

    listed in no particular order:

    tobias burchhardt - fastecho
    rob swindell - sbbsecho
    nick andre - d'bridge
    vince coen - mbse's tosser
    kim heino - bbbs' tosser
    wilfred van velzen - fmail
    james coyle - mystic

    i'm sure there are others but these seem to be the most common ones being used today...

    )\/(ark

    Once men turned their thinking over to machines in the hope that this would set
    them free. But that only permitted other men with machines to enslave them.
    ... Clear out, or I'll set the cats on you
    ---
    * Origin: (1:3634/12.73)
  • From Paul Quinn@3:640/1384 to mark lewis on Sunday, September 15, 2019 13:00:57
    Hi! mark,

    On 14 Sep 19 20:29, you wrote to Joe Martin:

    i'm sure there are others but these seem to be the most common ones
    being used today...

    HPT isn't worth it? I thought so too. ;)

    Cheers,
    Paul.

    ... Two wrongs don't make a right. But three lefts do.
    --- GoldED+/LNX 1.1.5-b20130515
    * Origin: Quinn's Rock - Live from Paul's Xubuntu desktop! (3:640/1384)
  • From LEE GREEN@1:102/401 to mark lewis on Sunday, September 15, 2019 01:11:10
    On 2019 Sep 12 16:33:22, you wrote to LEE GREEN:

    [TOP POSTING FIXED]

    Did you know that your messages are posting twice?

    Do know if this is this happening every time and if it's in other echos
    as
    well?

    it happens in ALL echos that are multiply linked al la fidoweb style...
    dupes
    are a fact of FTN life these days and one's mail tosser must be able to
    detect
    dupes by control information within the messages and eliminate them... if
    the
    necessary control information is not available, multiple copies will be
    posted
    into the echos...

    suggest that lee green check his tosser's settings to eliminate duplicate messages when they arrive...

    It is on and catches a lot of dupes, WC and PXW are pretty good
    at detecting them.

    )\/(ark

    Once men turned their thinking over to machines in the hope that this would
    set
    them free. But that only permitted other men with machines to enslave them. ... SMILE....You're on Candid Camera !!
    ---
    * Origin: (1:3634/12.73)
    SEEN-BY: 1/120 14/6 15/0 18/0 19/36 34/999 90/1 104/57 106/201 116/18
    120/331
    SEEN-BY: 123/0 25 120 140 150 755 135/300 153/757 7715 218/700 222/2 230/150 SEEN-BY: 230/152 250/1 261/38 100 266/512 267/155 275/100 282/1031 1056
    291/1
    SEEN-BY: 291/111 300/4 320/119 219 340/400 342/13 396/45 640/1384 712/848 SEEN-BY: 801/161 189 3634/0 12 15 24 119 5020/1042 103/705 218/210 720 601
    401
    SEEN-BY: 218/520 102/401 218/802 109 215 410 1 10/1 218/0 10/0

    --- Platinum Xpress/Win/WINServer v3.1
    * Origin: TECHWARE BBS - Since 1995 - www.techware.dynip.com/ (1:102/401)
  • From Terry Roati@3:640/1321 to mark lewis on Sunday, September 15, 2019 22:15:12
    Mark,

    Based on the paths that Lee Green sent me for a dupe message, the common address is 1:218/700 if I understand it correctly. If so then 1:218/700 is not killing the dupe for whatever reason.

    1st dupe post.
    PATH: 640/1321 1384 712/848 633/280 229/426 280/464 103/705 218/700 ------------------------------------------------------------
    2nd dupe post
    PATH: 640/1321 1384 3634/12 261/38 218/700

    On Sep 14, 2019 03:12pm, mark lewis wrote to Terry Roati:

    suggest that lee green check his tosser's settings to eliminate
    duplicate messages when they arrive...

    Terry Roati - 3:640/1321 tfb-bbs.org

    ... Platinum Xpress & Wildcat!..... Nice!!!!
    --- Platinum Xpress/Win/WINServer v7.0
    * Origin: The File Bank BBS! (3:640/1321)
  • From mark lewis@1:3634/12 to Paul Quinn on Sunday, September 15, 2019 09:36:39
    Re: Double postings
    By: Paul Quinn to mark lewis on Sun Sep 15 2019 13:00:57

    i'm sure there are others but these seem to be the most common ones being used today...

    HPT isn't worth it? I thought so too. ;)

    oh, damn! i forgot about HPT from the husky group... yeah, that one should be included, also... it'll be easier to check how it works since it is FOSS and the code is easily acquired...

    funny thing is that i run it on my point system... i knew there was at least one or two more but the neurons were taking a break :/

    TBH, i actually like HPT a lot because of its dupe database style... i have found that i prefer the individual database per area instead of one for all areas... plus the database has a separate retention policy than the messages in
    the area... my point easily keeps up with 3 years of checksums/MSGIDs where very few other tossers can come close...


    )\/(ark
    --- SBBSecho 3.09-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From mark lewis@1:3634/12 to LEE GREEN on Sunday, September 15, 2019 09:39:15
    Re: Double postings
    By: LEE GREEN to mark lewis on Sun Sep 15 2019 01:11:10

    suggest that lee green check his tosser's settings to eliminate duplicate
    messages when they arrive...

    It is on and catches a lot of dupes, WC and PXW are pretty good
    at detecting them.

    sounds like you need to take a closer look at the dupes you are seeing, then...
    especially the hidden control lines... only with that information can we (TINW)
    attempt to figure out why/how you are getting dupes...

    this meta data needs to be gathered from all of the dupes you are seeing of any
    specific message so it can be compared to locate the differences...


    )\/(ark
    --- SBBSecho 3.09-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From mark lewis@1:3634/12 to Terry Roati on Sunday, September 15, 2019 09:43:41
    Re: RE: Double postings
    By: Terry Roati to mark lewis on Sun Sep 15 2019 22:15:12

    Based on the paths that Lee Green sent me for a dupe message, the common
    address is 1:218/700 if I understand it correctly. If so then 1:218/700 is not killing the dupe for whatever reason.

    that system /may/ kill dupes or it may pass them on to their links... it is up to the links to eliminate dupes on import into their message bases...

    i'm of two minds as to whether distributors should filter out dupes or pass them on... my own backbone hub system is configured to pass them on to my links
    and let them decide what to do with this... this seems to be the better choice in most cases... i've tried both ways and there were times that mail was lost for some reason... configuring to send the dupes on covered that situation and serves the purpose of having multiple links for an echo...


    )\/(ark
    --- SBBSecho 3.09-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From Robert Wolfe@1:261/20 to Mark Lewis on Monday, September 16, 2019 00:00:00
    what i would do would be to ask other tosser devs what they use in
    their code...

    listed in no particular order:

    tobias burchhardt - fastecho
    rob swindell - sbbsecho
    nick andre - d'bridge
    vince coen - mbse's tosser
    kim heino - bbbs' tosser
    wilfred van velzen - fmail
    james coyle - mystic

    i'm sure there are others but these seem to be the most common ones
    being used today...

    And if you can get in touch with him, Dales Barnes, of InterEcho and InterMail.



    --- InterEcho 1.20
    * Origin: Omicron Theta/WC4 * Memphis TN * wc4.winserver.org (1:261/20)
  • From Joe Martin@1:104/57 to Mark Lewis on Sunday, September 15, 2019 08:31:21
    Just out of curiousity, what's the recommended way of dupe checki
    these days, MSGID, TID/PID, CRC, combination there of (including Header)?

    MSGID is the main way but older software doesn't generate MSGID so
    other methods need to be used...

    My mailer/tosser uses a combined approach. If the message contains a
    MSGID then use its value, otherwise CRC the header and message body
    including control lines but never the SEEN-BY/PATH lines (considering
    they change all the time). The tosser never duplicates an MSGID either
    as it maintains a file with the last used value seeded upon creation by
    the current date/time. This prevents issues should that file get
    deleted.

    To provide speed and limit disk space, I also have an expiration
    mechanism (user configurable) that will purge CRC entries after a given
    amount of time (ie: 2 weeks but not more than 30 days). So while it's efficient catching dupes in that time period, if someone does a rescan
    and dumps everything back into the echo a month later, it won't catch
    them. It's a trade off, but back in the day when we had 40mb drives and 8088/80286 processors, it was extremely important.

    instead of CRC... the problem then comes from those systems that
    mistakenly reformat the messages as they process them and write the reformatted messages to new PKTs... now the message body is

    Yeah this is and always will be an issue.

    is apparent on systems that only get, for example, one posting of an
    echos rules each month and only accept new postings of those rules

    It would seem to me, (me mind you) that if you're moderating an echo,
    your software "should" be able to generate a MSGID to prevent this issue entirely. But hey...

    what i would do would be to ask other tosser devs what they use in
    their code...

    listed in no particular order:

    tobias burchhardt - fastecho
    rob swindell - sbbsecho
    nick andre - d'bridge
    vince coen - mbse's tosser
    kim heino - bbbs' tosser
    wilfred van velzen - fmail
    james coyle - mystic

    Thanks Mark...

    --- ViaMAIL!/WC v2.00
    * Origin: ViaSoft Support BBS - Back online at 303-953-0568 (1:104/57)
  • From mark lewis@1:3634/12.73 to Robert Wolfe on Sunday, September 15, 2019 11:25:32
    On 2019 Sep 16 00:00:00, you wrote to me:

    listed in no particular order:

    tobias burchhardt - fastecho
    rob swindell - sbbsecho
    nick andre - d'bridge
    vince coen - mbse's tosser
    kim heino - bbbs' tosser
    wilfred van velzen - fmail
    james coyle - mystic

    i'm sure there are others but these seem to be the most common ones being used today...

    And if you can get in touch with him, Dales Barnes, of InterEcho and InterMail.

    yup... i had forgotten that he is still around and working on those...

    )\/(ark

    Once men turned their thinking over to machines in the hope that this would set
    them free. But that only permitted other men with machines to enslave them.
    ... Thursday night - Potluck Supper. Prayer and medication to follow.
    ---
    * Origin: (1:3634/12.73)
  • From mark lewis@1:3634/12.73 to Joe Martin on Sunday, September 15, 2019 11:26:14
    On 2019 Sep 15 08:31:20, you wrote to me:

    MSGID is the main way but older software doesn't generate MSGID so other methods need to be used...

    My mailer/tosser uses a combined approach. If the message contains a MSGID then use its value, otherwise CRC the header and message body including control lines but never the SEEN-BY/PATH lines (considering
    they change all the time).

    this is good but for one small thing... there is a package that is known to be reformatting messages in transit which is going to throw the message body CRC out the door... there is no estimate on when this but will be fixed as the developer is apparently quite busy with RL outside of FTNs...

    The tosser never duplicates an MSGID either as it maintains a file
    with the last used value seeded upon creation by the current
    date/time. This prevents issues should that file get deleted.

    sounds similar to what my MSGID code does... i've shared that information with several folks... not sure if you were one of those or not... i still have the original 1994 (i think) post that described it, too :)

    To provide speed and limit disk space, I also have an expiration
    mechanism (user configurable) that will purge CRC entries after a given amount of time (ie: 2 weeks but not more than 30 days). So while it's efficient catching dupes in that time period, if someone does a rescan
    and dumps everything back into the echo a month later, it won't catch them. It's a trade off, but back in the day when we had 40mb drives and 8088/80286 processors, it was extremely important.

    yeah and that's gonna likely be a problem since the spec states three years... in this day in time, retaining three years worth of dupe detection data should be a small drop in the bucket of available drive space and processing power needed to perform a lookup...

    instead of CRC... the problem then comes from those systems that mistakenly reformat the messages as they process them and write the reformatted messages to new PKTs... now the message body is

    Yeah this is and always will be an issue.

    not if the message body is not CRC'd ;)

    i really like (IIRC) the d'bridge method of taking the header and first 40 bytes (i think) of the message body to get those few initial control lines and using that... that'll take care of the different dates as well as the MSGID but
    i would also grab the MSGID if it exists and store it in the database as well... basically i'm thinking of at least two or three fields in each record...

    is apparent on systems that only get, for example, one posting of
    an
    echos rules each month and only accept new postings of those rules

    It would seem to me, (me mind you) that if you're moderating an echo,
    your software "should" be able to generate a MSGID to prevent this issue entirely. But hey...

    that depends on the software used... some text file posting tools are really old and do not have any concept of MSGID... i'm thinking of the old Harvey's Robot in at least one case...

    what i would do would be to ask other tosser devs what they use in their code...

    listed in no particular order:

    tobias burchhardt - fastecho
    rob swindell - sbbsecho
    nick andre - d'bridge
    vince coen - mbse's tosser
    kim heino - bbbs' tosser
    wilfred van velzen - fmail
    james coyle - mystic

    Thanks Mark...

    you're welcome... i hope that you've also seen the other two posts about HPT and intermail which should also be added to the above list...

    )\/(ark

    Once men turned their thinking over to machines in the hope that this would set
    them free. But that only permitted other men with machines to enslave them.
    ... You may never know who's right but you always know who is in charge!
    ---
    * Origin: (1:3634/12.73)
  • From Alan Ianson@1:153/757 to mark lewis on Sunday, September 15, 2019 11:45:16
    this is good but for one small thing... there is a package that is known to be
    reformatting messages in transit which is going to throw the message body CRC out the door... there is no estimate on when this but will be fixed as the developer is apparently quite busy with RL outside of FTNs...

    I've seen this happen here with BBBS. Twice in the last month or so I noticed I
    had two copies of a message in the message base and BBBS would have sent them on to my links. The message didn't have a MSGID so it was tossed into the message base and on to links. It arrived a second time and had been changed in transit so it was not detected as a dupe and was tossed into the msgbase again and sent on to links.

    --- BBBS/Li6 v4.10 Toy-4
    * Origin: The Rusty MailBox - Penticton, BC Canada (1:153/757)
  • From Joe Martin@1:104/57 to Mark Lewis on Sunday, September 15, 2019 17:35:56
    And if you can get in touch with him, Dales Barnes, of InterEcho InterMail.

    yup... i had forgotten that he is still around and working on those..

    He's still around, we exchange email on a regular basis...

    Regards,

    Joe Martin

    --- ViaMAIL!/WC v2.00
    * Origin: ViaSoft Support BBS - Back online at 303-953-0568 (1:104/57)
  • From Joe Martin@1:104/57 to Mark Lewis on Sunday, September 15, 2019 18:19:40
    My mailer/tosser uses a combined approach. If the message contai
    MSGID then use its value, otherwise CRC the header and message
    body JM> including control lines but never the SEEN-BY/PATH lines (considering JM> they change all the time).

    sounds similar to what my MSGID code does... i've shared that
    information with several folks... not sure if you were one of those
    or not... i still have the original 1994 (i think) post that
    described it, too :)

    Don't recall for sure.

    mechanism (user configurable) that will purge CRC entries after a
    amount of time (ie: 2 weeks but not more than 30 days). So
    while it's JM> efficient catching dupes in that time period, if
    someone does a rescan JM> and dumps everything back into the echo a
    month later, it won't catch JM> them. It's a trade off, but back in
    the day when we had 40mb drives and JM> 8088/80286 processors, it
    was extremely important.

    yeah and that's gonna likely be a problem since the spec states three
    in this day in time, retaining three years worth of dupe detection da
    be a small drop in the bucket of available drive space and processing
    needed to perform a lookup...

    Fortunately, for today's systems, I can easily kick that number up
    without much fear. If I need to, I will. In the mean time, I'll keep
    an eye out.

    The nice part is that I have released a new version just a few weeks
    back and as such, everything is all up to date and ready to go should I
    need to release a new version. I even went so far as to write a custom
    builder application that provides UI to define a application,
    its associated executables, help files and so on which will compile the programs and create a ZIP file out of them and even put the correct time
    stamps on them. So the hard part of updates is no longer an issue.

    Yeah this is and always will be an issue.

    not if the message body is not CRC'd ;)

    The issue is with reformatting. I can see the software that replies to
    the message and the tosser that places the message in the message base
    doing reformatting, but not the tosser reformatting a message and
    then 'forwarding' it up/down stream. In my mind, that's taboo --
    just wrong on so many levels, can you say tampering...

    It would seem to me, (me mind you) that if you're moderating an
    echo, JM> your software "should" be able to generate a MSGID to
    prevent this issue JM> entirely. But hey...

    that depends on the software used... some text file posting tools are
    old and do not have any concept of MSGID... i'm thinking of the old H
    Robot in at least one case...

    I know, but being a moderator implies many things and running outdated
    software shouldn't be one of them. My two cents anyway.

    you're welcome... i hope that you've also seen the other two posts ab
    and intermail which should also be added to the above list...

    I did, thanks.

    Joe Martin

    --- ViaMAIL!/WC v2.00
    * Origin: ViaSoft Support BBS - Back online at 303-953-0568 (1:104/57)
  • From Terry Roati@3:640/1321 to mark lewis on Monday, September 16, 2019 11:26:38
    Hi Mark,

    It's a catch 22 then.

    Lee Green has done what he can to remove dupes but they are still coming through.

    Even if we know why these dupoes are getting through, it is highly unlikely that PXW will be updated in the near future so we have to live with the occasional dupe now that the fidoweb system is in place to as you said we ensure all mail gets through.

    Only other choice is to ask 1:218/700 if he can do anything

    that system /may/ kill dupes or it may pass them on to their links...
    it is up to the links to eliminate dupes on import into their message bases...

    i'm of two minds as to whether distributors should filter out dupes or pass them on... my own backbone hub system is configured to pass them


    Terry Roati - 3:640/1321 tfb-bbs.org

    ... Platinum Xpress & Wildcat!..... Nice!!!!
    --- Platinum Xpress/Win/WINServer v7.0
    * Origin: The File Bank BBS! (3:640/1321)
  • From mark lewis@1:3634/12.73 to Alan Ianson on Tuesday, September 17, 2019 09:12:24

    On 2019 Sep 15 11:45:16, you wrote to me:

    this is good but for one small thing... there is a package that is known
    to be reformatting messages in transit which is going to throw the message
    body CRC out the door... there is no estimate on when this but will be
    fixed as the developer is apparently quite busy with RL outside of FTNs...

    I've seen this happen here with BBBS. Twice in the last month or so I noticed I had two copies of a message in the message base and BBBS would have sent them on to my links. The message didn't have a MSGID so it was tossed into the message base and on to links. It arrived a second time and had been changed in transit so it was not detected as a dupe and was
    tossed
    into the msgbase again and sent on to links.

    yup! that's exactly how it has been happening in recent times...

    )\/(ark

    Once men turned their thinking over to machines in the hope that this would set
    them free. But that only permitted other men with machines to enslave them.
    ... Life is too short to stuff a mushroom.
    ---
    * Origin: (1:3634/12.73)
  • From Dale Barnes@1:106/201 to Robert Wolfe on Sunday, September 15, 2019 11:12:51
    what i would do would be to ask other tosser devs what they use
    their code...

    listed in no particular order:

    tobias burchhardt - fastecho
    rob swindell - sbbsecho
    nick andre - d'bridge
    vince coen - mbse's tosser
    kim heino - bbbs' tosser
    wilfred van velzen - fmail
    james coyle - mystic

    i'm sure there are others but these seem to be the most common
    ones-> being used today...

    And if you can get in touch with him, Dales Barnes, of
    InterEcho and InterMail.


    I am always here. Some would say Troll, some will say not really. I spend most of my time in the shadows working away at code that was lost then found again and had to remember how the **&* works. :)


    --- InterEcho 1.20
    * Origin: Home Of InterMail/InterEcho (1:106/201)
  • From Dale Barnes@1:106/201 to Joe Martin on Sunday, September 29, 2019 17:30:41
    And if you can get in touch with him, Dales Barnes, of
    InterEcho-> RW> InterMail.

    yup... i had forgotten that he is still around and working on those..

    He's still around, we exchange email on a regular basis...


    Still around and reading behind the scenes... O-O


    --- InterEcho 1.20
    * Origin: Home Of InterMail/InterEcho (1:106/201)
  • From Joe Martin@1:104/57 to Dale Barnes on Wednesday, October 02, 2019 06:28:48
    He's still around, we exchange email on a regular basis...


    Still around and reading behind the scenes... O-O

    Good to see you here...

    Joe Martin

    --- ViaMAIL!/WC v2.00
    * Origin: ViaSoft Support BBS - Back online at 303-953-0568 (1:104/57)
  • From Dale Barnes@1:106/201 to Joe Martin on Wednesday, October 02, 2019 22:03:02
    He's still around, we exchange email on a regular
    basis...-> >

    Still around and reading behind the scenes... O-O

    Good to see you here...

    Joe Martin


    Trying to be a bit more active. Glad to see you here as well.

    Lets try to have some fun for a change... :)


    --- InterEcho 1.20
    * Origin: Home Of InterMail/InterEcho (1:106/201)
  • From Joe Martin@1:104/57 to Dale Barnes on Saturday, October 05, 2019 12:54:34
    Still around and reading behind the scenes... O-O

    Good to see you here...

    Joe Martin


    Trying to be a bit more active. Glad to see you here as well.

    Lets try to have some fun for a change... :)

    Interesting that you mentioned having fun.

    My current project is a Door Manager type program that functions pretty
    much what one would think. I haven't looked at how folks did this in
    the past so I'm enjoying working through the various issues. At this
    point, I have no idea how it would compare.... and there's the fun in
    it.

    Regards,

    Joe Martin

    --- ViaMAIL!/WC v2.00
    * Origin: ViaSoft Support BBS - Back online at 303-953-0568 (1:104/57)
  • From Robert Wolfe@1:116/17 to Dale Barnes on Tuesday, October 08, 2019 16:17:00
    He's still around, we exchange email on a regular basis...

    Still around and reading behind the scenes... O-O

    Hit me up via private email at my GMail address -
    wolfe.robwolfe@gmail.com.
    ---
    ■ OLXWin 1.00b ■ Rumored Federal Reserve System will raise interest rates. I'm
    st.
    * Origin: Lean Angle BBS * Southaven MS * winserver.org (1:116/17)
  • From Robert Wolfe@1:116/17 to Joe Martin on Tuesday, October 08, 2019 16:18:00
    My current project is a Door Manager type program that functions pretty JM>much what one would think. I haven't looked at how folks did this in
    the past so I'm enjoying working through the various issues. At this JM>point, I have no idea how it would compare.... and there's the fun in
    it.

    What are you going to write it in?
    ---
    ■ OLXWin 1.00b ■ Famous last words: "I think the dragon's asleep "
    * Origin: Lean Angle BBS * Southaven MS * winserver.org (1:116/17)
  • From Joe Martin@1:104/57 to Robert Wolfe on Tuesday, October 08, 2019 19:04:59
    My current project is a Door Manager type program that functions p JM>much what one would think. I haven't looked at how folks did this
    the past so I'm enjoying working through the various issues. At t JM>point, I have no idea how it would compare.... and there's the fun
    it.

    BP7. Sometime back I recreated the console mode of WC4 as a stand-alone framework and my first implementation was for ViaADDR! (for ViaMAIL!
    v2.00). This includes all the chat modes, scrollback, keyboard toggles, security upgrades, statusbar size changes, complete reading/writing of
    all WC4 drop files and so on. Literally everything WC4 does including
    prompt files. So if you want to see it in action, install the new ViaMAIL!/ViaADDR!.

    So this will be my second implementation of the framework. Now I've
    rounded up as many drop file formats as I can find (don't have all of
    them though) and will do my best to implement them all.

    So, if you have your own collection of the specs for various drop
    file types, I'm all ears and will gladly accept whatever you're willing
    to part with. Hint hint...

    I've already written the configuration program and am working on the
    program as we speak. There's too much to talk about at this point but
    it should be a fun project and am enjoying it so far.

    Regards,

    Joe Martin

    --- ViaMAIL!/WC v2.00
    * Origin: ViaSoft Support BBS - Back online at 303-953-0568 (1:104/57)
  • From Robert Wolfe@1:261/20 to Dale Barnes on Thursday, October 10, 2019 22:26:29

    I am always here. Some would say Troll, some will say not really. I spen

    most of my time in the shadows working away at code that was lost then found DB>again and had to remember how the **&* works. :)


    Heh.

    --- InterEcho 1.20
    * Origin: Home Of InterMail/InterEcho (1:106/201)



    ---
    ■ OLXWin 1.00b ■ If you can't laugh at yourself, make fun of other people.


    --- InterEcho 1.20
    * Origin: Omicron Theta/WC4 * Memphis TN * wc4.winserver.org (1:261/20)
  • From Robert Wolfe@1:261/20 to Dale Barnes on Tuesday, October 15, 2019 21:47:48
    I am always here. Some would say Troll, some will say not really.
    I spend most of my time in the shadows working away at code that
    was lost then found again and had to remember how the **&* works. :)

    Heh, been there, done that! :) Glad to see you lurking in the echo!


    --- InterEcho 1.20
    * Origin: Omicron Theta/WC4 * Memphis TN * wc4.winserver.org (1:261/20)
  • From Robert Wolfe@1:261/20 to Joe Martin on Tuesday, October 15, 2019 21:49:12
    So this will be my second implementation of the framework. Now I've
    rounded up as many drop file formats as I can find (don't have all of
    them though) and will do my best to implement them all.

    So, if you have your own collection of the specs for various drop
    file types, I'm all ears and will gladly accept whatever you're
    willing to part with. Hint hint...

    I've already written the configuration program and am working on the
    program as we speak. There's too much to talk about at this point
    but it should be a fun project and am enjoying it so far.

    Cool. What would I need to do to get my hands on a fully functional copy of ViaUUCP? Also, who developed WildMail, et al?


    --- InterEcho 1.20
    * Origin: Omicron Theta/WC4 * Memphis TN * wc4.winserver.org (1:261/20)
  • From Joe Martin@1:104/57 to Robert Wolfe on Tuesday, October 15, 2019 19:24:10
    So this will be my second implementation of the framework. Now I' rounded up as many drop file formats as I can find (don't have all
    them though) and will do my best to implement them all.

    So, if you have your own collection of the specs for various drop
    file types, I'm all ears and will gladly accept whatever you're
    willing to part with. Hint hint...

    I've already written the configuration program and am working on t program as we speak. There's too much to talk about at this point
    but it should be a fun project and am enjoying it so far.

    Cool. What would I need to do to get my hands on a fully functional ViaUUCP? Also, who developed WildMail, et al?

    First and foremost, I need a valid email address so I can send it to
    you. I don't remember if I took the keys out of ViaUUCP!, but if not, I
    can just make you one.

    As for Wildcat!, Derek Koopowitz wrote v1-3 of Wildmail! and for v3
    Eric Cozzi wrote the configuration/utility programs (there was no
    configuration program for earlier versions).

    For Wildmail! v4, I rewrote everything from scratch. Nothing could be
    reused because WC4 had an all new message database structure/design.
    Wildmail! v4 was the first decent sized program I had ever written with
    WMINFO being my first mid-size one.

    Of course, ViaMAIL! was all me. It's a huge program, nearly twice the
    size of Wildcat! (not counting all the ancillary ones).

    Regards,

    Joe Martin

    --- ViaMAIL!/WC v2.00
    * Origin: ViaSoft Support BBS - Back online at 303-953-0568 (1:104/57)
  • From Dale Barnes@1:106/201 to Robert Wolfe on Tuesday, October 15, 2019 13:41:09
    I've already written the configuration program and am working on
    the-> program as we speak. There's too much to talk about
    at this point
    but it should be a fun project and am enjoying it so far.

    Cool. What would I need to do to get my hands on a fully
    functional copy of
    ViaUUCP? Also, who developed WildMail, et al?


    I think Joe worked on WildMail


    --- InterEcho 1.20
    * Origin: Home Of InterMail/InterEcho (1:106/201)
  • From Robert Wolfe@1:261/20 to Joe Martin on Monday, October 21, 2019 10:10:37
    First and foremost, I need a valid email address so I can send it to
    you. I don't remember if I took the keys out of ViaUUCP!, but if
    not, I can just make you one.

    Cool. robert.wolfe@robertwolfe.org should work now -- especially seeing that I
    am hosting it on a VPS.


    As for Wildcat!, Derek Koopowitz wrote v1-3 of Wildmail! and for v3
    Eric Cozzi wrote the configuration/utility programs (there was no configuration program for earlier versions).

    For Wildmail! v4, I rewrote everything from scratch. Nothing could be
    reused because WC4 had an all new message database structure/design. Wildmail! v4 was the first decent sized program I had ever written
    with WMINFO being my first mid-size one.

    Of course, ViaMAIL! was all me. It's a huge program, nearly twice
    the size of Wildcat! (not counting all the ancillary ones).


    Interesting piece of history here! :)


    --- InterEcho 1.20
    * Origin: Omicron Theta/WC4 * Memphis TN * wc4.winserver.org (1:261/20)