• newsutil.js - content-type and binary usenet groups.

    From Ash-Fox@Ash-Fox@vert.synchro.net to DOVE-Net.Synchronet_JavaScript on Monday, November 19, 2007 13:00:39
    Greetings,

    I've noticed that Synchronet's NNTP server has 'weird' issues when mirroring binary groups. Upon further investigation I found that the 'default' content-type which was being inserted by newsutil.js was causing binary attachments in the messages to appear as text.

    writeln("Content-Type: text/plain; charset=IBM437");

    If I comment this line out, news readers suddenly don't have a issue
    figuring out there is a attachment in the message. I know this line is important as it assumes the messages are in the old IBM437 charset which is more compatible with BBS messages.

    I have read through RFC 2049 (http://www.ietf.org/rfc/rfc2049.txt) and try
    as might. No matter what content-type I specified, the messages did not
    display correctly.

    For this reason I'm trying to figure out a workaround that would work. Such
    as, somehow marking the messages that newslink imports
    as 'non-charsettable' or such. Unfortunately this isn't going too well and wondered if anyone had any ideas on this issue.

    ~Ash-Fox
  • From Digital Man to Ash-Fox on Monday, November 19, 2007 17:40:37
    Re: newsutil.js - content-type and binary usenet groups.
    By: Ash-Fox to DOVE-Net.Synchronet_JavaScript on Mon Nov 19 2007 01:00 pm

    Greetings,

    I've noticed that Synchronet's NNTP server has 'weird' issues when mirroring binary groups. Upon further investigation I found that the 'default' content-type which was being inserted by newsutil.js was causing binary attachments in the messages to appear as text.

    writeln("Content-Type: text/plain; charset=IBM437");

    The line is only inserted (from newsutil.js) if the original message does not contain a specified Content-Type header field. If you're mirroring between NNTP servers, then I would expect the original messages to have this Content-Type header field.

    If I comment this line out, news readers suddenly don't have a issue figuring out there is a attachment in the message. I know this line is important as it assumes the messages are in the old IBM437 charset which is more compatible with BBS messages.

    I have read through RFC 2049 (http://www.ietf.org/rfc/rfc2049.txt) and try as might. No matter what content-type I specified, the messages did not display correctly.

    For this reason I'm trying to figure out a workaround that would work. Such as, somehow marking the messages that newslink imports
    as 'non-charsettable' or such. Unfortunately this isn't going too well and wondered if anyone had any ideas on this issue.

    Can you check if the original messages (on USENET?) have the Content-Type header field or not?

    digital man (xbox-live: digitlman)

    Snapple "Real Fact" #39:
    An ant's smell is stronger then a dog's.
    Norco, CA WX: ---.-°F, --% humidity, NaN mph --- wind, --.-- inches rain/24hrs
  • From Ash-Fox@Ash-Fox@vert.synchro.net to Digital Man on Tuesday, November 20, 2007 04:13:26
    Digital Man wrote:

    writeln("Content-Type: text/plain; charset=IBM437");

    The line is only inserted (from newsutil.js) if the original message does
    not contain a specified Content-Type header field.

    I understood that from the comments in the code :)

    If you're mirroring
    between NNTP servers, then I would expect the original messages to have
    this Content-Type header field.

    Unfortunately, that isn't case with all messages.


    Can you check if the original messages (on USENET?) have the Content-Type header field or not?

    The messages that I've had problems with, do not have content-type headers.

    ~Ash-Fox
  • From Digital Man to Ash-Fox on Monday, November 19, 2007 23:50:05
    Re: Re: newsutil.js - content-typ
    By: Ash-Fox to Digital Man on Tue Nov 20 2007 04:13 am

    Digital Man wrote:

    writeln("Content-Type: text/plain; charset=IBM437");

    The line is only inserted (from newsutil.js) if the original message does not contain a specified Content-Type header field.

    I understood that from the comments in the code :)

    If you're mirroring
    between NNTP servers, then I would expect the original messages to have this Content-Type header field.

    Unfortunately, that isn't case with all messages.


    Can you check if the original messages (on USENET?) have the Content-Type header field or not?

    The messages that I've had problems with, do not have content-type headers.

    I'm not really sure what to suggest in that case. <shrug>

    digital man (xbox-live: digitlman)

    Snapple "Real Fact" #36:
    A duck's quack doesn't echo.
    Norco, CA WX: ---.-°F, --% humidity, NaN mph --- wind, --.-- inches rain/24hrs
  • From Ash-Fox@Ash-Fox@vert.synchro.net to Digital Man on Tuesday, November 20, 2007 19:14:19
    Digital Man wrote:


    I'm not really sure what to suggest in that case. <shrug>


    Well after a bit of thinking, I came up with this code, which appears to
    work well. It's a bit of hack though, and I'm not sure if this logic does enough to determine a message came from usenet.

    if(content_type==undefined) {
    //Check that we're not from Fidonet
    if(hdr.ftn_area==undefined) {
    //If we don't have a ftn, we're likely not from usenet, needs header.
    if(hdr.ftn_pid==undefined) { content_type_header(); }
    }
    //We're from fidonet, need a header.
    else { content_type_header(); }
    }
    }

    function content_type_header()
    {
    /* No content-type specified, so assume IBM code-page 437 (full ex-ASCII)
    */
    writeln("Content-Type: text/plain; charset=IBM437");
    writeln("Content-Transfer-Encoding: 8bit");
    }

    Do you think I need to add any other checks?

    ~Ash-Fox
  • From Digital Man to Ash-Fox on Tuesday, November 20, 2007 17:34:47
    Re: Re: newsutil.js - content
    By: Ash-Fox to Digital Man on Tue Nov 20 2007 07:14 pm

    Digital Man wrote:


    I'm not really sure what to suggest in that case. <shrug>


    Well after a bit of thinking, I came up with this code, which appears to work well. It's a bit of hack though, and I'm not sure if this logic does enough to determine a message came from usenet.

    if(content_type==undefined) {
    //Check that we're not from Fidonet
    if(hdr.ftn_area==undefined) {
    //If we don't have a ftn, we're likely not from usen
    if(hdr.ftn_pid==undefined) { content_type_header();
    }
    //We're from fidonet, need a header.
    else { content_type_header(); }
    }
    }

    function content_type_header()
    {
    /* No content-type specified, so assume IBM code-page 437 (full ex-A */
    writeln("Content-Type: text/plain; charset=IBM437");
    writeln("Content-Transfer-Encoding: 8bit");
    }

    Do you think I need to add any other checks?

    The correct way to check if a message came from a network (any network) is to see if hdr.from_net_type!=NET_NONE. If you specifically want to see if the message came from the Internet (e.g. USENET), then check if hdr.from_net_type==NET_INTERNET.

    digital man (xbox-live: digitlman)

    Snapple "Real Fact" #72:
    The average person uses 150 gallons of water per day for personal use.
    Norco, CA WX: ---.-°F, --% humidity, NaN mph --- wind, --.-- inches rain/24hrs
  • From Ash-Fox@Ash-Fox@vert.synchro.net to Digital Man on Wednesday, November 21, 2007 04:41:43
    Digital Man wrote:


    The correct way to check if a message came from a network (any network) is
    to see if hdr.from_net_type!=NET_NONE. If you specifically want to see if
    the message came from the Internet (e.g. USENET), then check if hdr.from_net_type==NET_INTERNET.

    Ah! Much easier and less prone to error.

    Thanks for the help, DM.

    ~Ash-Fox