1. Forum
  2. DOVE-Net
  3. Unix

  • Encoding issue

    From Nelgin@EOTLBBS to All on Wednesday, August 30, 2023 12:42:36
    Hi all,

    I have a problem with character encoding and wonder if someone who deals with this sort of thing more often than I do can help.

    On my linux system I have two files that report as "Unicode text, UTF-8 text" when using file.

    While one file correctly displays é (e with an acute accent) the other file displays an A with a tilde above followed by the copyright symbol.

    Obviously, other characters are also displayed incorrectly.

    I've tried various iterations of iconv to try and correct the output of the misprinting file but with no success.

    A hexdump shows that the incorrectly displaying file has the following
    c3 83 c2 a9

    Whereas the correctly displayed file has
    c3 a9

    So, I'm open to suggestions on how to fix this using some native program rather than having to do search and replace.

    Thanks,
    ---
    ■ Synchronet ■ End Of The Line BBS - endofthelinebbs.com
  • From Digital Man to Nelgin on Wednesday, August 30, 2023 11:39:30
    Re: Encoding issue
    By: Nelgin to All on Wed Aug 30 2023 12:42 pm

    Hi all,

    I have a problem with character encoding and wonder if someone who deals with this sort of thing more often than I do can help.

    On my linux system I have two files that report as "Unicode text, UTF-8 text" when using file.

    While one file correctly displays é (e with an acute accent) the other file displays an A with a tilde above followed by the copyright symbol.

    Obviously, other characters are also displayed incorrectly.

    I've tried various iterations of iconv to try and correct the output of the misprinting file but with no success.

    A hexdump shows that the incorrectly displaying file has the following
    c3 83 c2 a9

    That sounds correct. See table here: https://www.utf8-chartable.de/unicode-utf8-table.pl

    Whereas the correctly displayed file has
    c3 a9

    That's also correct.

    So, I'm open to suggestions on how to fix this using some native program rather than having to do search and replace.

    More background is needed with the problem here as it sounds like both files contain the correct UTF-8 sequence for the Unicode codepoints you're saying are being displayed.
    --
    digital man (rob)

    Synchronet "Real Fact" #20:
    Michael Swindell was directly responsible for Synchronet's commercial success Norco, CA WX: 93.0°F, 33.0% humidity, 1 mph ESE wind, 0.00 inches rain/24hrs