I'm using this library in my Server News package (ronoaldo/minenews), and version 2.0 seems to break it for my Brazilian Portuguese files that I have hosted with my server. These were working fine in version 1.3.
This is not too big of a surprise, as i had to end up using a utf-8 hodgepodge library to get everything working, as the PEG based real lua markdown parsing library is designed for lua5.2+, and the hodgepodges i put together expect full lua 5.1, which we dont support for security reasons, which led to this current mess, which means mediocre to trash utf-8 support, but great markdown support.
There are two solutions for you:
The obvious, use 1.3 and fork, but that's a bad answer.
the better, but harder answer is that the utf-8 standin I'm using has to spell out every single codepoint manually, and i bet they missed the ones you are trying to use. I will open the repo to issues once I'm back in real civilization, and not on a mobile phone, and see if I can point you to the correct spot to test out fixing your specific code points.
Thanks for using the mod, Its hard to get this stuff right, and always nice knowing someone is using it
I had a problem with markdown files with bullet point lists for this reason, i think.
Your code translated them to formspecs just fine, but it inserted some zero-width-characters in them (in front of the bullet points), which caused the formspec to be displayed as a single "<Invalid UTF-8>" text in minetest (instead of as a wall of text as intended).
I think one easy fix for this common issue would be if your code manually replaced zero-width-separators with normal whitespace or removed them after the formspec code was generated.
Interesting, I have never seen that error before from the bullet points. What minetest version are you two running? That might point me in the right direction for testing.
As for the zero width separators, they're kinda a must, because tabs don't exist in minetest, becausespace is ignored mostly.
I used the function that takes a file name rather than a string, in case that's relevant.
(trying to print the resulting formspec into the console results in these <?> characters that indicate a letter that it didn't understand, in case that says anything about minetest's ability to handle these characters and isn't just console related)
It has been a while but today while updating my server mods I decided to dig deeper into this. The issue to me seems to be that \160 as a lua string does not equates to the UTF-8 non-breaking whitespace.
I did a little trick printing that char in python and copying it to the lua file and voilá. No errors anymore on client.
Sent a PR, kindly update the mod when you have a chance.
I'm using this library in my Server News package (ronoaldo/minenews), and version 2.0 seems to break it for my Brazilian Portuguese files that I have hosted with my server. These were working fine in version 1.3.
This is a sample file that breaks the rendering: https://github.com/ronoaldo/mercurio/blob/94d82aeaec736de1503b022aeac6a02e364b5ea1/news/news_pt_BR.md
I'm posting this here because I could not find any other way to submit issues on the Github repo.
Thanks for the work on this mod.
Hello Ronoaldo,
This is not too big of a surprise, as i had to end up using a utf-8 hodgepodge library to get everything working, as the PEG based real lua markdown parsing library is designed for lua5.2+, and the hodgepodges i put together expect full lua 5.1, which we dont support for security reasons, which led to this current mess, which means mediocre to trash utf-8 support, but great markdown support.
There are two solutions for you:
The obvious, use 1.3 and fork, but that's a bad answer.
the better, but harder answer is that the utf-8 standin I'm using has to spell out every single codepoint manually, and i bet they missed the ones you are trying to use. I will open the repo to issues once I'm back in real civilization, and not on a mobile phone, and see if I can point you to the correct spot to test out fixing your specific code points.
Thanks for using the mod, Its hard to get this stuff right, and always nice knowing someone is using it
I had a problem with markdown files with bullet point lists for this reason, i think.
Your code translated them to formspecs just fine, but it inserted some zero-width-characters in them (in front of the bullet points), which caused the formspec to be displayed as a single "<Invalid UTF-8>" text in minetest (instead of as a wall of text as intended).
I think one easy fix for this common issue would be if your code manually replaced zero-width-separators with normal whitespace or removed them after the formspec code was generated.
Interesting, I have never seen that error before from the bullet points. What minetest version are you two running? That might point me in the right direction for testing.
As for the zero width separators, they're kinda a must, because tabs don't exist in minetest, becausespace is ignored mostly.
minetest 5.4
I used the function that takes a file name rather than a string, in case that's relevant.
(trying to print the resulting formspec into the console results in these <?> characters that indicate a letter that it didn't understand, in case that says anything about minetest's ability to handle these characters and isn't just console related)
Awesome, will test. thanks!
It has been a while but today while updating my server mods I decided to dig deeper into this. The issue to me seems to be that \160 as a lua string does not equates to the UTF-8 non-breaking whitespace.
I did a little trick printing that char in python and copying it to the lua file and voilá. No errors anymore on client.
Sent a PR, kindly update the mod when you have a chance.
Awesome, will update when i have the chance. thanks!