Parser error when modifying the monsters file

[Gauss]

Originally Posted by Pete Mack

Pretty common actually--I was guessing BOM too, but according to the notepad++ docs, the default encoding is supposed to apply only to new docs, not existing ones. In retrospect, I suspect what happened is that 'save as' is interpreted as a new doc.

Notepad normally uses ANSI encoding, so if it reads the file as UTF-8 then it has to guess the encoding based on the data in the file. If you save a file as UTF-8, Notepad will put the BOM (byte order mark) EF BB BF at the beginning of the file. Notepad makes an educated guess i think.

[wobbly]

Nice, glad someone who knows more about these things got there first.

Originally Posted by Pete Mack

Pretty common actually--I was guessing BOM too, but according to the notepad++ docs, the default encoding is supposed to apply only to new docs, not existing ones. In retrospect, I suspect what happened is that 'save as' is interpreted as a new doc.

In preferences, new document, encoding. the default is UTF-8 with a little tick box "Apply to open ANSI files" defaulting to on. Presumably if you hit save as it will save it in UTF-8.

[Tibarius]

Some more details eventually interesting for window users:
An unzipped Angband text file in /lib/gamedata (version 4.1.3)
has a disc size of 4096 or a multiple, so the operating system used to create the zip file most likely uses a block write / read disc driver functionality (because it is much faster than reading/writing single chars).

Example: p_race.txt has a size of 4621 bytes, and a disc size of 8192 bytes

The text files have no Byte order Mark, they are encoded in the ANSI text format. So if you want to modify them, make sure you save with ANSI encoding.

Wordpad: does that automatically
Editor: you can choose the type (ANSI / unicode / UTF-8) if you Save as
Notepad++: requires to be set to ANSI (even if you open an ANSI file with notepad ++ the predefined save option is UTF-8 encoding)

The Editor has some kind of problems saving monster.txt in ANSI encoding, afterwards Angband comes up with an parse error. The size of the file changed, but i didn't bother to analyze what exactly changed.

Also noteable ... the original files use Hex '0A' (decimal 10) as Newline character. So opening with an Editor that does not add a '0D' carriage return to it (for example the simple Editor) will render the file almost unreadable, because the number of chars per line is 1024 in that case.

Notepad++ can handle this format and shows readable lines. And even if you save it in ANSI format, there are still only 0As as end of line characters.

Wordpad does add an hex '0D' character at the end of each line. In the case of p_race.txt the file size increases to 4879 bytes.

Angband application can parse both 0A alone as well as 0D 0A.

Conclusion: Best editor of those 3 for me is notepad++, does not change file size by addind 0D chars, has line numbers and you can choose which encoding to use if you save the file. The only drawback, it always uses UTF-8 as predefined encoding, even if you open an ANSI encoded file.