The unicode standard has what's call a Byte-Order-Mark in the different encodings (UTF-8, UTF-16, UTF-32). In the other encodings (16, 32) the BOM actually does its job and shows in what endianness the stream is encoded. This does not matter for UTF-8, so the only job of the UTF-8 BOM is to show that the stream is encoded in UTF-8 and not UTF-16/32/latin1/... Some editors create this BOM automatic, and other editor just don't render it, so the user is not always sure whether or not it exists.

As example, the start of a file with UTF-8 BOM might look like this (in hex):

EF BB BF 23 23 20 4C ....

The EF BB BF part is the BOM, the rest is the actual content of the file.

What @jayvdb now proposes is that this BOM gets ignored automatically in lark, even if only optionally. I am currently working on a PR, but I don't know enough about the way configs are done in Lark to know how to implement the API.

I would propose that a new keyword-only argument get's added to the Lark class, which then get's somehow passed down to lexer.Lexer where approximately the following code get's run at the beginning of the stream:

# global constant
_BOM = '\xEF\xBB\xBF'

# somewhere, probably in TraditionalLexer.lex and ContextualLexer.lex
        if self.ignore_bom and stream.startswith(BOM):
            stream = stream[len(_BOM):]

The primary problem with this is that the indicies will not match up quite from a python point of few, but will match up from a user/editor point of view. (or, depending on the exact implementation, the other way around)

Uh oh!

Skip BOM (Byte order mark) #407

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions