package lexer

import "github.com/azin-lang/Azin/internal/lexer"

Package lexer converts Azin source text into lexical tokens.

Index

• type Lexer
  • func New(file *source.File, diag *diagnostics.Engine) *Lexer
  • func (l *Lexer) Tokenize() []token.Token

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer breaks source code into tokens.

func New

func New(file *source.File, diag *diagnostics.Engine) *Lexer

New returns a new Lexer for the given file.

func (*Lexer) Tokenize

func (l *Lexer) Tokenize() []token.Token

Tokenize reads the entire file and returns its tokens.