https://ssalewski.de/nimprogramming.html

Computer Programming with the Nim Programming Language

A gentle Introduction (c) Dr. Stefan Salewski 2020, 2021, 2022
Table of Contents

  * About this book
  * Part I: Introduction
      + What is a Computer?
      + Analog and Digital
      + What is an Operating System?
      + What is Computer Programming?
      + What is a Computer Program?
      + What is an Algorithm?
      + What is a Programming Language?
      + Compilers and Interpreters
      + Types of Programming Languages
      + Why Nim?
          o Some Facts About Nim
          o Nim supports many programming styles
          o Nim is Efficient
          o Nim is Expressive and Elegant
          o Nim is Open and Free
          o Nim has a friendly and helpful growing community
          o Nim has a encouraging future
          o Why is Nim not a popular mainstream language yet?
          o Is Nim a good choice as first language for a Beginner?
          o Is Nim really a good teaching language?
          o So is Nim really the best start for me?
          o After learning Nim, will I still have to learn other
            programming languages?
          o Why should I not use Nim?
      + Our first Nim Program
      + Binary Numbers
      + Hexadecimal Numbers
      + Installation of the Compiler
      + Creation of Source Code Files
      + Launching the compiler and running the program
  * Part II: The Basics
      + Declarations
      + Statements
      + Input and Output
      + Data types
          o Integer types
          o Floating point types
          o Distinct types
          o Subrange types
          o Enums
          o Boolean types
          o Characters
          o Ordinal types
          o Sets
          o Strings
          o Comments
          o Other data types
      + Nim Source Code
          o Blocks, Scopes, Visibility, Locality and Shadowing
          o Global code
          o White space, punctuation and operators
          o Operators
          o Order of Execution
      + Control Structures
          o If Statement and If Expression
          o The When Statement
          o The Case Statement
          o The While Loop
          o The Block Statement
          o For Loops and Iterators
      + Objects
      + Arrays and Sequences
          o Some details
      + Slices
      + Value Objects and References
      + References and Pointers
          o Introduction to Pointers
          o Pointer Arithmetic
          o Allocating Objects
          o References to Objects
      + Procedures and Functions
          o Introduction
          o Special Argument Types: OpenArray and Varargs
          o Procedures bound to a Data Type
          o Scoping , Visibility and Locality
          o Generics
          o Example for the use of Generics
          o Method Call Syntax
          o Procedure Variables
          o Nested Procedures and Closures
          o Anonymous Procedures
          o Compile time proc execution
          o Inlining Procedures
          o Recursion
          o Converters
      + Object-Orientated Programming and Inheritance
      + Other Builtin Data Types
          o Tuple Types
          o Object Variants
      + Iterators
      + Templates
          o Typed vs untyped parameters
          o Passing a code block to a template
          o Passing operators to templates
          o Advanced template use
      + Casts and Type Conversion
      + Bitwise Operations
      + Exceptions
      + Destructors
          o Destructors and Inheritance
      + Finalizers
      + Modules
          o Cyclic Imports
      + Include
  * Part III: The Nim Standard Library
      + Command Line Arguments
      + Reading data from the terminal
      + Writing text to the terminal window
      + Serialization -- storing data permanently on external storage
      + Streams and Files
          o Files
          o Streams
      + String Processing
          o Basic string operations
          o Module stringutils
          o Module parseutils
          o Module strscans
          o Module strformat
      + Arrays and Sequences
          o Module sequtils
      + Random Numbers
      + Timers
      + Hash Tables
          o User defined hash values
          o Equality and Identity
          o Performance
          o Tuples or other containers as Keys
      + Regular Expressions
          o Greedy matching
          o Escape sequences
          o Final remarks
  * Part IV: Some Programming Tasks
      + Sorting
          o Selection Sort
          o Insertion Sort
          o Quick Sort
          o Merge Sort
      + Some small exercises
          o Removing adjacent duplicates
          o Array difference
          o Binary search
          o Integer to string conversion
          o No Game Programming?
  * Part V: External Packages
      + Parsing Expression Grammars
          o Capturing data
  * Part VI: Advanced Nim
      + Macros and Meta-Programming
          o Introduction
          o Types of macro parameters
          o Quote and the quote do: construct
          o Building the AST manually
          o The Assert Macro
          o Pragma Macros
          o Pragma Macro for Iterator
          o Macros to generate new operator symbols
      + Process execution
          o Module threadpool
          o Using the threads module to create new threads
          o Using Channels for communication between Threads
          o Race conditions
          o Guards and Locks
          o Exceptions in Threads
      + Code execution with async/await
          o Is async/await faster than multi-threading?
          o Nim's asynchronous dispatcher
          o Asynchronous procedures
          o Simple example
          o File download
          o A chat server application
          o The client application
      + Concepts
  * Appendix
      + Acknowledgments
      + ASCII Table
      + Div and Mod operation
      + Text styles
      + ChangeLog
          o November 2021
          o Februar 2022

    When you are not able to explain it with words, you may have to
    add pictures. And when you even do not manage it with pictures,
    you can still make a video.

About this book

In the year 1970 Prof. Niklaus Wirth invented the Pascal programming
language to teach his students the fundamentals of computer
programming. While the initial core Pascal language was designed for
teaching purposes only, it was soon expanded by commercial vendors
and got some popularity. Later, Wirth presented the language Modula-2
with improved syntax and support of modules for larger projects, and
the Oberon language family with additional support for 
Object-Oriented Programming.

The Nim programming language can be seen in this tradition, as it is
basically an easy language suited for beginners with no prior
programming experience, but at the same time is not restricted in any
way. Nim offers all the concepts of modern and powerful programming
languages in combination with high performance and some sort of
universality -- Nim can be used to create programs for tiny
microcontroller as well as large desktop apps and web applications.

Most books about programming languages concentrate on the language
itself and assume that the reader is already familiar with the
foundations of computer hardware and already has some programming
experience. This is generally a valid approach, as today most people
are taught this fundamental knowledge, sometimes called Computer
Science (CS) in school. But still, there are people who missed this
introduction in school for various reasons and decide later that they
need some programming skills, maybe for a technical job. And there
may exist some children that are not satisfied with the introduction
to computer science taught at school. So we have decided to start
this book with a short introduction to fundamental concepts -- most
people can skip that part. In part II we explain the basics of
computer programming step by step in a way which should enable even
children to learn independently. In this part we may repeat some of
the stuff which we already mentioned in part I. We do that by intent,
as some people may skip part I, and because it is generally not a bad
idea to support the learning process of the reader with some
repetitions. Part III will give you an overview of the Nim standard
library, and part IV will introduce some useful external packages.
Part V will introduce advanced concepts like asynchronous operations,
threading and parallel processing, and macros and meta-programming.
Nim macros are very powerful but difficult at first. Part VI may
finally present some advanced examples.

This book is basically a traditional text book, a very simple one
with detailed explanations, so that kids from 14 years upwards can
read and understand it on their own, with no, or only minimal help
from adults. The English language may still be a problem for many
kids not born in a country with a good English language tradition
unfortunately. But luckily automatic translations are already
supported for some languages, and maybe we will be able to offer some
translated editions later, maybe a Chinese and a German translation?

In the last decades for the area of computer programming traditional
text books have partly been replaced by videos and "Crash course" and
"Learning by doing" books. Well, maybe a good video may indeed help
you starting with a new language, and a video may enable people with
problems reading printed texts or with problems concentrating on a
topic for a few minutes to learn a programming language.
Unfortunately the quality of most videos is very bad, some are made
by kids just having learned the first steps of computer programming
themselves. And of course watching videos does not improve the
reading and concentration problems that people have. "Crash course"
and "Learning by doing" books may give you a good start, but for that
we already have a lot of textual tutorials. The problem with these
kinds of books is, that they may help you with solving some common
tasks, but they do not really support a deeper understanding. The
idea of a "Crash course" and "Learning by doing" may not be that bad
in general, but in computer science starting with a larger example
application may be a overwhelming process, as you have to learn a lot
of stuff in parallel. It may work for you, but there is the danger
that you forget all the details very quickly again. And these kinds
of books are not very helpful when you have to look up something. The
other problem with "Learning by doing" in computer science is, that
learning materials may have only examples which you may not really be
interesting in: Of course we can create a simple chat application, a
simple Twitter clone and do some basic web scraping using async/
await. Or create a basic game, or a simple GUI with one of the dozen
available tool-kits. But what when you are not interested in chatting
and twittering, and that single selected toolkit? We think that for
such a case, reading the detailed examples can be very frustrating.
So we would recommend that after reading the first tutorial, and
maybe a few pages of this book, you just start coding with stuff you
are interested in. Maybe together with some friends? Whenever you
should need some concrete help, you should find it on the internet,
using search engines, Wikipedia or a discussion platform of your
choice. And if you have really no idea whatsoever for a project with
which you can start, then maybe computer programming is just not the
right profession for you.

While Nim has a JavaScript backend and so supports web related
development well, this book concentrates on native code generation
using the C and C++ backends. We will discuss some peculiarities of
the JavaScript backend in the second half of the book, and we may
give some complete examples for the use of the JavaScript backend in
the final part VI of the book. If you are strongly interested in web
development and the JavaScript backend, then you may also consult the
book Nim in Action of Dominik Picheta, which gives some detailed
examples for the development of web based software with the Nim
programming language, including a simple chat application and the
skeleton of a microblogging and social networking service. And you
may consult the tutorials and manuals of Nim web packages like Karax,
Jester or basolato.

This book will not try to explain things that are already explained
well elsewhere or that should have been explained well elsewhere -- at
least not in this first edition, where we still have so much other
important stuff. So what we will leaf out for now is the installation
of the compiler, installing and using text editors or IDEs with
special Nim support, using Nim package managers like nimble, nimph or
others, using the foreign function interface (FFI) to create bindings
to C libraries, and internal compiler details like the various memory
management options, all the pragmas and similar. Also we do not
intend to fill the book up with redundant stuff, like tables listing
all the Nim keyword or Nim's primitive data types and such, as you
can find all that in the compiler manual easily.

While creating graphical user interfaces (GUIs) is an important
topic, we can not give much details for that due to various reasons:
Nim has not yet the one and only accepted GUI library, but more than
20 attempts -- from pure Nim ones like xnim or fidget, over wrapped
libs like GTK or QML, or GUIs that try to provide a native look for
various operating systems like xwidgets or nigui, to web based GUIs.
And for each of these, at least for the more serious ones, we could
write a separate GUI book. So we will give only a few minimal
examples for some of them in part IV of the book.

Also we will not explain game programming, as game programming is a
broad area, and there are already a lot of tutorials available. Maybe
in later editions of the book we will add some of this topics, e.g.
game programming, as so many people like it. But we will always have
to ensure that a possible printed book version will not get more than
500 pages, so we may then leave out some stuff in the printed
version.

General when learning a new programming language, people start with
some short tutorials before really learning the language following a
book. This way is indeed a good start. So we recommend you to read
the short official tutorials part 1 and 2 and maybe also some of the
other tutorials freely available online. Tutorials generally only
scratch the topics, so you may not be able to understand all really
well, but this way you get already a feeling for the language. There
exists also some video tutorials, for the case that you have problems
reading, but in that case this book will not be of much use for you
at all. When you know already some computer science and have already
experience with other languages like C++, Haskell or Rust, then the
tutorials and the Nim compiler manual may be fully sufficient for
you, and you may not need this book at all.

This book is based on the Nim reference implementation of the team
around Mr. A. Rumpf. While the first pages of this book have been
written already in spring 2020, it should be mostly up to date with
the current stable version Nim v1.6. We will try to keep the book up
to date with further Nim releases, as a 1.8 or maybe already a 2.0
release with planned support for incremental compilation.

The source code of the book is hosted at https://github.com/
StefanSalewski/NimProgrammingBook. You may use the Github issue
tracker to point us to mistakes or unclear explanations, we will try
to fix that. Please note that we are more interested in remarks to
the content of the book currently, not that much in typos or grammar
issues. Before the book will be officially published or a printed
version will be created, we will run it through some correction
software or hire a professional proof reader.

New sections, and serious content changes are now (end of 2021)
marked with a yellow background, and not that new stuff is still
marked with a light yellow. For details you may see the change log in
the appendix.

About the Author

Dr. S. Salewski studied Physics, Mathematics and Computer Science at
Hamburg University (Germany), where he got his PhD in 2005 in the
area of laser physics. He has worked in the field of fiber laser
physics, electronics, and software development, using languages like
Pascal, Modula-2, Oberon, C, Ruby and Nim. Some of his software
projects, including the Nim GTK GUI bindings and Nim implementations
of an N-dimensional RTree and a fully dynamic, constrained delaunay
triangulation are freely available as open source projects at https:/
/github.com/StefanSalewski.

Part I: Introduction

For using computers and writing computer programs, you initially do
not have to know many details. It is basically like driving a car:
Although a car is a powerful and complicated tool, kids generally can
drive it after a 3-minute introduction. Still, good racing drivers
generally need a much broader insight into the inner working of all
the technical components, and finally, a lot of practice.

What is a Computer?

A computer is primarily a device which can run computer programs, by
following instructions about how to manipulate data.

Nearly all of the computers currently in use, from the tiny ones
integrated in electronic gadgets, the well known desktop computers
(PCs), to large and powerful super computers filling out entire
rooms, work internally with digital data only.^[1] Digital data are
basically integer (whole) numbers encoded in binary form, which are
represented by sequences of the symbols 0 and 1. We will discuss the
term digital in the next section in more detail.

The most important part of a digital computer is the CPU, the Central
Processing Unit. That tiny device is built of digital electronic
circuits and can perform very basic mathematical and logical
operations on numbers, like adding two numbers or deciding if a
number is larger or smaller than another number. Most computer CPU's
can only store very few numbers internally, and forget the numbers
when the power is switched off. So the CPU is generally electrically
connected to a RAM module, a Random Access Memory, which can store
many more numbers and allow fast access to these numbers, and to a 
Harddisk or SSD device which can permanently store the numbers, but
does not allow such fast access. The stored numbers are most often
called just data -- basically that data is nothing more than numbers,
but it can be interpreted in many different ways, such as pictures,
sounds and much more.

The traditional hard disk drives (HDD), which store data
electro-mechanical on rotating magnetic disks, as well as the more
modern variants, the solid-state-devices (SDD), which store data
using modern semiconductor technologies, can store data persistently
for longer time periods, even when no electric power supply is
available. Both, SSDs and HDDs, can be optionally split into multiple
partitions, e.g. one or multiple OS partitions for executable
programs or pure data partition for passive data like text files or
pictures. Before used, each partition is generally formatted and a
file system (FS) is created. These two steps create an internal
structure on the storage device, which allows us to store and
retrieve individual data blocks like programs, text files or
pictures.

Nearly all of today's desktop computers, and even most notebooks and
cellphones contain not only a single CPU, but multiple CPUs, also
called "Cores", so they can run different programs in parallel, or a
single program can run parts of it on different CPUs, to increase
performance or reduce total execution time. The so called super
computers can contain thousands of CPUs. Beside CPUs most computers
have also at least one GPU, a Graphic Processing Unit, that can be
used to display data on a screen or monitor, maybe for doing
animations in games or for playing video. The distinction between CPU
and GPU is not really sharp; generally a CPU can also display data on
screens and monitors, and GPUs can do also some data processing that
CPUs can do. But GPUs are optimized for the data display task.

More visible to the ordinary computer user are the peripheral devices
like keyboard, mouse, screen and maybe a printer. These enable human
interaction with the computer, but are in no way a core component of
it; the computer can run well without them. In notebook or laptop
computers or in cellphones, the peripheral devices are closely
integrated with the core components. All the physical parts of a
computer are also called hardware, while the programs running on that
hardware are called software.

A less visible but also very important class of computers are 
microcontroller and so called embedded devices, tiny pieces with
generally a hull of black plastic with some electrical contacts. The
devices generally contain all necessary elements, that is the CPU,
some RAM and a persistent storage that can store programs when no
electric power supply is available. These devices may be restricted
in computing power and the amount of data that they can store and
process, but they are contained in many devices. They control your
washing machine, refrigerator, television and radio and much more.
Some devices in your home may even contain multiple microcontrollers
and often the microcontrollers can already communicate with each
other by RF (Radio-Frequency), or access by WLAN the internet, which
is sometimes called Internet of Things (IoT).

Another class of large and very powerful digital computers are called
mainframe computers or super computers, which are optimized to
process large amount of data very fast. The key to their gigantic
computing power is that many fast CPUs work in parallel -- the problem
or task is split into many small parts that are solved by one CPU
each, and the final result is then the combination of all the solved
sub-tasks. Unfortunately it is not always possible to split large
problems into smaller sub-tasks.

Digital computers are generally driven by a clock signal that pulses
at a certain frequency. The CPU can do simple operations like the
addition of two integers at each pulse of the clock signal. For more
complicated operations like a multiplication or a division it may
need more clock pulses.

So a rough measure for the performance of a computer is the clock
rate, that is the number of clock pulses per second, divided by the
number of pulses that the CPU needs to perform a basic operation,
multiplied by the number of CPUs or Cores that the computer can use.

A totally different kind of computers are Quantum Computers, large,
expensive high-tech devices, which use the rules of quantum mechanics
to calculate many computations in parallel. Today only a few of them
exist, for research at universities and some large commercial
institutes. Quantum computers may at some time in the future
fundamentally change computing and our whole world, but they are not
the topic of this book.

Analog and Digital

Whenever we measure a quantity based on one tiny base unit, then we
work in the digital area, we measure with some granularity. Our
ordinary money is digital in some way, as the cent is the smallest
base unit; you will never pay a fraction of a cent for something.
Time can be seen as a digital quantity as long as we accepts the
second as the smallest unit. Even on so called analogue watches the
second hand will generally jump forwards in steps of a second, so you
can not measure fractions of a second with that watch.

An obvious analogue property is the thermodynamic temperature and its
classic measurement device is the well known capillary thermometer
consisting of a glass capillary filled with alcohol or liquid
mercury. When temperature increases the liquid in a reservoir expand
more than the surrounding glass and partly fills the capillary. That
filling rate is an analogue measure for the temperature.

While the hourglass works digitally (you can count the tiny sand
stones), the sundial does not.

Most of the quantities in our real world seem analog, and digital
quantities seem to be some sort of arbitrary approximation.

But quantum mechanics has taught us that many quantities in our world
really have a granularity. Physically quantities like energy or
momentum are indeed multiplies of the tiny planck constant. Or
consider electric charge, which is always a multiple of the 
elementary charge unit of one electron. Whenever an electrical
current is flowing through an electrically conducting wire, an
ionized gas or an electrolyte like salt water, there are flowing
multiplies of the elementary charge only, never fractions of it. And
of course light and electromagnetic radiation also has some form of
granularity, which the photoelectric effect as well as compton
scattering proves.

An important and useful property of digital signals and digital data,
is that they map directly to integral numbers.

The simplest form of digital data is binary data, which can have only
two distinct values. When you use a mechanical switch to turn the
light bulb in your house on, or of, you change the binary state of
the bulb. And your neighbor, when watching your house, receives
binary signals.^[2]

Digital computers are generally using binary electric states
internally -- voltage or current on or off. Such an on/off state is
called a bit. We will learn more about bits and binary logic later.
One bit can store obviously only two states, which we may map to the
numbers 0 and 1. Larger integer numbers can be represented by a
sequence of multiple bits.

The morse code was an early application to transmit messages encoded
in binary form.

A very important property of digital encoded numbers (data) is that
they can be copied and transmitted exactly without loss of precision.
The reason for this is that digital numbers have a well defined clean
state, there is no noise which overlays the data and may accumulate
when the data is copied multiple times. Well, that statement is not
really true -- under bad conditions the noise can become so large that
it changes the binary state of signals. Imagine we try to transfer
some whole numbers encoded in binary form, maybe by binary states
encoded as voltage level 0 Volt and 5 Volt, over an electric wire and
a long distance. It is clear that the long wire can pick up some
electromagnetic noise that can change the true 0 Volt data to a
voltage that is closer to 5 Volt than to the true 0 Volt level, so it
is received incorrectly. To catch such types of transmission errors 
checksums are added to the actual data. A checksum is derived by a
special mathematical formula from the original data and transferred
with it. The receiver applies the same formula to the received data
and compares the result with the received checksum. If it does not
match, then it is clear that data transmission is corrupted, and a
resend is requested.

The opposite of digital is generally called analogue, a term which is
used for data which have or seems to have no granularity. For example
we speak of an analogue voltage when the voltage can have each value
in a given range and when the voltage does not "jump" but change
continuous.^[3] For observing analogue voltages or currents, one can
use a moving coil meter, a device where the current flows through a
coil in a magnetic field and the magnetic force moves the hand/
pointer.

We said in the previous section that nearly all of our current
computers work with digital data only. Basically that is that they
work internally with integer numbers, stored in sequences of binary
bits. All input for computers must have the form of integer numbers,
and all output has the form of integer numbers. Whenever we want to
feed computers with some sort of analogue data, like an analogue
voltage, we have to convert it into a digital approximation. For that
task special devices called analog to digital converters (ADC)
exists. And in some cases we have to convert the digital output data
of computers to analogue signals, like when a computer plays music:
The computer output in form of digital data is then converted by a
device called digital to analog converter (DAC) into an analogue
voltage, that generates an analogue current through a coil in the
speakers of our sound box, and that electric current in the coil
generates a magnetic field which exercise mechanical forces and moves
the membrane of the speaker, resulting in oscillating motions, which
generates air pressure variations that our ear can detect and that we
finally hear as sound.

What is an Operating System?

Most computers, from cellphones to large super computers, use an 
operating system (OS). A well known OS is the GNU/Linux kernel.
Operating Systems can be seen as the initial program that is loaded
and started when we switch the computer on and that works as some
kind of supervisor:^[4] it can load other programs and it distributes
resources like CPU cores or RAM between multiple running programs. It
also controls user input by keyboard and mouse, displays output data
on the screen -- as text or graphics, controls how data is loaded and
stored to nonvolatile storage media like hard-disk or SSD, manages
all the network traffic and many more tasks. An important task of the
OS is to allow user programs to access all the various hardware
components from different vendors in a uniform high level manner. An
OS can be seen as an intermediate layer between user programs like a
text processor or a game, and the hardware of the computer. The OS
allows user programs to work on a higher level of abstraction, so
they do not need to know much about the low level hardware details.

Computer operating systems have generally a close relation to
software libraries, which are collections of data types and functions
working with that data types. Libraries can be a part of the OS, or
can be more or less independent of the OS. Libraries are software
components that provide data types and functions with a well defined
interface (API, Application Programming Interface) and behaviour.

Libraries can be used as shared libraries, which are single binary
files stored on the file system of a computer, often with the file
extension .so or .dll, which can be accessed from different computer
programs simultaneously, or as static libraries which are part of
single programs. Shared libraries have some advantages: we need only
one instance on the file system of the computer, and the library is
loaded only once into the computer memory (RAM), even when it is used
by different apps simultaneously. This saves space, and when the
library has serious errors, it is in principle possible to replace
the library with a corrected version, which is then used by all the
software on the computer. Shared libraries come often in numbered
versions, where a higher number denotes a newer, improved or extended
library version. Sometimes some of the programs we use may need still
an older library version, while other software needs already a never
one. In that case our file system has to provide multiple versions of
a shared library, which can be used independently. On the other hand,
statically linked libraries are directly glued with a single computer
program. That makes the distribution of the program easier, as it can
be shipped as a single entity, and we do not have to ensure that all
the needed dynamic libraries are available on the destination
computer. But if a statically linked library has serious errors, then
we have to replace all the programs that are linked statically with
that corrupted library.

Small microcontrollers and embedded devices often do not need to use
an operating system, as they generally run only one single user
program and because they usually do not have a large variety of
hardware components to support.

What is Computer Programming?

Computer programming includes the creation, testing and optimizing of
computer programs.

What is a Computer Program?

A computer program is basically a sequence of numbers, which make
some sense to a computer CPU, in such a way that the CPU recognizes
the numbers as so called instructions or numeric machine code, maybe
the instruction to add two numbers.

The first computers, built in the 1950's, were indeed programmed by
feeding sequences of plain numbers to the device. The numbers were
stored on so called punch cards, consisting of strong paper where the
numbers were encoded by holes in the cards. The holes could be
recognized by electrical contacts to feed the numbers into the CPU.
As plain numbers do not match well human thinking, soon more abstract
codes where used. A very direct code, which matches numerical
instructions to symbols, is the assembly language. In that language
for example the character sequence "add A0, $8" may map directly to a
sequence of numbers which instructs the CPU to add the constant
integer number 8 to CPU register A0, where A0 is a storage area in
the CPU where numbers can be stored. As there exist many different
types of CPUs, all with their own instruction sets, there exists many
different assembly instruction sets, with similar, but not identical
instructions. The rules that describe how these basic instructions
have to look are called the syntax of the assembly language.

The numerical machine code, or the corresponding assembly language,
is the most basic instruction set for a CPU. Every instruction which
a CPU can execute maps to a well-defined assembly instruction. So
each operation that a computer may be able to perform can be
expressed in a sequence of assembly instructions. But complicated
tasks may require millions of assembly instructions, which would take
humans very long to write, and even much longer to modify, proof and
debug.^[5]

Just a few years after the invention of the first computers, people
recognized that they would need even more abstract instruction sets,
like repeated execution, composed conditionals, or other data types
than plain numbers as operands. So higher level programming languages
like Algol, Fortran, C, Pascal or Basic were created.

What is an Algorithm?

An algorithm is a detailed sequence of more or less abstract
instructions to solve a specific task, or to reach a goal. Cooking
recipe books and car repair instructions are examples of algorithms.

The basic math operations kids learn in school -- to add, multiply or
divide two numbers with paper and pencil -- are algorithms too. Even
starting a car follows an algorithm -- when the temperature is below
zero, and snow covers the vehicle, than you first have to clean the
windows and lights. And when you first drive again after a longer
break, you would have to check the tires before you start the engine.
Algorithm can be carried out by strictly following the
instructions -- it is not necessary to really understand how and why
it works.

So an algorithm is a perfect fit for a computer, as computers are
really good at following instructions without really understanding
what they are trying to accomplish.

A math algorithm to sum up the first 100 natural numbers may look
like:

use two integer variables called i and sum
assign the value 0 to both variables

while i is less than 100 do:
  increase i by one
  add value of i to sum

optionally print the final value of sum

What is a Programming Language?

Most traditional programming languages were created to map algorithms
to elementary CPU instructions. Algorithms generally contain nested
conditionals, repetition, math operations, recovery from errors and
maybe plausibility checks. A more complicated algorithm generally can
be split into various separate logical parts, which may include
reading in data at one point, multiple processing steps at another,
and storing, or displaying data as plain text, graphic or animation
at yet another point. This splitting into parts is mapped to
programming languages by grouping tasks into subroutines, functions
or procedures which accept a set of input parameters and can return a
result.

As algorithms often work not only with numbers, but also with text,
it makes sense to have a form of textual data type in a programming
language too. And all the data types can be grouped in various ways,
for example, as sequences of multiple data of the same type, like
lists of numbers or names. Or as collections of different types, like
name, age and profession of a citizen in an income tax database. For
all these use cases programming languages provide some sort of
support.

Compilers and Interpreters

We already learned that the CPU in the computer can execute only
simple instructions, which we call numeric machine code or assembly
instructions.

To run a program written in a high level language with many
abstractions, we need some sort of a converter to transform that
program into the basic instructions that the CPU can execute. For the
conversion process we have basically two options: We can convert the
whole program into machine code, store it to disk, and than run it on
the CPU. Or we can convert it in small portions, maybe line by line,
and run each portion whenever we have converted it. Tools that
convert the whole program first are called compilers. Compilers
process the program that we have written, include other source code
like needed library modules, check the code for obvious errors and
then generate and store the machine code that we then can run.

Tools that process the source code in small portions, like single
statements, are called interpreters. They read in a line of source
code, investigate it to check if it is a valid statement, and then
feed the CPU with corresponding instructions to execute it. It is
similar to picking strawberries: you can pick one and eat it at once,
or you can put them all into a basket and eat them later. Both
interpreters and compilers have advantages and disadvantages for
special use cases. Compilers can already detect errors before the
program is run, and compiled programs generally run fast, as all the
instructions are already available when the programs runs. The
compiling step takes some time of course, at least a few seconds, but
for some languages and large programs it may take much longer. That
can make the software development process slow because as you add or
change code, you have to compile it before you can execute and test
your program. That may be inconvenient for unskilled programmers as
they may have to do much testing. Some use a programming style that
is: change a tiny bit of the source code, then run it and see what is
does. But a more common practice is that you think about the problem
first and then write the code, that then in most cases does nearly
that what you intended. For this style of programming, you do not
have to compile and execute your code that often. Compilers have one
important benefit: they can detect many bugs, mostly typing errors,
already in the compile phase, and they give you a detailed error
message. Interpreters have the advantage that you can modify your
code and immediately execute it without delay. That is nice for
learning a new language and for some fast tests, but even simple
typing errors can only be detected when they are encountered while
running the program. If your test does not try to run a faulty
statement, there will be no error, but it may occur later. Generally
interpreted program execution is much slower than running compiled
executables, as the interpreter has to continually process the source
code in real-time as it's being run, while the compiler does it only
once before the program is run. At the end of this section, a few
additional notes:

Compilers are sometimes supported by so called linkers. In that case
the compiler converts the source code, that can be stored in multiple
text files, each in a sequence of machine code instructions, and
finally the linker joins all these machine code files to the final
executable. Some compilers do not need the linking step or call the
linker automatically. And some interpreters convert the textual
source code in one very fast, initial pre-processing step ("on the
fly") to so called byte code, that can then be interpreted faster.
The languages Ruby and Python do that. Some languages, like Java, can
compile and optimize the source code while the program is running.
For that process a so called virtual machine is used, which builds an
intermediate layer between the hardware and the user program.

Types of Programming Languages

There are many different styles that software can be written in. A
programming paradigm is a fundamental style of writing software, and
each programming language supports a different set of paradigms.
You're probably already familiar with one, or more of them, and at
the very least you know what object-oriented programming (OOP) is,
because it's taught as part of many computer science courses.

We already mentioned the assembly languages, which provide only the
basic operations that the CPU can perform. Assembly languages provide
no abstractions, so maybe we should not even call them programming
languages at all. Then there are low level languages like Fortran or
C, with some basic abstractions which still work close to the
hardware and which are mostly designed for high performance and low
resource consumption (RAM) but not to detect and prevent programming
errors or to make life easy for programmers. These languages already
support some higher order data types, like floating point numbers or
text (strings), and homogeneous, fixed size containers (called arrays
in C) or heterogeneous fixed size containers (called structs in C).

A different approach is taken by languages like Python or Ruby, which
try to make writing code easier by offering many high level
abstractions and which have better protection against errors, but are
not as efficient. These languages support also dynamic containers
which can grow and shrink, or advanced data structures like hash
tables (maps) or support for textual pattern matching by regular
expressions (regex).

Another way to differentiate programming languages is if they are
statically, or dynamically typed. Ruby, Python and JavaScript are all
examples of dynamically typed languages, that is, they use variables
which can store any data type, so the variable's type of data that it
accepts can therefore dynamically change during program execution.
That seems comfortable for the user, and sometimes it is, especially
for short programs which may be written for one-time use only and are
sometimes called scripts. But dynamic typing makes discovery of
logical errors harder -- an illegal addition of a number to a letter
may be detected only at run-time. And dynamically typed languages
generally waste a lot of memory and their performance is not that
great. It is as we would own a set of large, equally sized moving
boxes, and we would store all of our goods in it, each piece in one
box.

For statically typed languages, each variable has a well defined data
type like integer number, real number, a single letter, a text
element and many more. The data type is either assigned by the author
of the program with a type declaration, or is detected by the
compiler itself when processing the program source code, called type
inference, and the variable's type does never change. In this way,
the compiler can check for logical errors early in the compile
process, and the compiler can reserve memory blocks exactly
customized to the variables that we want to store, so total memory
consumption and performance can be optimized. Referring again to our
boxes example, statically typing is like using customized boxes for
all your goods.

All these types of programming languages are often called imperative
programming languages, as the program describes detailed what to do.
There are other types of programming languages too, for example
languages like Prolog, which try to give only a set of rules and then
let the computer try to solve a problem with these rules. And of
course there are the new concepts of artificial intelligence (AI) and
machine learning (ML), which are less based on algorithms and more on
neural nets which are trained with a lot of data until it can provide
the desired results. Nim, the computer language this book is about,
is an imperative language, so we will focus on the imperative
programming style in this book. But of course, Nim can be used to
create AI applications.

Further still, we can differentiate between languages like C, C++,
Rust, Nim and many more that compile to native executables and can
run directly on the hardware of the computer, contrasted with
languages like Java, Scala, Julia and some more, that use a large 
Virtual Machine (VM) as an intermediate layer between the program and
the hardware, and interpreted languages like Ruby and Python.
Languages using a virtual machine generally need some startup time
when a program is invoked, as the VM must be loaded and initialized,
and interpreted languages are generally not very fast.^[6] The
distinction between languages that compile to native executables, and
those that are executed on a virtual machine, is not really sharp.
For example Kotlin and Julia where executed on a virtual machine
initially, but now can compile the source code to native executables.

An important class of programming languages are the so called 
Object-Oriented-Programming (OOP) languages, which uses inheritance
and dynamic dispatch, and become popular in the 1990's. For some time
it was assumed that Object-Oriented-Programming was the ultimate
solution to manage and structure really large programs. Java was the
most prominent example of the OOP languages. Java forces the
programmer to use OOP design, and languages like C++, Python or Ruby
strongly push programmer to use the OPP design. Practice has shown
that OOP design is not the ultimate solution for all computing
problems, and OPP design may prevent optimal performance. So newer
languages, like Go, Rust and Nim, support some form of OOP
programming, but use it only as one paradigm among many other.

Another popular and important class of programming languages is
JavaScript and its more modern cousins like TypeScript, Kotlin or
Dart and others. JavaScript was designed to run in web browsers to
support interactive web pages and programs and games running in the
browser. In this way the program became nearly independent from the
native operating system of the computer. Note that unlike the name
may indicate, JavaScript is not closely related to the Java language.
Nim can compile to a JavaScript backend, so it supports web
development well.

Language    Type     Runtime        Memory      Generics    Macros
           System                 Management
Nim      static,    native     GC, Destructors, Yes      AST based,
         strong     binaries   manual                    hygenic

Sometimes source code written in one programming language is
converted into another one. A prominent target for such conversions
is JavaScript, as JavaScript enables execution of programs in web
browsers. Another important target language is C or C++. Creating
intermediate C code, which is then compiled by a C compiler to native
executables has some advantages compared to direct compilation to
native executables: C compilers exists for nearly all computer
systems including microcontrollers and embedded systems, so the use
of a language is not restricted to systems for which a native
compiler backend is provided. And C as intermediate code simplifies
the use of system libraries which generally provide a C compatible
interface. Due to decades of development C compilers generally can do
better code optimizations than young languages may manage to do. Some
people fear that intermediate C code carries the problems of the C
language, like verbosity, confusing and error-prone code or undefined
behavior to the source languages. But these well known problems of C
occur only when humans write C code directly, in the same way when
humans write assembly code directly. Automatic conversions are well
defined and well tested, which means they are free of errors to the
same degree as direct machine code generation would be. But indeed
there are some small drawbacks when C or C++ is used as a backend of
a programming language: C does not always allow direct access to all
CPU instructions, which may make it difficult to generate optimal
code for some special constructs like exceptions. And C uses wrap
around arithmetic for unsigned integer types, which may not be what
modern languages desire. The current Nim implementation provides a
JavaScript and a C and C++ backend. While the JavaScript backend is a
design decision to enable web development, the C and C++ backends are
more a pragmatic decision, and may be later replaced or at least
supported by direct native code generation or use of the popular LLVM
backend. ^[7] When computer languages are converted from one language
to another, then sometimes the term transpiler is used to
differentiate the translation process to a direct compilation to a
binary executable. When program code is converted between very
similar languages with nearly the same level of abstractions, then
the term transpiler may be justified. But Nim is very different from
C and has a higher abstraction level, and the Nim compiler performs
many advanced optimizations. So it should be not called a transpiler,
even when compiling to JavaScript or to the C++ backend.

Why Nim?

 In this section we are using a lot of new Computer Science (CS)
 expressions but do not explain them. That is intentional -- when you
 already know them you may get a better feeling of what Nim is, and
 when you do not know them, you will at least learn that we can
 describe Nim with fancy-sounding terms.

Three well known traditional programming languages are C, Java and
Python. C is basically a simple, close-to-the-hardware language
created in 1972, for which compilers can generate fast, highly
optimized native machine code, but it has cryptic syntax, some
strange semantics, and is missing higher concepts of modern
languages. Java, created in 1995, forces you strongly to the
object-orientated style of programming (OOP) and runs on a virtual
machine, which makes it unsuitable for embedded systems and
microcontrollers. Python, created in 1991, is generally interpreted
instead of compiled, which makes program execution not very fast, and
it does not really allow writing low level code which operates close
to the hardware. As many libraries of the Python language are written
in highly optimized C, Python can appear really fast if a standard
task, like sorting of data, processing of CSV or JSON files or web
site crawling is performed. So Python is not a bad solution when we
use it mostly for calling library functions, but it reveals its low
performance when we have to write some actual Python code in order to
solve a problem. Of course there are many more programming languages,
each with its own advantages and disadvantages -- with some optimized
for special use cases.

Nim is a state-of-the-art programming language well suited for
systems and application programming. Its clean Python-like syntax
makes programming easy and fun for beginners, without applying any
restrictions to experienced systems programmers. Nim combines
successful concepts from mature languages like Python, Ada and Modula
with a few established features of the latest research. It offers
high performance with type and memory safety while keeping the source
code short and readable. The compiler itself and the generated
executables support all major platforms including Windows, Linux, BSD
and Mac OS X. Cross-compiling to Android and other mobile and
embedded devices and microcontrollers is possible, and the JavaScript
backend allows to create web apps and to run programs in web
browsers. The custom package managers, Nimble or Nimph, makes use and
redistribution of programs and libraries easy and secure. Nim
supports various "backends" to generate the final code. The C, C++
and LLVM-based backends allow easy OS library calls without
additional glue code, while the JavaScript backend generates high
quality code for web applications. The integrated "Read/Eval/Print
Loop" (REPL), "Hot code reloading", incremental compilation (expected
for version 1.8), and support of various development environments
including debugging and language server protocols makes working with
Nim productive and enjoyable.

Some Facts About Nim

  * Nim is a multi-paradigm programming language. Unlike some popular
    programming languages, Nim doesn't focus on the OOP paradigm.
    It's mainly a imperative and procedural programming language,
    with varying support for OOP, data-orientated, functional,
    declarative, concur- rent, and other programming styles. Nim
    supports common OOP features, including inheritance,
    polymorphism, and dynamic dispatch.

  * The generated executables are dependence free and small: a simple
    chess program with a plain GTK-based graphical user interface is
    only 100 kB in size and the size of the Nim compiler executable
    itself is about 6.5 MB. It is possible to shrink the executable
    size of "Hello World" programs to about 10 kB for use on tiny
    microcontrollers.

  * Nim is fast. Generally performance is very close to other
    high-performance languages such as C or C++. There are some
    exceptions still: other languages may have libraries or
    applications that have been tuned for performance for many years,
    while similar Nim applications are so far less tuned for
    performance, or maybe are more written with a priority of short
    and clean code or run-time safety.

  * Clean Python-like syntax with significant white-space, no need
    for block delimiters like {} or begin/end keywords, and no need
    for statement delimiters like ;

  * Safety: Nim programs are type- and memory-safe -- memory
    corruption is prevented by the compiler as long as unsafe low
    level constructs like casts, pointers and the addr operator or
    the {.union.} pragma are not used.

  * Fast compiler. The Nim compiler can compile itself and other
    medium-size packages in less than 10 seconds, and upcoming
    incremental compilation will increase that speed further.

  * Nim is statically typed: each object and each variable has a
    well-defined type, which catches most programming errors already
    at compile time, prevents run-time errors, and ensures highest
    performance. At the same time the statically typing makes it
    easier to understand and to maintain larger code bases.

  * Nim supports various memory management strategies, including
    manually allocations for critical low-level tasks as well as
    various garbage collectors including a destructor based, fully
    deterministic memory manager.

  * Nim produces native, highly optimized executables and can also
    generate JavaScript output for web applications.

  * Nim has a clean module concept which helps to structure large
    projects.

  * Nim has a well-designed library which supports many basic
    programming tasks. The full source code of the library is
    included and can be viewed easily from within the HTML-based API
    documentation.

  * Library modules like the OS module provides OS independent
    abstractions, which allows to compile and run the same program on
    different operating system without modifications.

  * The Nim standard library is supported by more than 1000 external
    packages for a broad range of use cases.

  * Asynchronous operation, threading and parallel processing is
    supported.

  * Nim supports all popular operating systems like Linux, Windows,
    MacOS and Android.

  * Usage of external libraries written in C is easy and and occurs
    directly without any glue code, and Nim can even work together
    with code written in other languages, for example there are some
    Nim <-> Python interfaces available.

  * Many popular editors have support for Nim syntax highlighting and
    other IDE functionality like on-the-fly checking for errors and
    displaying detailed information about imported functions and data
    types.

  * In the last few years Nim has reached some important milestones:
    Version 1.0 with some stability promises was released, and with
    the ARC and ORC memory management strategies and full destructor
    support fully deterministic memory management comparable to
    memory management in C++ or Rust is available. So problems of
    conventional garbage collectors like delayed memory deallocation
    or longer pausing of programs due to the GC process are gone. And
    some larger companies have started using Nim in production, the
    most important may be currently the Status Corp. with their
    Etherium client development.

Nim supports many programming styles

We mentioned already that Nim is a multi-paradigm programming
language, that supports various programming styles. While we may
regard Nim in first line as an imperative, procedural programming
language, it supports the popular functional and object-orientated
programming styles well.

In classical OOP programming languages, we have the concept of 
classes with attributes, and methods that are very closely bound to
the classes, as in Python:

class User:
  def say(self):
    print("It does not work!")

user = User()
user.say()

In this Python snippet, we declare a class User, with a custom method
named say() bound to this class. Then we create an instance variable
of this class and call its say() method.

This tight bounding of methods to classes is not very flexible, e.g.
extending the set of methods of a class may be difficult or
impossible. Another problem with such a class concept is, that it is
not always clear to which class a method belongs when more than just
one single class is involved: Imagine that we need a method that
appends a single character to a text string. Is that method a member
of the character class, or a member of the text string class?

Nim avoids such a strict class concept, while its generalized method
call syntax allows us to use a class like syntax for all of our data
types: e.g. to get the length of a string variable, we can write len
(myString) in classic procedural notation, or we can use the method
syntax myString.len() or just myString.len. The compiler regards all
these notations as equivalent, so we have the method syntax available
without the restrictions of the class concept. The method call syntax
can be used in Nim for all data types, even for plain numbers -- so
the notations abs(myNum) is fully equivalent with myNum.abs.

The Python code from above may look in Nim like

type User = object

proc say(self: User) =
  echo ("It does not work!")

let user = User()
user.say()

Instead of the classes we use object types in Nim, and we define
procedures and methods that can work on objects or other data types.

As an example for the functional programming style in Nim we may look
at some code fragment from a real world app that has to generate a
string from four numbers, separated by commas. Using the mapIt()
procedure imported from the sequtils module and the fmt() macro from
the strformat module, we may write that in functional programming
style in this way:

from strutils import join
from sequtils import mapIt
from strformat import fmt
const DefaultWorldRange = [0.0, 0, 800, 600]
let str = DefaultWorldRange.mapIt(fmt("{it:g}")).join(", ")
echo str # "0, 0, 800, 600"

In the imperative, procedural style we would write it like

var str: string
for i, x in pairs(DefaultWorldRange):
  str.add(fmt("{x:g}"))
  if i < DefaultWorldRange.high:
    str.add(", ")

Nim is Efficient

Nim is a compiled and statically-typed language. While for
interpreted, dynamically-typed languages like Python we have to run
every statement to check even for trivial errors, the Nim compiler
checks for most errors during the compile process. The static typing
together with the well-designed Nim type system allows the compiler
to catch most errors already in the compile phase, like the undefined
addition of a number and a letter, and to report the errors in the
terminal window or directly in the editor or IDE. When no errors are
found or all errors have been fixed, then the compiler generates
highly optimized dependency free executables. And this compilation
process is generally really fast, for example the compiler compiles
itself in maybe 10 to 30 seconds on a typical modern PC.^[8]

Modern concepts like zero-overhead iterators, compile-time evaluation
of user-defined functions and cross-module inlining in combination
with the preference of value-based, stack-located data types leads to
extremely efficient code. Multi-threading, asynchronous input/output
operations (async IO), parallel processing and SIMD instructions
including GPU execution are supported. Various memory management
strategies exists: selectable and tuneable high performance Garbage
Collectors (GC), including a new fully deterministic destructor based
GC, are supported for automatic memory management. These can be
disabled for manual memory management. This makes Nim a good choice
for application development and close-to-the-hardware system
programming at the same time. The unrestricted hardware access, small
executables and optional GC will make Nim a perfect solution for
embedded systems, hardware driver and operating system development.

Nim is Expressive and Elegant

Nim offers a modern type system with templates, generics and type
inference. Built-in advanced data types like dynamic containers,
sets, and strings with full UTF support are completed by a large
collection of library types like hash tables and regular expressions.
While the traditional Object-Oriented-Programming programming style
with inheritance and dynamic dispatch is supported, Nim does not
enforce this programming paradigm and offers modern concepts, like
procedural and functional programming. The optional method call
syntax allows to use all data types and functions in an OOP like
fashion, e.g. instead of len(myStr) we can also use the OOP style
myStr.len.^[9] The powerful AST-based hygienic macro system offers
nearly unlimited possibilities for the advanced programmer. This
macro and meta-programming system allows compiler-guided code
generation at compile time, so the Nim core language can be kept
small and compact, while many advanced features are enabled by user
defined macros. For example the support of asynchronous IO operations
has been created with these forms of meta-programming, as well as
many Domain Specific Language (DSL) extensions.

Nim is Open and Free

The Nim compiler and all modules of the standard library are
implemented in Nim. All source code is available under the less
restricted MIT license.

Nim has a friendly and helpful growing community

The Nim forum is hosted at:

https://forum.nim-lang.org/

and the software running the forum is coded in Nim.

Real-time chat is supported by IRC, Gitter, Discord, Telegram and
others.

Nim is also supported by Reddit.com and Stackoverflow.com:

  * https://www.reddit.com/r/nim/

  * https://stackoverflow.com/questions/tagged/nim-lang

Nim has a encouraging future

Started more than 12 years ago as a small community project of some
bright CS students led by Mr. A. Rumpf, it is now considered as one
of the most interesting and promising programming languages,
supported by countless individuals and leading companies of the
computer industry, for instance, it's actively used in the areas of
application, game, web and crypto-currency development. Nim has made
a large amount of progress in the last few years: it reached version
Nim v1.6 with some stability guarantees and a new deterministic
memory management system was introduced, which will improve support
of parallel processing and the use of Nim in the area of embedded
systems development.

Why is Nim not a popular mainstream language yet?

Nim was created by Mr. A. Rumpf in 2008, supported by a few
volunteers. Finally, in 2018 Nim got some significant monetary
support by Status Corp. and in 2019 the stable Nim version 1.0 was
released. But still Nim is developed by a small core team and some
volunteers, while some other languages like Java, C#, Go, or Rust are
supported by large companies, or like C and C++ have a very long
history and well-trained users. And finally there are many competing
languages, some with a longer history, and some maybe better suited
for special purposes, like JavaScript, Dart or Kotlin for web
development, Julia or R for numeric applications, or Zig, C and
Assembly for the tiny 8-bit microcontrollers with a small amount of
RAM.

Nim is already supported by more than 1000 external packages which
cover many application areas, but that number is still small compared
to really popular languages like Python, Java or JavaScript. And some
Nim packages can currently not really compare with the libraries of
other languages, which have been optimized for years by hundreds or
thousands of full-time developers.

Indeed the future of Nim is not really secure. Core developers may
vanish, financial support may stop, or maybe a better language may
appear. But even if the development of Nim should stop some day, you
will still be able to use it, and many concepts that you may have
learned with Nim can be used with other modern languages too.

Is Nim a good choice as first language for a Beginner?

When you use C as your first language, you may learn well how
computers really work, but the learning experience is not that nice,
progress is slow and C lacks many concepts of modern programming
languages. C++, Rust or Haskell are really too difficult for
beginners. So currently many starts with Python. While you can learn
high level concepts well with Python and you get useful results fast,
you learn not much about the internal working of computers. So you
may never understand why your code is slow and consumes so much
resources, and you will have no idea how to improve the program or
how you could run it successfully on restricted hardware. It's like
learning to drive a car, without any knowledge about how a combustion
engine, the transmission, or the brakes really work. Nim has none of
these restrictions, as we have high level concepts available like in
Python, but we have access to low level stuff too, so we can really
understand the internal workings, if we want. Learning resources for
Nim are still not that good as for mainstream languages, but there
exists some nice tutorials already, and hopefully this book will help
beginners also a bit.

Is Nim really a good teaching language?

Generally yes, in the same way as Pascal was in the 1980's, and
Modula/Oberon were at the end of the last century. But Nim still has
the same problems as the wirthian languages: They do not really help
with finding a job. When we teach the kids some JavaScript or C, they
may find at least a simple employment when they have to leave the
intended education path early for some reason. With niche languages
this is unfortunately not the case, so teachers should know about
their responsibility. And of course teaching against the interests of
the kids makes not much sense. When they want to learn some
JavaScript to make some visual effects or whatever easily, then it is
hard to teach another language which may be not immediately available
on the PC at home or their smartphone.

So is Nim really the best start for me?

Maybe not. When you intend to learn a programming language today and
make a great video game tomorrow, then definitely not. This is just
not possible. While there are nice libs for making games with Nim
already available, there exists easier solutions in other languages.
With some luck you may find some source code for that languages, so
that you can patch a few strings and maybe modify some colors and the
background music and call it your game.

After learning Nim, will I still have to learn other programming
languages?

Nim is a quite universal language, so it is a good candidate for
someone who intends to learn only one single language. But of course
it is always a good idea to learn a few other languages later.
Generally we can not really avoid learning C, as so much C code
exists world wide. Most algorithm that have ever been invented are
available as a C implementation somewhere, and most libraries are
written in C or have at least a C API, which you can use from other
languages including Nim. As C is a small language without difficult
language constructs, some minimal C knowledge is generally sufficient
to convert a C program to another language. Often that conversion
process is supported by tools, like the Nim c2nim tool. So learning
some C later is really a good idea, and when you have some basic
understanding of Nim and CS in general, learning some C is an easy
task. Learning C before Nim would be an option still, as for C more
learning resources exists. So years ago some people recommended
learning C or Python before Nim. But Nim has enough learning
resources now, so we recommend indeed starting with Nim directly.

Why should I not use Nim?

Maybe it is just not the ideal solution for you. A racing bicycle or
a mountain bike are both great devices, but for cycling a few hundred
meters to the bakers shop both may be not the perfect solution. A
plain old bicycle would do better. Even as Nim seems to join the
benefits of a racing bicycle and a mountain bike well -- high
performance and robust design -- and is not expensive, it is just not
the optimal solution for everybody. People who write only tiny
scripts and have not to care about performance can continue using
Python. People who are only interested in special applications, maybe
only in web development or only in tiny 8 bit microcontrollers may
not really need Nim. Nim can do this and much more well, but for
special use cases better suited languages may still exist. And
someone who has managed to learn C++ really well over a period of
many years may decide to continue with C++ also. Currently another
possible reason for not using Nim can be missing libraries. When you
need some important libraries for your project, and these are
currently not available for Nim, this can be of course a big problem
in the case that you have not the skills or the time to write them
from scratch or at least create high level bindings to a C library.

Our first Nim Program

To keep our motivation, we will present a first tiny Nim program now.
Actually we should have delayed this section until we have installed
the Nim compiler on our computer, but we can already run and test the
program by just copying it into one of the available Nim online
playgrounds like

https://play.nim-lang.org/

In the section What is an Algorithm? we described an algorithm to sum
up the first 100 natural numbers. Converting that algorithm into a
Nim program is straightforward and results in the text file below.
You can copy it into the playground and run it now if you want. The
program is built using some elementary Nim instructions for which we
will give only a very short description here. Everything is explained
in much more detail in the next part of this book.

var sum: int
var i: int
sum = 0
i = 0
while i < 100:
  inc(i, 1)
  inc(sum, i)
echo sum

We write Nim programs with an editor tool in the form of plain text
files, and you will learn how to create them soon. We call these text
files the source code of the program. The source code is the input
for the compiler. The compiler processes the source code, checks it
for obvious errors and then generates an executable file, which
contains the final CPU instructions and can be run. Executable files
are sometimes called executables or binary files. The term binary is
misleading, as all files on computers are indeed stored as binary
data, but the expression "binary" is used to differentiate the
executable program from text files like the Nim source code which we
can read, print and edit in an editor. Don't try to load the
executable files generated by the Nim compiler into a text editor, as
the content is not plain text, but numeric machine code that may
confuse the editor. On the Windows OS, executable files generally get
a special name extension .exe, but on Linux no special name
extensions are used.

Nim source code files are processed by the Nim compiler from the top
to the bottom, and for the generated executable the program execution
also starts in principle at the top. But for the program execution
there exists some exceptions, e.g. program code enclosed in functions
is not immediately executed where it appears in the program source
code file, but later when the function is called. And the program
execution is not a linear process -- we can use conditional
expressions to skip parts of the program, or various loop constructs
to repeat the execution of some program segments. Generally the
program execution in Nim is more similar to languages like Python or
Ruby than to the C language: A C program always needs a main()
function with exactly this name, and the execution of a C program
always starts by a compiler generated call of this function.

Elementary entities of computer programs are variables, which are
basically named storage areas in the computer. As Nim is a compiled
and statically-typed language, we have to declare each variable
before we can use it. We do that by choosing a meaningful name for
the variable and specifying its data type. To tell the compiler about
our intention to declare a variable, we start the line with the var
keyword, followed by the chosen name, a colon and the data type of
our variable. The first line of our program declares a new variable
named sum of data type int. Int is short for integer and indicates,
that our variable should be able to store negative or positive
integer numbers. The var at the start of the line is a keyword.
Keywords are reserved symbols which have a special meaning for the
compiler. Var indicates that we want to introduce a new variable. The
compiler will recognize that and will reserve a memory location in
the RAM of the computer which can store the actual value of the
variable.

The second line is nearly identical to the first line: we declare
another variable again with int type and plain name i. Variable names
like i, j, k are often used when we have no idea for a meaningful
name and when we intend to use that variable as a counter in a loop.

In the lines 3 and 4 of our program we initialize the variables, that
is, we give them a well-defined initial value. To do that we use the 
= operator to assign it a value. Operators are special symbols like +
, -, * or / to indicate our desire to do an addition, a subtraction,
a multiplication or a division. Note that the = operator is used in
Nim like in many other programming languages for assignment, and not
like in traditional mathematics as an equality test. The reason for
that is that in computer programming, assignments occur more often
than equality tests. Some early languages like Pascal use the
compound := operator for assignment, which may be closer to
mathematics use, but is more difficult to type on a keyboard and
looks not too nice for most people. An expression like x = y assigns
the content of variable y to x, that is, x gets the value of y, the
former value of x is overwritten and lost, and the content of y
remains unchanged. After that assignment, x and y contain the same
value. In the above example we do not assign the content of a
variable to the destination, but instead use a literal numeric
constant with value 0. When the computer has executed lines 3 and 4
the variables sum and i each contain the start value 0.

Line 5 is much more interesting: it contains a while condition. The
line starts with the term while, which is again a reserved keyword,
followed by the logical expression i < 100 and a colon. An expression
in Nim is something which has a result, like a math expression as 2 +
2 which has the result 4 of type integer. A logical expression has no
numerical result, but a logical (boolean) one, which can be true or 
false. The logical expression i < 100 depends on the actual content
of variable i. The two lines following the line with the while
keyword are each indented by two spaces, meaning that these lines
start with two spaces more than the line before. That form of
indentation is used in Nim (and Python) to indicate blocks. Blocks
are grouped statements. The complete while loop consists of the line
containing the while keyword followed by a block of statements. The
block after the while condition is executed as long as the while
condition evaluates to the logical value true. For the first loop
iteration i has the initial value 0, the condition i < 100 evaluates
to the boolean value true and the block after the while condition is
executed for the first time. In the following block we have the inc()
instruction. inc is short for increment. inc(a, b) increases the
value of a by b, b remains unchanged. So in the above block i is
increased by one, and after that sum is increased by the current
value of i. So when that block has been executed for the first time i
has the value 1 and sum also has the value 1. At the end of that
block execution starts again at the line with the while condition,
now testing the expression i < 100 with i containing the value 1.
Again it evaluates to true, the block is executed again, i gets the
new value 2, and sum gets the value 3. This process continues until i
has the value 100, so the condition i < 100 evaluates to false and
execution proceeds with the first instruction after the while block.
That instruction is an echo statement, which is used in Nim to write
values to the terminal or screen of the computer. Some other
languages use the term print or put instead of echo.

Don't worry if you have not understood much of this short
explanation, we will explain all that in much more detail later.

If you should decide to try the above program, maybe in a playground
internet page or already on your local computer, then it is best to
copy the source code verbatim instead to type it in from scratch, as
for beginners tiny typos can generate a lot of trouble. For the case
that you should decide to type it in with your keyboard, you should
try to type it exactly as displayed above. All the program code
should start directly at the first column, but the two lines after
the while keyword should start with two spaces. This strict
indentation is used in Nim and some other programming languages like
Python or Haskell to structure the program code and to mark the
extend of code blocks. Some other programming languages like C do a
similar alignment of the source code for readability, but that
alignment is ignored by the C compiler -- instead blocks have to be
enclosed in curly braces {}. Note that you have to do the indentation
really with spaces, as Nim does not accept tabulator characters in
its source files. Also note that the Nim compiler does distinguish
between words starting with a lower or an upper case letter. Nim
keywords are written always in lower case, and when we define a
variable as sum then we should always refer to it in exactly this
notion.footnote:[Actually Nim relaxes this strict notation a bit,
which is called

Also note that spaces in the Nim source code are important and can
change the semantic: While in C spaces are mostly only used to
separate distinct symbols, in Nim spaces have some more
functionality. For instance in mathematically expressions, a - b or 
a-b is both a valid subtraction in the case when a and b both have a
numeric type for which an infix subtraction operator is defined, but
the code segment a -b may give us an error message from the compiler.
The reason is, that in this case the - sign is directly attached to b
but separated to a by at least one space. In this case the Nim
compiler would interpret the - sign as an unary operator attached to 
b. Even in the case that such an unary - may have been defined
before, then the operands a and b would be not separated by an infix
operator, which is an invalid syntax in Nim. An expression like a -
-b would be a valid syntax instead -- unary minus attached to b, and a
and (-b) separated by an infix - operator. In this example we have
learned already that the same symbol can have a different meaning in
the Nim language, depending on the context. For operators or
functions this is called overloading, which most modern programming
languages use. This sensitivity to the asymmetrically use of spaces
applies also to the less than operator that we used in the above
example: a < b or a<b is the infix notation that we generally intend
for a comparison operation, while a <b would be generally invalid
code. For infix operators we generally put a space on each side, as
this improves readability, but it is not really needed and so some
people do not insert these spaces. Unary operators, like the unary -
sign, should always proceed a variable or a literal without a space.

All this may sound a bit complicated, and for beginners the compiler
error messages about this formatting rules may be not always fully
clear. But finally it is just how we would write the code with paper
and pencil, and after the initial learning phase you just will do it
right without thinking about it.

Binary Numbers

When we write numbers in ordinary life we generally use the decimal
system with base 10 and the 10 available digits 0, 1, ...  9. To get
the value of a decimal number we multiply each digit with powers of
10 depending on the position of the digit and sum the individual
terms. The rightmost digit is multiplied with 10^0, the next digit
with 10^1, and so on. A literal decimal number like 7382 has then the
numerical value 2 * 10^0 + 8 * 10^1 + 3 * 10^2 + 7 * 10^3. We have
used here the exponential operator ^ -- with 10^3 = 10 * 10 * 10.
Current computers use binary representation internally for numbers.
Generally we do not care much about that fact, but it is good to know
some facts about binary numbers. Binary numbers work nearly
identically to decimal numbers. The distinction is that we have only
two available digits, which we write as 0 and 1. A number in binary
representation is a sequence of these two digits. Like in the decimal
system, the numerical value results from the individual digits and
their position: The binary number 1011 has the numerical value 1 * 2^
0 + 1 * 2^1 + 0 * 2^2 + 1 * 2^3, which is 11 in decimal notation. For
binary numbers the base is 2, so we multiply the binary digits by
powers of two. Formally addition of two binary numbers works like we
know it from the decimal system: we add the matching digits and take
carry into account: 1001 + 1101 = 10110 because we start by adding
the two least significant digits of each number, which are both 1.
That addition 1+1 results in a carry and result 0. The next two
digits are both zero, but we have to take the carry from the former
operation into account, so result is 1. For the next position we have
to add 0 and 1, which is just 1 without a carry. And finally we have
1 + 1, which results in 0 with a carry. The carry generates one more
digit, and we are done. In the decimal system with base 10 a
multiplication with 10 is easily calculated by just shifting all
digits one place to the left and writing a 0 at the now empty
rightmost position. For binary numbers it is very similar: a
multiplication by the base, which is two in the binary system, is
just a shift left, with the rightmost position getting digit 0.^[10]

In the binary system we call the digits often bits, and we number the
bits from right to left, starting with 0 for the rightmost bit -- we
say that the binary number 10010101 is an 8-bit number because
writing that number in binary representation needs 8 digits. Often we
imagine the individual bits as small bulbs, a 1 bit is imagined as a
lit bulb, and a 0 bit is imagined as a dark bulb. For lit bulbs we
say also that the bit is set, meaning that in the binary number
10010101, bits 0, 2, 4 and 7 are set, and the other bits are unset or
cleared.

Groups of 8 bits are called a byte, and sometimes 4 bits are called a
nibble.

One, two, four or 8 bytes are sometimes called a word, where a word
is an entity which the computer can process in one single
instruction. When we have a CPU with 8 byte word size this means that
the computer can for example add two variables, each 8 byte in size,
in one single instruction.

Let us investigate some basic properties of binary numbers. Let us
assume that we have an 8-bit word (a byte). An 8-bit word can have 2^
8 different states, as each bit can be set or unset independently
from the other bits. That corresponds to numbers 0 up to 255 -- we
assume that we work with positive numbers only for now, we will come
to negative numbers soon. An important property of binary numbers in
computers is the wrapping around, which is a consequence of the fact
that we have only a limited set of bits available to store the
number. So when we continuously add 1 to a number, at some point all
bits are set, which corresponds to the largest number that can be
stored with that number of bits. When we then add again 1, we get an
overflow. The run-time system may catch that overflow, so we get an
overflow error, or the number is just reset to zero, as it may happen
in our car when we manage to drive one million miles, or when the
ordinary clock jumps from 23:59 to 00:00 of the next day. An useful
property of binary numbers is the fact that we can easily invert all
bits, that is replace set bits by unset ones and vice versa. Let us
use the prefix ! to indicate the operation of bit inversion, then !
01001100 is 10110011. It is an obvious and useful fact that for each
number x we get a number with all bits set when we add x and !x. That
is x + !x = 11111111 when we consider a 8 bit word. And when we
ignore overflow, then it follows that x + !x + 1 = 0 for each number
x. That is a useful property, which we can use when we consider
negative numbers.

Now let us investigate how we can encode negative numbers in binary
form. In the binary representation we have only two states available,
0 or 1, a set bit or an unset bit. But we have no unitary minus sign.
We could encode the sign of a number in the topmost bit of a
word -- when the topmost bit is set that indicates that the number is
regarded as negative. Generally a modified version of this encoding
is used, called two's complement: a negative number is constructed by
first inverting all the bits -- a 0 bit is transferred into a 1 bit
and vice versa -- and finally the number 1 is added. That encoding
simplifies the CPU construction, as subtraction can be replaced by
addition in this way:

Consider the case that we want to do a subtraction of two binary
encoded numbers. The operation has the symbolic notation A - B for
arbitrary numbers A and B. The subtraction is by definition the
inverse operation of the addition, that is A + B - B = A for each
number A and B, or in other words, B - B = 0 for each number B.

Assume we have a CPU that can do additions and that can invert all
the bits of a number. Can we do subtraction with that CPU? Indeed we
can. Remember the fact that for each number X X + !X + 1 = 0 as long
as we ignore overflow. If that relation is true for each number, than
it is obviously true for each B in the expression A - B, and we can
write A - B = A + (B + !B + 1) - B = A + (!B + 1) when we use the
fact that in mathematics addition and subtraction is associative,
that is we can group the terms as we want. But the term in the
parenthesis is just the two's complement, which we get when we invert
all bits of B and add 1. So to do a subtraction we have to invert the
bits of B, and then add A and !B and 1 ignoring overflow. That may
sound complicated, but bit inversion is a very cheap operation in a
CPU, which is always available, and adding 1 is also a very simple
operation. The advantage is that we do not need separate hardware for
the subtraction operation. Generally subtraction in this way is not
slower than addition because the bit inversion and the addition of 1
can be performed at the same time in the CPU as an ordinary addition.

From the equation above indicating A - B = A + (!B + 1) it is obvious
that we consider the two's complement (!B + 1) as the negative of B.
Note that the two's complement of zero is again zero, and two's
complement of 00000001 is 11111111. All negative numbers in this
system have a bit set to 1 at the leftmost position. This restrict
all positive numbers to all the bit combinations where the leftmost
bit is unset. For an 8-bit word this means that positive numbers are
restricted to the bits 00000000 to 01111111, which is the range 0 to
127 in decimal notation. The two's complement of decimal 127 is
10000001. Seems to be fine so far, but note there exists also the bit
pattern 10000000 which is -128 in decimal. For that bit pattern there
exists no positive value. If we try to build the two's complement of
that bit pattern, we would get the same pattern again. This is an
asymmetry of two's complement representation, which can not be
avoided. It generally is no problem, with one exception. We can never
invert the sign of the smallest available integer; that operation
would result in a run-time error.^[11]

Summary: when we work only with positive numbers, we can store in an
8-bit word, which is generally called a byte, numbers from 0 up to
255. In a 16-bit word we could store values from 0 up to 2^16 - 1,
which is 65535. When we need numbers which can be also negative we
have for 8-bit words the range from -128 to 127 available, which is
-2^7 up to 2^7 - 1. For a signed 16-bit word the range would be -2^15
up to 2^15 - 1.

While we can work with 8 or 16-bit words, for PC programming the CPU
usually supports 32 or 64 bit words, so we have a much larger number
range available. But when we program microcontrollers or embedded
devices we may indeed have only 8 or 16-bits words available, or we
may use such small words size intentionally on a PC to fit all of our
data into a smaller memory area.

One important note at the end of this section: whenever we have a
word with a specific bit pattern stored in the memory of our
computer, then we can not decide from the bit pattern directly what
type of data it is. It can be a positive or a negative number, but
maybe it is not a number at all but a letter or maybe something
totally different. As an example consider this 8 bit word: 10000001.
It could be 129 if we have stored intentionally positive numbers in
that storage location, or could be -127 if we intentionally stored a
negative value. Or it could be not a number at all. Is that a
problem? No it is not as long as we use a programming language like
Nim which use static typing. Whenever we are using variables we
declare their type first, and so the compiler can do bookkeeping
about the type of each variable stored in the computer memory. The
benefit is, that we can use all the available bits to encode our
actual data, and we do not have to reserve a few bits to encode the
actual data type of variables. For languages without static typing
that is not the case. In languages like Python or Ruby we can use
variables without a static type, so we can assign whatever we want to
it. That seems to be comfortable at first, but can be confusing when
we write larger programs and the Python or Ruby interpreter has to do
all the bookkeeping at run-time, which is slow and wastes memory for
the bookkeeping.

To say it again in other words: for deciding if an operation is
valid, it is generally sufficient to know the data type of the
operands only. We do not have to know the actual content. The only
exception is if we invert the sign of the most negative integer
number or if we do an operation with causes an overflow, as there are
not enough bits available to store the result -- we may get a run-time
error for that case.^[12] In a statically-typed language each
variable has a well-defined type, and the compiler can ensure at
compile time that all operations on that variables are valid. If an
operation is not valid then the compiler will give an error message.
Then when these operations are executed at run-time they are always
valid operations, and the actual content, like the actual numeric
value, does not matter.

Hexadecimal Numbers

These number type with base 16 is by far not that important than the
binary numbers, and it has not really a technical justification to
exist, but you may get in touch with these numbers from time to time.
Hexadecimal numbers are mostly a legacy from early days of computers,
where computer programming was done not in real programming languages
but with numeric codes. To represent the 16 hexadecimal digits the 10
decimal digits are supported by the characters 'A' .. 'F'. The most
important property of a hexadecimal digit is that it can represent
four bits, a unit halve of a byte which is called sometimes a nibble.
In old times when it was necessary to type in binary numbers it was
sometimes easier to encode a nibble with a hexadecimal digit:

Decimal Binary Hexadecimal
0       0000   00

1       0001   01

2       0010   02

3       0011   03

4       0100   04

5       0101   05

6       0110   06

7       0111   07

8       1000   08

9       1001   09

10      1010   0A

11      1011   0B

12      1100   0C

13      1101   0D

14      1110   0E

15      1111   0F

The only location where we hear about hexadecimal characters again in
this book should be when we introduce the character and string data
types -- there control characters like a newline character are
sometimes specified in hexadecimal form like "\x0A" for a newline
character.

Installation of the Compiler

We will not describe in too much detail how you can install the Nim
compiler, because that strongly depends on your operating system, and
because the install instructions may change in the future. We assume
that you have a computer with an installed operating system and
internet access, and you are able to do at least very basic
operations with your computer, such as switching it on, log in and
and opening a web browser or a terminal window. If that is not the
case then you really should ask someone for help for this basic step,
and maybe for some more help for other basic tasks.

Detailed installation instructions are available on the Nim internet
homepage at To visit and read that page, you have to enter this
string in the address input field of your internet browser. Try to
follow those instructions, and when they are not sufficient, then
please ask at the Nim forum for help: https://forum.nim-lang.org/

If you are using a Linux operating system, then your system generally
provides a package manager, which should make the installation very
easy.

For example for a Gentoo Linux system you would open a root terminal
and simple type emerge -av nim. That command would install Nim
including all necessary dependencies for you. It may take a few
minutes as Gentoo compiles all packages fresh from source code, but
then you are done. Similar commands exist for most other Linux
distributions.

Another solution, which is preferable when you want to ensure that
you get the most recent Nim compiler, is compiling directly from the
latest git sources. That process is also easy and is described here:
https://github.com/nim-lang/Nim. But before you can follow those
instructions you have to ensure that the git software and a working C
compiler is available on your computer.

Creation of Source Code Files

Nim source code, as most source code of other programming languages,
is based on text files. Text files are documents saved on your
computer that contain only ordinary letters which you can type on
your keyboard. No images or videos, no HTML content with fancy CSS
styling. Generally source code should contain only ordinary ASCII
text, that is no umlauts or unicode characters.

To create source code we generally use a text editor, which is a tool
designed for creating and modifying of plain text files. If you do
not have a text editor yet you may also use a word processor for
writing some source code, but then you have to ensure that the file
is finally saved as plain ASCII text. Editors generally support
syntax highlighting, that is keywords, numbers and such are displayed
with a unique color or style to make it easier to recognize the
content. Some editors support advanced features like checking for
errors while you type the program source code.

A list of recommended editors is available at https://nim-lang.org/
faq.html

If you do not want to use a special editor now, then for Linux gedit
or at least nano should be available. For Windows maybe something
like notepad.

Generally we store our Nim source code files in its own directory,
that is a separate section of your hard-disk. If you work on Linux in
a terminal window, then you can type

cd
mkdir mynimfiles
cd mynimfiles
gedit test.nim

You type these commands in the terminal window and press the return
key after each of the above lines -- that is you type cd on your
keyboard and then press the return key to execute that command. The
same for the next three commands. What you have done is this: you go
to your default working area (home directory), then create a subarea
named mynimfiles, then you go into that subarea and finally you
launch the gedit editor -- the argument test.nim tells gedit that you
want to create a new file called test.nim. If gedit is not available,
or if you work on a computer without a graphical user interface, then
you may replace the gedit command by nano. While gedit opens a new
window with a graphical interface, nano opens only a very simple
interface in the current terminal. An interesting editor without a
GUI is vim or neovim. That is a very powerful editor, but it is
difficult to learn and it is a bit strange as you have a command mode
and an ordinary text input mode available. For neovim there is very
good Nim support available.

If you do not want to work from a terminal, or if you are using
Windows or MAC OS, then you should have a graphical user interface
which enables you also to create a directory and to launch an editor.

When the editor is opened, you can type in the Nim source code from
our previous example and save it to a file named test.nim. Then you
can terminate the editor.

Note that the return key behaves differently in editors than in the
terminal window: In the terminal window you type in a command and
finally press the return key to "launch" or execute the command. In
an editor the return key is not that special: if you press ordinary
keys in your editor, than that key is inserted and the cursor moves
one position to the right. And when you press the return key then an
invisible newline character is inserted and the cursor moves to the
start of the next line.

Launching the compiler and running the program

If you are working from a Linux terminal then you can type

ls -lt
cat test.nim

That is you first show the content of your directory with the ls
command and then display the content of the Nim source code file that
you just have typed in with the cat command.

Now type

nim c test.nim

That invokes the Nim compiler and instructs it to compile your source
code. The "c" letter is called an option, it tells the Nim compiler
to compile your program and to use the C backend to generate an
executable.

The compiler should display nearly immediately a success message. If
it displays some error messages instead, then you launch gedit or
nano again, fix your typing error, save the modified file and call
the compiler again.

Finally, when the source text is successfully compiled, you can run
your program by typing

./test

In your terminal window you see a number now, which is the sum of the
numbers 1 to 100.

You may wonder why you have to type the prefix ./ in front of the
name of your generated executable program, as you can launch most
other executables on your computer without such a prefix. The prefix
is generally needed to protect you and your computer from erroneously
launching a program in the current directory while you intended to
launch a system command. Imagine you downloaded a zip file from
internet, extract it, cd into the extracted directory and type ls to
see the directory content. Imagine now that the directory contains an
executable named ls which is executed instead of system ls. That
foreign ls command may damage your system. So to execute non system
commands, you generally have to use the prefix ./ where the period
refers to the current directory. Of course you can install your own
programs in a way that you don't need such a prefix any more -- just
ask your Mom or Grandma if you don't know yourself already.

If you have not managed to open a terminal where you can invoke the
compiler -- well maybe then you should install some of the advanced
editors like VS-Code. They should be able to launch the compiler and
run the program from within the editor directly.

The command

nim c test.nim

is the most basic compiler invocation. The extension .nim is
optional, the compiler can infer that file extension. This command
compiles our program in default debug mode, it uses the C compiler
back end and generates a native executable. Debug mode means, that
the generated executable contains a lot of checks, like array index
checks, range checks, nil dereference checks and many more. The
generated executable will run not very fast and it will be large, but
when your program has bugs then the program will give you a
meaningful error message in most cases. Only after you have tested
your program carefully you may consider compiling it without debug
mode. You may do that with

nim c -d:release test.nim

nim c -d:danger test.nim

The compiler option -d:release removes most checks and debugging code
and enables the backend optimization by passing the option "-O3" to
the C compiler backend, giving a very fast and small executable file.
The option -d:danger removes all checks, it includes -d:release. You
should be aware that compiling with -d:danger means that your program
may crash without any useful information, or even bad, may run, but
contain uncatched errors like overflows and so may give you wrong
results. Generally you should compile your program with plain nim c
first. When you have tested it well and you may need the additional
performance, you may switch to -d:release option. For games,
benchmarks or other uncritical stuff you may try -d:danger.

There exists many more compiler options, you can find them explained
in the Nim manual or you may use the command nim --help and nim
--fullhelp to get them displayed. One important new option is
--gc:arc to enable the new deterministic memory management. You may
combine --gc:arc with -d:useMalloc to disable Nim's own memory
allocator, this reduces the executable size and enables the use of
Valgrind to detect memory leaks. Similar to --gc:arc is the option
--gc:orc which can deal with cyclic data structures. Finally a very
powerful option is --passC:-flto. This option is for the C compiler
backend and enables link time optimization (LTO). LTO enables
inlining for all procedure calls and can significantly reduce the
final program size. For a recent Nim compiler version instead of
--passC:-flto also -d:lto can be used. We should mention that you can
also try the C++ compiler backend with the cpp command instead of
plain c command, and that you may compile with clang backend instead
of default gcc backend with the --cc:clang option. You can additional
specify the option -r to immediately run the program after successful
build. For testing small scripts the compiler invocation in the form
"nim r myfile.nim" can be used to compile and run a program without
generation of a permanent executable file. Here is an example how we
use all these options:

nim c -d:release --gc:arc -d:useMalloc --passC:-flto --passC:-march=native board.nim

In this example we additional pass -march=native to the C compiler
backend to enable use of the most efficient CPU instructions of our
computer, which may result in an executable that will not run on
older hardware. Of course we can save all these parameters in
configuration files, so that we don't have to actual type then for
each compiler invocation. You may find more explanations to all the
compiler options in the Nim manual or in later sections of this book,
this includes the options for the JavaScript backend.

Part II: The Basics

In this part we will introduce the most important constructs of the
Nim programming language, like statements and expression, conditional
and repeated execution, functions and procedures, iterators,
templates, exceptions and we will discuss various basic data types
including the basic container types array, sequence and string.

Declarations

We can declare constants, variables, procedures or our custom data
types. Declarations are used to give information to the compiler, for
example about the type of a variable that we intend to use.

We will explain type and procedure declarations in later sections.
Currently only constant and variable declarations are important.

A constant declaration in its simplest form maps a symbolic name to a
value, like

const Pi = 3.1415

We use the reserved word const to tell the compiler that we want to
declare a constant which we have named Pi and we assign it the
numeric decimal value 3.1415. Nim has a small set of reserved words
like var, const, proc, while and others, to tell the compiler that we
want to declare a variable, a constant, a procedure or that we want
to use a while loop for some repeated execution. Reserved words in
Nim are special symbols that have a special meaning for the compiler,
and we should avoid using these symbols as names for other entities
like variables, constants or functions, as that would confuse the
compiler. The = is the assignment operator in Nim, it assigns the
value or expression on the right side of it to the symbol on the
left. You have to understand that it is different from the equal sign
we may use in mathematics. Some languages like Pascal initially used
the compound operator := for assignments, but that is not easy to
type on the keyboard and looks a bit angry for sensible people. And
source code usually contains a lot of assignments, so use of = makes
some sense. We call = an operator. Operators are symbols which
perform some basic operation, like + for the addition of two numbers,
or = for the assignment of a value to a symbol. With the above
constant declaration we can use the symbol Pi in our program's source
code and don't have to remember or retype the exact sequence of
digits. Using named constants like our Pi above makes it easy to
modify the value -- if we notice that we need more precision, we can
look up the exact value of Pi and change the constant at one place in
our source code, we don't have to search for the digit sequence 3.14
in all our source code files.

For numeric constants like our Pi value the compiler will do a
substitution in the source code when the program is compiled, so
where we write the symbol Pi the actual numeric value is used.

For constant declarations it must be possible to determine its value
at compile time. Expressions assigned to constants can contain simple
operations like basic math, but some functions calls may be not
allowed.

Variable declarations are more complicated, as we ask the compiler to
reserve a named storage location for us:

var velocity: int

Here we put the reserved keyword var at the beginning of the line to
tell the compiler that we want to declare a variable, then we give
our chosen name for that variable followed by a colon and the data
type of the variable. The int type is a predefined numeric type
indicating a signed integer type. The storage capacity of an integer
variable depends on the operating system of your computer. On 32-bit
systems 32 bits are used, and on 64-bit systems 64 bits are used to
store one single integer variable. That is enough for even large
signed integer numbers: the range is -2^31 up to 2^31 - 1 for 32 bit
systems and -2^63 up to 2^63 - 1 for 64-bit systems.

For variables we generally use lower case names, but names of
constants may start with an upper case letter.

In some of the Nim documentation and in this book the terms 
declaration and definition may be used alternately, which is not
fully correct. Precisely a declaration is a statement that something
exists, while a definition is a more detailed description. In the C
programming language we differentiate between a function declaration,
which describes only the name of the function and the number and data
types of its parameters, and a function definition, which also has to
specify the names of the function parameters as well as the source
code of the function body. In Nim function declarations are not used
that often, as they are only really needed when two functions call
each other. For that case we declare the first function, so that we
can already use it in the definition of the other function, before we
finally also define the first function. For other entities like
constants, variables, data types or modules a distinction between the
terms declaration and definition makes not that much sense, that is
why we may use both terms alternately.

Statements

Statements, or instructions are a core component of Nim programs:
they tell the computer what it shall do. Often statements are
procedure calls, like the call of the echo() or inc() procedure which
we have already seen in part I of the book. We will learn what
procedures exactly are we will learn in later sections. For now, we
just regard procedures as entities that perform a well defined task
for us when we call (or invoke) them. We call them by just writing
their name in our source file, followed by a list of parameters, also
called arguments. When we write echo 7 then echo is the procedure
which we call, and 7 is the argument, an integer literal in this
case. When the parameter list has more than one argument, then we
separate the arguments with a comma each, and generally we put an
optional space after that comma. The effect of our procedure call is
that the decimal number 7 is written to the terminal when we run the
program after compilation. While in languages like C the parameter
list has to be always enclosed in brackets, in Nim we can often leave
the brackets out, which is called command invocation syntax.

const SquareOfFive = 5 * 5
echo(5 * 5, SquareOfFive) # ordinary procedure call
echo 5 * 5, SquareOfFive # command invocation syntax

The command invocation syntax is often used with the echo()
procedure, or when a procedure has only one single argument. For
multiple arguments, or when the argument is a complicated expression,
the use of brackets may be preferable. Some coding styles of other
programming languages, like C, sometimes put a space between the
procedure name and the opening bracket. For Nim we should not do
that, the reason will become clear when we later explain the tuple
data type. A few procedures have no parameters at all. When we call
these functions, we have to use always the syntax myProc() with an
empty pair of brackets, to make it for the compiler clear that we
want to call that function. res = myProc() assigns the result of the 
proc call to a, while res = myProc would assign the proc itself to a,
which is very different.

A special form of procedures are functions, that are procedures which
perform operations to return a value, or a result. In mathematics,
sin() or cos() would be functions -- we pass an angle as argument and
get the sine, or cosine as a result.

Let's look at this minimal Nim program:

var a: int
a = 2 + 3
echo a
echo(cos(0) + 2)

The Nim program above consists of a variable declaration and three
statements: in the first line we declare the variable we want to use.
In the next line we assign the value 2 + 3 to it, and finally in line
3 we use the procedure echo() to display the content of our variable
in the terminal window. In the last line we use again the echo
procedure with an ordinary parameter list enclosed in brackets. The
parameter list has only one parameter, which is the sum of a function
call and the literal value 2. Here the compiler would first call cos
(0) and then add the literal value 2 to that result, before finally
the sum is passed to the echo proc to print the value.^[13]

Nim programs are generally processed from top to bottom by the
compiler, and when we execute the program after successful
compilation, then it also executes from top to button. A consequence
of this is that we have to write the lines of above program exactly
in that order. If we moved the variable declaration down, then the
compiler would complain about an undeclared variable because the
variable is used before it has been declared. If we exchanged lines 2
and 3, then the compiler would be still satisfied, and we would be
able to compile and run the program. But we would get a very
different result, because we would first try to display the value of
variable a, and later assign a value to it.

When we have to declare multiple constants or variables, then we can
use a block, that is we write the keyword var or const on its own
line, followed by the actual declarations like in

const
  Pi = 3.1415
  Year = 2020
var
  sum: int
  age: int

These blocks are also called sections, .e.g. const section or var
section, as known from the wirthian languages. Note the
indentation -- the lines after const and var start with some space
characters, so they build a block which allows the compiler to detect
where the declaration ends. Generally we use two spaces for each
level of indentation. Other numbers would work also, but the
indentation scheme should be consistent. Two spaces is the general
recommendation, as it is clearly recognizable for humans in the
source code, and because it doesn't waste too much space, that is, it
would not generate long lines which may not fit onto the screen.

Also note that in Nim we generally write each statement onto its own
line. The line break indicates to the compiler that the statement has
ended. There are a few exceptions -- long mathematical expressions can
continue on the next line (see the Nim manual for details). We can
also put multiple statements on a single line when we separate them
by a semicolon:

var a: int
echo a; inc(a) (1)
a = 2 * a + (2)
  a * a

1 two statements separated by a semicolon on a single line
  a longer math expression split over multiple lines. An operator as
2 last character on a line indicates that the expression continues on
  the next, indented line.

We can also declare multiple variables of the same type in one single
declaration, like

var
  sum, age: int

or we can assign an initial start value to a variable like in

var
  year: int = 1900

Finally, for variable declarations we can use type inference when we
assign an initial start value, that is we can write

var
  year = 1900

The compiler recognizes in this case that we assign an integer
literal to that variable and so silently gives the variable the int
type for us. Type inference can be comfortable, but may make it
harder for readers to understand the code, or the type inference may
not always do exactly what we want. For example in the above code
year gets the type int, which is a signed 4 or 8 byte number. But
maybe we would prefer an unsigned number, or a number which occupies
only two bytes in memory. So use type inference with some caution.

Note: For integral data we mostly use the int data type in Nim, which
is a signed type with 4 or 8-byte size. It usually does not make
sense to use many different integral types -- signed, unsigned, and
types of different byte size. Mixing them in numerical expressions
can be confusing and maybe even decrease performance, because the
computer may have to do type conversion before it can do the math
operation. For unsigned types, another problem is that math
operations on unsigned operands could have a negative result.
Consider the following example where we use a hypothetical data type
"unsigned int" to indicate unsigned integers:

var a, b: unsigned int
a = 3
b = 7
a = a - b

The true result would be -4, but a is of unsigned type and can never
contain a negative content. So what should happen -- an incorrect
result or a program termination?

Related to variable declarations is the initial start value of
variables. Nim clears for us all the bits of our variables when we
declare them, that is, numbers get always the initial start value
zero if we do not assign a different value in the variable
declaration.

In this declaration

var
  a: int = 0
  b: int

both variables get the initial value zero.

There exists a variant for variable declarations which uses the let
keyword instead of the var keyword. Let is used when we need a
variable which only once gets a value assigned, while var is used
when we want to change the content of the variable during program
execution. Let seems to be similar to const, but in const
declarations we can use only values that are known at compile time.
Let allows us to assign to variables values that are only available
at program run time, maybe because the value is a result of a prior
calculation. But let indicates, at the same time, that the assignment
occurs only once, the content does not change later, during the
program's execution. We say that the variable is immutable. Use of
the let keyword may help the human reader of the source code with
understanding what is going on, and it may also help the compiler
doing optimizations to get faster, or more compact code. For now, we
can just ignore let declarations and use var instead -- later ,we may
use let where appropriate, and the compiler will tell us when let
will not work and we have to use var.

The way how we declare constants, variables, types and procedures in
Nim is very similar as it was done in the wirthian languages Pascal,
Modula and Oberon. People coming from languages like C argue
sometimes that C uses a shorter and better variable declaration of
the form int velocity; instead Nim's var velocity: int. Indeed that
declaration is shorter in this case. And some people like it better
that the data type is written first, they consider the data type more
important than the name of the variable. That is a matter of taste,
and the C notation would not work well for var/let/const distinction
and for type declarations.

With what we have learned in this section we can rewrite our initial
Nim example from part I in this form:

const
  Max = 100
var
  sum, i: int
while i < Max:
  inc(i)
  inc(sum, i)
echo sum

In the code above we declare both variables of type int in a single
line and take advantage of the fact that the compiler will initialize
them with 0 for us. And we use a named constant for the upper loop
boundary. Another tiny fix is that we write inc(i) instead of inc(i,
1). We can do that because there exists multiple procedures with the
name inc() -- one which takes two arguments, and one which takes only
one argument and always increases that argument by one. Procedures
with the same name but different parameter lists are called
overloaded procedures. Instead of inc(i) we could have written also i
= i + 1 and instead of inc(sum, i) we could write sum = sum + i. That
would generate identical code in the executable, so we can use
whatever we like better.

Input and Output

We have already used the echo() procedure for displaying textual
output in the terminal window. In the code examples of the previous
sections we always passed arguments of integer type to the echo proc,
and the echo proc automatically converted the integer numbers to a
textual sequence of decimal digits, so that we could read it in the
terminal. In the Nim programming language text is a predefined, built
in data type that is called string. We will learn all the details of
the string data type in the next section, for now it is sufficient
that it exists and that we can use the echo() p[.]roc to print text
strings. The echo() proc has the ability to convert other data types
like numbers or the boolean data type (true/false) automatically to
human readable text strings, so that we can read the output in the
terminal. Recall that most data types are stored internally in our
computer as bits and bytes, which have no true human readable
representation by default. Numbers, as most other data types stored
in the computer, are actual abstract entities. We have learned
already that all data in the computer is internally stored in binary
form, that is as a bit pattern of 0 and 1. But even that bit pattern
is still an abstraction, we would need a procedure that prints a 0
for each unset bit and a 1 for each set bit to display the content of
an internally stored number in binary form in the terminal or
elsewhere. In the same way we need a procedure to print an internally
stored number as a human readable sequence of decimal digits. Even
text strings are internally stored as abstract bit patterns, and we
need conversion procs to print the content for us as ordinary text.
All that can be done by the echo proc, but we do not care for the
actual details at this point of the book.

For our further experiments we may also want to be able to enter some
user data in the terminal. As we do not know much about the various
available data types and the procs that can be used to read them in,
we will just present a procedure that can read in a text string that
the user has typed in the terminal window. We use a function with the
name readLine() for this task.

echo "Please enter some text"
var mytext = readLine(stdin)
echo "you entered: ", mytext

Note that you have to press the return key after you have entered
your text.

The first line of our program show how we can print a text literal
string with the echo() proc. To mark text literals unambiguously and
to separate them from other literals like numeric literals or from
variables, the string literals have to be enclosed in quotation
marks. In the second line of our example program we use the readLine
() function to read textual user input. Note that we call readLine()
a function, not a procedure, to emphasize that it returns a value.
The readLine() function needs one parameter to know from where it
should read -- from the terminal window or from a file for example.
The stdin parameter indicates that it should read from the current
terminal window -- stdin is a global variable of the system (io)
module and indicates the standard input stream. Finally in line 3 we
use again the echo() procedure to print some text. In this case we
pass two arguments to echo(), a literal text enclosed in quotes, and
then separated by a comma, the mytext variable. The mytext variable
has the data type string. We used type inference in this example to
declare that data type: the readLine() procedure always returns a 
string, the compiler knows that, so our mytext variable is
automatically declared with type string. We will learn more about the
data type string and other useful predefined data types in the next
section.

Nim supports the method call syntax, which was earlier called Uniform
Function Call Syntax in the D programming language. With that syntax
we can write procedure calls in the form a.f instead of f(a). We will
discuss that syntax in more detail when we explain procedures and
functions. For now it is enough that you know about the existence of
that syntax, as we may use it at some places in the following
sections. For example for the length of text strings we generally
write myTextString.len instead of len(myTextString). Both notations
are fully equivalent.^[14]

When you try the example code from above, you may want a variant of
it that does read in the textual input not on its own line, but
directly after the prompt like "What is your name: Nimrod". As the 
echo proc writes always a newline character after the last argument
has been written, we have to use a different function to get the
input prompt on the same line. We can use the write() proc from the 
system module for this. As write() can not only write to the
terminal, but also to files, it needs an additional parameter which
specifies the destination. We can pass the variable stdout from the 
system module to indicate that write() should write to our terminal
window. Another desire of beginners is generally, to have the ability
to read single character input without the need to press additional
the return key. For that we can use the getch() function from the 
terminal module -- that functions waits (blocks) until a key is
pressed and returns the ASCII character of the pressed key:

from terminal import getch

stdout.write("May you tell me your name: ")
var answer = readLine(stdin)
if answer != "no":
  echo "Nice to meet you, ", answer
echo "Press any key to continue"
let c = getch()
echo "OK, let us continue, you pressed key:", c

Don't get confused by the fact that the first write() call and the
following readline() call does not appear on the same line in our
example. The actual format of our source code does in this case not
influence the program output. We could write both function calls on a
single line, separated with a semicolon. But that would make no
difference for the program output. The significant difference of the
code above is just, that write() prints the text, but does not move
the cursor in the terminal window to the next line, while echo()
moves the cursor to the next line when all arguments have been
printed. We say that echo prints automatically a "\n" character,
which we call newline character, after all the arguments have been
printed.

Data types

The most fundamental data type -- in real life and in computer
science -- are integer (whole) numbers. All other numeric data types,
like fractional, floating point or complex numbers, and other
fundamental types like the boolean type with its two values true and 
false, or character and text string types, can be represented as
integers. For that reason the early computers built in the 1950's as
well as today's tiniest microcontrollers work internally only with
integer numbers. As all CPUs are able to do basic bit operations like
setting or clearing individual bits, and as bit patterns map well to
mathematical sets, set data types are well supported by all CPUs, and
so set operations are generally very efficient. Advanced computers
built in the 1980's got support for the important class of floating
point numbers by special floating point processors for fast numerical
computations. These floating point units are today generally
integrated into the CPU, and GPUs can even process many floating
point operations in parallel, where the precision is often restricted
to ranges needed for games and graphics animation, that is 32 or even
16 bit. Modern CPUs have often also some form of support for vector
data types to process multiple values in one instruction (SIMD,
Single instruction, multiple data).

None numeric types like characters or text strings are internally
represented by integer numbers -- in the C language the data type to
present text strings is called char, but it is indeed only a 8 bit
integer type which supports all the mathematical operations defined
for ordinary int types. In Nim and the wirthian languages most math
operations are not directly allowed for the char data type, which
should prevent misuse and allows to catch logical errors by the
compiler.

Nim supports also some built in homogeneous container types like
arrays and sequences, and many built in derived types like enums,
sub-ranges and slices, distinct types and view types (experimental).
The built in inhomogeneous container types object and tuple, which
allow to group other types, are supported by a variant type
container, which allows instances of that type to contain different
child types at runtime. These inhomogeneous container types are
similar as the struct and union types from the C programming
language.

Other basic and advanced data types like complex and fractional
numbers, types with arbitrary-precision arithmetic as well as hash
sets and hash tables, dynamically linked list or tree structures are
available through the Nim standard library or external packages. And
of course we are able to define our own custom data types with our
own operators, functions and procedures working on them.

Note that all the data types that are build into the language, like
the primitive types int, float or char, as well as the built in
container types like tuple, object, seq and string, are written in
lower case, while data types that are defined by the Nim standard
library or that we define our self, by convention starts with a
capital letter like the CountTables type defined in the tables
module. Some people may regard this as an inconsistency, some may say
that in this way we can differentiate built in types from types
defined by libraries. At least we may agree that using capital
notation for common types as in Int, Float or String would be more
difficult to type and would look not that nice.

Integer types

We have already used the int data type, which indicates a signed
integer type of 4 or 8 byte size, depending on the operating system.
The reason why the size of that type depends on the word size of the
OS will become clear later, when we explain what references and
pointers are.

Beside the int data type, Nim has some more data types for signed and
unsigned integers: int8, int16, int32 and int64 are signed types with
well-defined bit and byte size, and uint8, uint16, uint32 and uint64
are the unsigned equivalents. The number at the end of the type name
is the bit size; we get the byte size when we divide that value by 8.
Additional we have the type uint, which corresponds to int and has
same size, but stores unsigned numbers only. ^[15] Generally we
should try to use the int type for all integral numbers, but
sometimes it can make sense to use the other types. For example, when
you have to work with a large collection of numbers, you know that
each number is not very big, and your RAM is not really that large,
then you may decide for example to use int16 for all your numbers. Or
when you know that your numbers will be really big and will not fit
in a 4 byte integer, then you may use the int64 type to ensure that
the numbers fit in that type even when your program is compiled and
executed on a computer with a 32 bit OS.

For integer numbers we have the predefined operators +, - and *
available for addition, subtraction and multiplication. Basically
these operations works as we may expect, but we have to remember that
we may get overflows. For signed ints we get compile- or run-time
errors in that case, while unsigned ints just wrap around, see
example at the end of this section. For division of integers we have
the operators div, mod, and / available. The div operator does an
integer division ignoring the remainder, mod is short for modulus and
gives us the remainder of the division, and / finally is currently
only predefined for the signed int type and gives us a fractional
result of data type float. That type is introduced in the next
section.

Remembering how div and mod behaves when the divisor or dividend are
negative can be confusing, and it may differ for other programming
languages. You may find a detailed justified explanation for the
concrete behaviour in the Nim manual and at Wikipedia.

Result of i div j
   -4 -3 -2 -1  0  1  2  3  4
-4  1  1  2  4    -4 -2 -1 -1
-3  0  1  1  3    -3 -1 -1  0
-2  0  0  1  2    -2 -1  0  0
-1  0  0  0  1    -1  0  0  0
 0  0  0  0  0     0  0  0  0
 1  0  0  0 -1     1  0  0  0
 2  0  0 -1 -2     2  1  0  0
 3  0 -1 -1 -3     3  1  1  0
 4 -1 -1 -2 -4     4  2  1  1

Result of i mod j
   -4 -3 -2 -1  0  1  2  3  4
-4  0 -1  0  0     0  0 -1  0
-3 -3  0 -1  0     0 -1  0 -3
-2 -2 -2  0  0     0  0 -2 -2
-1 -1 -1 -1  0     0 -1 -1 -1
 0  0  0  0  0     0  0  0  0
 1  1  1  1  0     0  1  1  1
 2  2  2  0  0     0  0  2  2
 3  3  0  1  0     0  1  0  3
 4  0  1  0  0     0  0  1  0

When performance matters we generally should try to use the "CPU
native" number type what for Nim is the int type. And we should try
to avoid using math expressions with different types, as the CPU may
have to do type conversion in that case before the math operation can
be applied. Adding two int8 types can on some CPU's be slower than
adding two ints, because the CPU may have to size extend the operands
before the math operation is performed. But this depends on the
actual CPU, and there are important exceptions: Multiplying two ints
would result in an int128 result if int size is 64 bit, which can be
slow when the CPU does not support that operation well. Another
important point to consider for maximum performance is the cache
usage. If you are performing operations on a large set of data, then
you may get a significant performance gain when large fractions of
your data fits in the caches of your computer, as cache access is
much faster than ordinary RAM access. So using smaller data types,
i.e. int32 instead Nim's default int which is int64 on a 64 bit OS
may increase performance in this special application.

When we use Nim on tiny microcontrollers, maybe even on 8-bit
controllers like the popular AVR devices, it may be best to use only
integers of well defined size like int8.

When we write integer literal numbers, then we use generally our
common decimal notation, as in var i = 100. To increase the
readability for long number literals we can use the underscore
character as in 1_000, that underscore character is just ignored by
the compiler. We can also write integer literals in binary, octal or
hexadecimal notation. For that we prefix the literal value with 0b, 
0o or 0x. The leading zero is necessary, and the next letter
indicates binary, octal or hexadecimal encoding. But such integer
literal notation is very rarely used.

More important is the actual size of integer literals, in particular
when we use type inference. Ordinary integer literals have the int
type, but integer literals not fitting in 32 bit have int64 type. We
can also specify the type of integer literals by appending the
literal with i8, i16, i32 or i64 for signed types and with u, u8, u16
, u32 or u64 for unsigned types. We can separate the actual number
and the suffix with a ' character, but that is not necessary for the
integer literals.

var
  a = 100 # int literal in decimal notation
  b = 1234567890000 # int64
  c = 5'i8 # 8 bit integer
  d = 7u16 # unsigned integer with 2 byte size
  e = 0b1111 # ordinary integer in binary notation, value is 15 in decimal notation
  f = 0o77 # integer in octal notation, value is 7 * 8^0 + 7 * 8^1 in decimal notation
  g = 0xFF # integer in hexadecimal notation

echo g, typeof(g)

In arithmetic expressions integer types of different sizes are
generally compatible when all the types are signed or unsigned, e.g.
in the example code from above we could write echo a + b + c, and
typeof(a + b + c) is int64, that is the expression is propagated to
the largest type of all the involved operands. But echo a + b + c + d
would not compile, as for such a mix of signed and unsigned operands
it is not clear if signed or unsigned arithmetic should be used. Also
note that even on a 64 bit OS echo typeof(a) is typeof(b) would print
false.

An important property of the current Nim implementation of A. Rumpf
used with the C backend is the fact that unsigned integers does not
generate overflow errors but simple wrap around:

var x: int8 = 0

while true:
  inc(x)
  echo x

Above code would print the numbers 0 up to 127 and then terminate
program execution due to an overflow error. But when we change the
data type to uint8 we would get a continues sequence of the numbers 0
up to 255. After the value 255 is reached, the value wraps around to 
0 again and the process continues. This behavior can lead to strange
bugs and is the reason that the Nim team generally recommends to
avoid unsigned integers.

For compatibility with external libraries Nim has also the integer
types cint and cuint, which exactly match the C types int and uint
when we compile for the C or C++ backend. For the JavaScript backend,
the LLVM backend or other backends these types may be also available,
for details you should consult the compiler documentation. For most
operating system and C compilers the int and uint types in C are 4
bytes in size, but there can be exceptions, so we better should not
write code that depends on the actual byte size of the types. The Nim
types cint and cuint are generally only used for parameter lists of (c)
library functions. To match other integer types like C char, short, 
long, longlong Nim supports these types when we put a c letter in
front of the name like clong. Again you should consult the Nim
compiler manual when you need more details, i.e. when you create
bindings to external libraries.

Floating point types

Another important numeric data type is float, for floating point
numbers. Floats are an approximation of real numbers. They can also
store fractions, and are most often printed in the decimal system
with a decimal point, or in scientific notation with an exponent.
Examples for the use of variables of float data type are

var
  mean = 3.0 / 7.9
  x: float = 12
  y = 1.2E3

The variable mean is assigned the result of the division of two float
literals -- the result has again the data type float, and so the
compiler can infer for the type of variable mean that same type. If
we printed the result of the division there would be a decimal point
and some digits following it. For variable x we specify the float
type explicitly and assign the value 12. We could use type inference
if we assigned 12.0, because the compiler can recognize by the
decimal point that we want a float, not an int variable. In line 3 we
use scientific notation for the float literal that we assign to y,
and the assigned value is 1.2 * 10^3 = 1200.0. Literal values like 
2E3 are also valid float literals -- the value would be 2000.0. But
literals with a decimal point and no digits before or after the
point -- 1. or .2 -- are not valid in Nim.

In the current Nim implementation float variables always occupy 64
bits. Nim has also the data type float64 which is currently identical
to plain float, and float32 which can store only smaller numbers and
has less precision.^[16] Floats can store values up to a magnitude of
approximately 1E308 with a positive or negative sign, and floats have
a typical precision of 16 digits. That is, when you do a division of
two arbitrary floats and print the result, you will get up to 16
valid digits. If you would try to print more than 16 significant
digits, then the additional decimal places would be just some form of
random garbage. Note: The number of significant digits of a floating
point number is the total number of digits before and after the
decimal point, but possible leading zero digits would not be counted.
The reason that leading zeros are not significant is just that in the
ordinary notation of numbers we always assume that there is just
nothing before the first non zero digit. For our car odometer 
001234.5 km is identical to 1234.5 km. And if we give our body size
as 1.80 m or 180 cm makes no difference, both values have 3
significant digits.

Generally we use floats whenever integers are not sufficient for some
reason. For example when we have to do complicated mathematical
operations which include fractional operands like Pi, or when we have
to do divisions and need the exact fractional value.

The float, float32 and float64 data types provides the +, -, * and /
operators for addition, subtraction, multiplication and division.
Unlike for the int types, for the float types we get never overflow
or underflow errors, and also no error for a division by zero. But
the result of an operation of two float operands can be a special
value like system.Inf, system.NegInf or system.NaN. The first two
indicate an over- or underflow, and NaN (Not a Number) indicates that
the result of an operation is not a valid number at all, for example
the result of a division by zero or the result of calculating the
square root of a negative number. This behaviour is sometimes called
saturated arithmetic. When a variable has one of these special
values, and we apply further math operations, then this value is
kept. So we can detect at the end of a longer mathematical
calculation if something went wrong -- we have not to check after each
single operation.^[17] An interesting property of floating point
numbers is, that when we test two variables of float type for
equality, and one has the value NaN, then the test is always false.
That is the test a == NaN is always false. When we forget this fact,
we may initialize a float variable to the value NaN and later test
with if a == NaN: to check if we have already assigned a value, which
is not what we really had in mind, as that test has always a negative
result. The actual test for the value NaN is a == a, which is only
false when a has the value NaN, or we may use math.isNaN(). More
useful constants and functions for the float data types can be found
in the std/fenv module, and functions working with floats like the
trigonometric ones are available from the std/math module.

For floats we have the operators +, -, * and / for addition,
subtraction, multiplication and division. For powers with integral
exponents you can use the ^ operator, but you have to import it from
the std/math module. The expression x ^ 3 is the same as x * x * x.
The math module contains many more functions like sin() or cos(), 
sqrt() and pow(). The function name sqrt() is short for square-root,
and pow() stands for power, so pow(x, y) is x to the power of y, when
both operands have type float. For performance critical code you
should always keep in mind that pow() is an actual function call,
maybe a call of an dynamic library which can not be inlined, so a
call of pow(x, 2) may be a lot slower than a plain x * x. And even
when using the ^ operator as in x ^ 3 we should be a bit critical.
But of course we always hope that the compiler will optimize all that
for us.

The operators +, -, * and / can be used also when one operand is a 
float variable and the other operand is an int literal. In that case
the compiler knows that we really intend to do a float operation and
converts the int literal automatically to float type. But when one
operand is a float variable and the other is an int variable, then an
explicit type conversion is necessary like in float(myIntVal) *
myFloatVal. One explanation why in this case the int value is not
automatically converted to float is, that this may mean a loss in
precision, as large int64 values can not be presented as an float.
Well, for int32 this reason does not really apply, but still there is
no automatic conversion. But indeed as Nim is used as a systems
programming language, it seems to be a good decision to need explicit
conversions in this case, as it makes it more clear what really is
intended. And generally we should try to avoid to use a lot
operations with mixed types, as that may make type conversions
necessary, which may cost performance. If we really do not care, we
may import the module std/lenientOps, which defines the arithmetic
operations for mixed operands.

Float literals have the float data type by default, but as for
integer literals we can also explicitly specify the data type: The
suffixes f and f32 specify a 32 bit float type, and d and f64 specify
a 64 bit type. We can separate the suffix from the actual number with
a ' character, but that is not required as long as there is no
ambiguity. We can also specify float literals in binary, octal or
hexadecimal notation, when we append one of these suffixes. In case
of hexadecimal notation, the ' is obviously needed to separate the
suffix, as f and d are valid hex digits.

As for integer variables Nim supports also the compatible types 
cfloat and cdouble which match the C types float and double when the
C backend is enabled. For most C compilers C float matches Nim's
float32 and C double matches Nim's float64.

Some CPUs and C compilers support also other floating point types
beside the common float32 and float64 types. Intel x86 compatible
CPUs generally support float80 math operations, and the gcc C
compiler may support float128. But these types are not yet supported
by the Nim compiler of A. Rumpf. But there may exists external
packages which supports these types by calling C functions, when the
C backend is used.

Two important properties of floats are that not all numbers can be
represented exactly and that math operations are not absolutely
accurate. Recall that in our decimal system, some fractions like 1/2
can be represented exactly as 0.5 in decimal notation, while others
like 1/3 can be only approximated as 0.3333...  As all data, floats are
stored internally in binary form following the IEEE Standard for
Floating-Point Arithmetic (IEEE 754). In that format some values,
e.g. the value 0.1, can not be represented exactly. As a result, some
simple arithmetic operations, executed in the computer, will give us
not exactly that result that we may expect. As we should really
remember this important fact, we will investigate this behaviour with
a small example program where we divide a few small integer numbers
after conversion to float by another to float converted integer n and
sum the result n times:^[18]

for i in 1 .. 10:
  echo "--"
  for j in 2 .. 9:
    let a = i.float / j.float
    var sum: float
    for k in 1 .. j:
      sum += a
    echo sum

which generates this output:

--
1.0
1.0
1.0
1.0
0.9999999999999999
0.9999999999999998
1.0
1.0
--
2.0 # for all iterations!
--
3.0 # for all iterations!
--
4.0
4.0
4.0
4.0
4.0
3.999999999999999
4.0
4.000000000000001
--
5.0
5.0
5.0
5.0
5.0
5.0
5.0
4.999999999999999
--
6.0
6.0
6.0
6.0
6.0
5.999999999999999
6.0
6.0
--
7.0
7.0
7.0
7.0
7.000000000000001
7.0
7.0
7.0
--
8.0
8.0
8.0
8.0
7.999999999999999
7.999999999999998
8.0
8.000000000000002
--
9.0 # for all iterations!
--
10.0
10.0
10.0
10.0
10.0
10.0
10.0
9.999999999999998

The echo() procedure prints up to 16 significant digits for a float
value, and so the accumulated tiny arithmetic errors become visible.
After our remarks above that should be not surprising any more, the
general solution is to round results to less than 16 decimal digits
before printing. Various ways to do that will be shown later in the
book. A related issue of float arithmetic is caused by scaling and
extinction. When we add numbers with very different magnitudes, the
result may be just the value of the largest number, as in echo 1.0 ==
1.0 + 1e-16 which prints true. The tiny summand is just too small to
actually chance the result, that is as when you switch on a torch on
a sunny day, it will not really become brighter. Maybe more
surprising is, that calling echo() with some simple float literals
will print a different value, like echo 66.04 which gives 
66.04000000000001 for Nim v1.6, while with Python3 we get 66.04
exactly. But indeed that is only surprising for people who do not
understand well what a statement like echo 66.04 really does: We know
already, that the value 66.04 is converted by the compiler to an
internally binary representation, and then converted back to a
decimal string when we run the program. So it is not that surprising
that in this process some tiny inaccuracies can accumulate. Actually
it is possible to get exact 16 digits precision when a very smart
conversion routine like the ryu or dragonbox algorithm is used.

From the discussions above it should be clear that testing two floats
for equality is often problematic. Instead for just testing for
equality we may better define a small epsilon value like eps = 1e-14
and then write (a - b).abs < eps. Seems to be not bad, and can be
seen often and works often, but not always. Imagine you write a
program which processes chemical elements and you work with atomic
mass and radii. So maybe the result with above test is that all the
atoms of the periodic table have equal mass and equal size, at least
when you should use the SI system with meter and kilogram as base
units. So a equality test like

const eps = 1e-16 # an arbitrary relative precision
if (a == 0 and b == 0) or (a - b).abs / (a.abs + b.abs) < eps: # avoid div by zero

if (a - b).abs / (a.abs + b.abs + 1e-32) < eps: # a similar check, avoiding also a div by zero

may be a better solution in the general case. Whenever you need a
general equality test you should think about the problem and do some
tests -- the code above are just untested possible examples.

At the end of this sections some remarks about the performance of
float data types compared to plain ints: On modern hardware like the
popular x86 systems for the basic operations performance of floats
and ints is very similar, addition, subtraction and even
multiplication is basically done in only one clock cycle, and
division may be a bit slower. Even operations like sqrt() which have
been regarded as slow in the past are now close to a plain addition
on modern hardware. As the CPU does its float arithmetic internally
with 64 or even with 80 bit, float32 is not faster than float 64, as
long as the operations are not memory bound, that is large data sets
are processed, so that it is an advantage when the data types are
smaller so that more of it fits into the cache. For tiny
microcontrollers and embedded devices things are very different, as
these devices generally have no floating point units. So the compiler
has to emulate all the float arithmetic, maybe by use of libraries.
This is very slow and produces large executables. So when writing
software for modern desktop PCs, there is no reason to try to avoid
float math, when solving the problem with float is easier. When the
data extends a very width range, e.g. from nm to millions of km, or
when operations like square root or trigonometric functions are
needed, then there is generally no way and reason to avoid float. In
the case that float or ints may work both, it is generally a good
strategy to try to use ints as first try. Ints may still provide
better performance for SIMD, threading and parallel processing, as
ints may avoid the expensive saving of floating point CPU registers.
For restricted hardware we should better try to avoid float math. But
all this is a difficult topic, and these advice can give you only
some simple recommendations, which may be wrong for a concrete case.
So finally you have to decide yourself, and as always it is a good
idea to do some performance tests.

References:

  * https://en.wikipedia.org/wiki/Floating-point_arithmetic

  * https://stackoverflow.com/questions/2100490/
    floating-point-inaccuracy-examples

  * https://forum.nim-lang.org/t/5983

Distinct types

Before we continue with subrange types we should introduce the
distinct types. In the real world we have a lot of quantities for
which the set of meaningful math operations is restricted and which
should not be mixed with quantities of other types. For example we
may have the quantities time and distance measured in seconds and
meters and mapped to the float or int data type. While adding seconds
and adding meters is a valid operation, adding seconds to meters
makes no sense and would be a program bug if it should occur in the
program code. But again dividing a distance by a time period
resulting in the average speed would be a valid operation. Nim
provides the distinct keyword which allows us to define new data
types that are based on existing types, but that are not compatible
with them or with other distinct types. And the new defined distinct
types have no predefined operations, we have to define all desired
operations our self.

type
  Time = distinct float # in seconds
  Distance = distinct float # in meters

var t: time = 0.2 # not allowed
var t: Time = Time(0.2)

For distinct types we have to define all the allowed operations our
self. We can convert distinct types to the base types and then use
operations of the base type or we can borrow operations from the base
type by use of the {.borrow.} pragma. Using distinct types can be
complicated when the new type should support many operations, but it
can make our code more save. For some data type with a very limited
set of operations distinct types can be used easily. Distinct types
are explained in detail in the Nim manual, we may explain then in
more detail in later sections. For now it is enough that we know
about their existence.

Subrange types

Sometimes it makes sense to limit the range of numeric variables to
only a sub-range. For this Nim uses the range keyword with this
notation: range[LowVal .. HighVal]. Values of this type can never be
smaller than LowVal or larger than HighVal. For Nim v1.6 we can
define range types also by leaving out the range[], that is by just
two constants separated by ...

type
  Year = range[2020 .. 2023] # software update required at least for 2024!
  Month = range[1 .. 12]
  Day = 1 .. 31 # same as range[1 .. 31]

var a: int = 0
var d: Day = 1 # OK
d = 0 # compile time error
d = a # run time test and error
echo d

For the above example the base type of the defined ranges is int, so
the ranges are compatible with the predefined int type, we can assign
values of int type to our range types and vice versa. In our example
the size of the range types is the size of the int base type, but of
course we could use other base types, like type Weekday = 1.int8 ..
7.int8. If we try to assign to a range type a value that falls not
into the allowed range then we get a compile-time or run-time range
error. This can help us to prevent or to discover errors in our
programs. Note that whenever we use range types, the compiler may
have to add additional checks to ensure that variables are always
restricted to the specified range. This check is active in debug mode
and also when we compile with option -d:release, and is only ignored
when we compile with -d:danger or explicitly disable range checks. So
using a lot of range types may increase code size and decrease
performance. For the example above, the line with the assignment d =
a generates a runtime check. An important and often used range type
is the data type Natural defined as range[0 .. int.high]. That type
is compatible with the int type and does not wraps around as uint
would do. It is often used as type for procedure parameters when the
arguments has to be not negative. In the proc body we sometimes copy
arguments of natural type to an ordinary integer -- that way we can
ensure a none negative start value, and can avoid many range checks
in the procedure body.

We can also declare sub-range types with float base types like type
Probability = range[0.0 .. 1.0].

Note that we can still mix different sub-range type:

var d: Day = 13
var m: Month = 3
d = d + m

Such an operation is generally a bug, to prevent it we can put the
distinct keyword in front of our ranges. But then again we have to
define the allowed operations our self or to borrow them from the
base type.

Enums

While enums in C are nothing more than integers with some special
syntax for creation, Nim's enums are more complex.

In Nim enums can be used whenever some form of symbols are needed
like the colors red, yellow and green of a traffic light or the
directions north, south, east and west for a map or a game.

Most of the time we declare an enum type and the corresponding values
by simple listing them like

type
  TrafficLight = enum
    red, yellow, green

We can use variables of type TrafficLight then like

var tl: TrafficLight
tl = green
if tl == red:
  tl = ...

Enums support assignment, plain tests for (un)-equality and for
smaller or greater. Additional the functions succ() and pred() are
defined for enums to get the successor or predecessor of an enum, ord
() or int() deliver the corresponding integer number and the $
operator can be used to get the name of an enum. We can also iterate
over enums, so we can print all the colors of our TrafficLight by

for el in TrafficLight:
  echo el.ord, ' ', $el

Ordinary enums start at 0 and uses continues numbers for the internal
numeric value, so that enums can be used as array indices.^[19]

type
  A = array[TrafficLight, string]

var a: A
a[red] = "Rot"
echo a[red]

But we can also assign custom numbers like

type
  TrafficLigth = enum
    red = -1, yellow = 3, green = 8

We should avoid that, as these "enums with holes" generate some
problems for the compiler and may be later deprecated. For example
array indexing or iterating is obviously not possible for enums with
holes.

It is also possible to set the string that the stringify operator $
returns, like in

type
  TrafficLigth = enum
    red = "Stop"
    yellow = (2, "Caution")
    green = ("Go")

Here the assigned numerical values should be 0, 2 and 3. Currently
the enums numerical values must be specified in ascending order
always.

When we have many enums in a program then name conflicts may occur,
for example we may have an additional enum type named BaseColor which
also has red and green members. For that case the {.pure.} pragma
exists:

type
  BaseColor {.pure.} = enum
    red, green, blue

With the pure pragma applied we can use the full qualified enum name
when necessary, like BaseColor.red. But we can still use unqualified
names like blue when there is no name conflict.

Boolean types

Boolean types are used to store the result of logic operations. The
type is called bool in Nim and can store only two values, false and 
true. Although we have only two distinct states for a boolean
variable and so one single bit would suffice to store a bool,
generally a whole byte (8 bits) is used for storing a boolean
variable. Most other programming languages including C do the same.
The reason is that most CPU's can not access single bits in the
RAM -- the smallest entity that can be directly accessed in RAM is a
byte. The default initial state of a boolean variable is false,
corresponding to a byte with all bits cleared.

var
  age = 17
  adult: bool = age > 17
  iLikeNim = true
  iLikeOtherLangaugeBetter = false.

In line three we assign to the variable adult the result of a logical
comparison. The next two lines assign the boolean constants true and 
false to the variables, with their type bool inferred.

Variables of type bool support the operators not, and, or and xor. 
Not inverts the logic value, a and b is only true when both values
are true, and false otherwise. And a or b is true when at least one
of the values is true, and only false when both values are false. Xor
is not used that often. It is called exclusive or, a xor b is false
when both values have the same logic state, that is when both are
true, or both are false. When the values are not the same, than the
result of the xor operator is true. The xor operator makes more sense
for bit operations, which we will learn later -- for the boolean type,
a xor b is identical to a != b.

When using conditional execution, some people like to write
expressions like if myBoolExp == false:, which is identical to if not
myBoolExp:. Well they may do, but please never write if myBoolExp ==
true:, that looks really too stupid.

Sometimes it is useful to know that false is mapped to the int value 
0, and true to the int value 1. That is similar to the C language,
but C has not a bool type, instead the numerical value 0 is
interpreted as false in conditional expressions, and all none zero
values are interpreted as true.

var a: int = 0
if cond:
  a = 7

a = 7 * cond.int

The effect of the last line is identical to the if statement above.
In very, very rare cases working with the actual int value of boolean
variables may make sense, but generally we should avoid that. Later
in the book there is a section about branchless code where we will
present a proc that actually may get faster by using such a trick.

Characters

The data type for single characters is called char in Nim. A variable
of type char has 8 bits and can store single characters. Indeed it
stores 8-bit integers which are mapped to characters. The mapping is
described by the ASCII table. For example the integer value 65 in
decimal is mapped to the character A. When we use single character
literals, then we have to enclose the letter in single quotes. As
only 8 bits are used to store characters, we only have 256 different
values, including upper and lower case letters, punctuation
characters and some characters with a special meaning like a newline
character to move the cursor in the terminal to the next line, or a
backspace character to move the cursor one position backwards. Single
characters are not used that often since we generally group them in
sequences called strings to build text.

The initial ASCII table contains only the characters with numbers 0
up to 127, here is an overview generated with the small program
listed in the appendix:

Visible ASCII Characters

      +0   +1   +2   +3   +4   +5   +6   +7   +8   +9  +10  +11  +12  +13  +14  +15
  0
 16
 32        !    "    #    $    %    &    '    (    )    *    +    ,    -    .    /
 48   0    1    2    3    4    5    6    7    8    9    :    ;    <    =    >    ?
 64   @    A    B    C    D    E    F    G    H    I    J    K    L    M    N    O
 80   P    Q    R    S    T    U    V    W    X    Y    Z    [    \    ]    ^    _
 96   `    a    b    c    d    e    f    g    h    i    j    k    l    m    n    o
112   p    q    r    s    t    u    v    w    x    y    z    {    |    }    ~

The position in the table is the sum of the number on the left and
the number on the top, i.e, character A has position 64+1=65, which
is the value the Nim standard function ord('A') or int('A') would
return. The characters with a decimal value less than 32 can not be
printed and are called control characters, like linefeed, carriage
return, backspace, audible beep and such. Character 127 is also not
printable, and is called DEL. An important property of this table is
the fact that decimal digits and upper- and lower-case letters form
contiguous blocks. So to test for example if a characters is an
uppercase letter we can use this simple condition: c >= 'A' and c <=
'Z'.

Characters with ord() > 127 are so called umlauts, exotic characters
of other languages, and some special characters. But these characters
may be different on different computers, as the characters depend on
the active code-page, which maps position to actual character, and
there are multiple code pages. When we need more than the plain ASCII
characters, then we use strings in Nim, which display many more
glyphs by using UTF-8 encoding.

The control characters with a decimal value less than 32 can not be
typed on the keyboard directly and for some characters with decimal
value greater than 126 it can be difficult to enter them on some
keyboards. For these characters as well as for all other characters
escape sequences can be used. Escape sequences start with the
backslash character, and the following characters are interpreted in
a special way: The backslash can follow a numeric value in decimal or
hexadecimal encoding, or a letter which is interpreted in a special
way. We mentioned already that the character 'A' is mapped to the
decimal value 65, which is its position in the ASCII table. So
instead of 'A' we could use the escape sequence '\65' for this
character. Or, as decimal 65 is 41 in hexadecimal notation (4 * 16^1
+ 1 * 16^0) we can use '\x41' where the x indicates that the
following digits are hexadecimal. For common, often used control
characters it is not easy to remember their numeric value, so another
notation with a letter following the backslash can be used. For the
important newline character we can use the decimal numeric value '\
10', the hexadecimal value '\xA' or the symbolic form '\n'. Here the
letter n stands for newline.

We can regard the backslash character, which introduces escape
sequences, as a special hinting symbol for the compiler: Caution, the
following characters must be interpreted in a special way.

It is important that you understand that all these escape sequences
are only a way to help the programmer to enter these invisible
control characters -- the compiler replaces the control sequences
immediately with the correct 8 bit value from the ASCII table, so in
the final compiled executable '\65' or '\n' are both only a plain 8
bit integer value:

var a, b: char
a = 'A'
b = '\65'
echo a, ord(a), b, ord(b) # if you don't know the output, read again this section and run this code.

The following table lists a few important control characters:

Decimal Hexadecimal Symbolic                 Meaning
10      xA          \n, \l   newline or linefeed -- move cursor one
                             position down

12      xC          \f       formfeed

9       x9          \t       tabulator

11      xB          \v       vertical tabulator

92      x5C         \\       backslash

39      x27         \'       single quote, apostrophe

7       x7          \a       alert, audible beep

8       x8          \b       backspace

27      x1B         \e       Escape, [ESC]

13      xD          \r, \c   return or carriage return -- move cursor
                             at the beginning of the line

The hexadecimal numbers after the \x character can be upper or lower
case and can have one or two hexadecimal digits. For symbolic control
characters like '\a' for alert the upper case variant '\A' seems to
be identical currently. The single quote entered as ''' does give an
error message, so you have to escape it as '\''. Unfortunately by
supporting this form of escaping it becomes impossible to enter a
backslash character directly, so we have to escape the backslash
character as '\\' to print a single backslash.

For Nim the most important control character is '\n' which is used to
start the output in a terminal window at the beginning of a new line.
But '\n' is generally not used as a single character but embedded in 
strings, that is sequences of characters. We will learn more about
strings soon. Note that the echo() function inserts a newline
character automatically after each printed line, but the write()
function does not:

echo 'N', 'i', 'm'
stdout.write 'N', 'i', 'm', '\n'

What may be a bit confusing is the fact that we use the backslash
character as escape symbol, and at the same time above table has an
entry '\e' which is also called [ESC]. These '\e' control character
with decimal value 27 is fully unrelated to the backslash character
that we use to type in control characters. [ESC] is a different
special character to start control sequences, it was used in the past
to send special commands to printers or modems, and can be used to
control font style or colors in terminal windows.

Nim's control characters should, with few exceptions, be identical
with control characters of the C language, so you may also consult C
literature for more details.

Ordinal types

In Nim integers, enumerations, characters and the boolean types are
ordinal types. Ordinal types are countable and ordered, and for each
of these types a lowest and largest member exists. The integer
ordinal types supports the inc() and dec() operations to get the next
larger or next smaller value, and the other ordinal types use succ()
and pred() for this operation. These operations can produce overflow
or underflow like errors if applied to largest or smallest value. The
function ord() can be used on ordinal types to get the corresponding
integer value. Note that unsigned integers are currently not called
ordinal types in Nim, and that these unsigned types wrap around
instead of generation overflow and underflow errors.

Sets

From mathematics we know that sets are some form of unordered
collection for which we can test membership (x is included in mySet)
and we can perform general set operations like union of multiple
sets. In Nim we can have sets of all the ordinal types and the
unsigned integer types, but due to memory restrictions integer types
larger than two bytes can not be used as set base types. All elements
in a set must have the same base type. A set can be empty, or it can
contain one or multiple elements. For a specific element it can be
contained in a given set or it can be not contained, but it can be
never contained multiple times. One very basic set operation is the
test if an element is contained in a set or is not contained in it.
Sets are unordered data types, that is sets containing the same
elements are always equal, it does not matter in which sequence we
added the elements. Important set operations are building the union
and building the difference of two sets with the same base type: The
union of set a and set b is a set which contains all the elements
that are contained in set a or in set b (or in both). The
intersection of set a and set b is a set which contains only elements
which are contained in set a and in set b.

The mathematical concept of sets maps well to words and bits of
computers, as most CPU's have instructions to set and clear single
bits and to test if a bit is set or unset. And CPU's can do and, or
and xor operations which corresponds to the union and intersection
operation in mathematical set.

Nim supports sets with base type bool, enum, char, int8, uint8, int16
and uint16. Note that we need a bit in the computer memory for each
member of the base type. The types char, int8 and uint8 are 8 bit
types and can have 2^8 = 256 distinct values, so we need 256 bits in
the computer memory to represent such a set. That would be 32 bytes
or four 64 bit words. To represent a set of the base type uint16 or
int16 we need already 2^16 bits, that is 2^13 bytes or 2^10 words on
a 64 bit CPU. So it becomes clear that supporting base types with
more than 16 bit makes not much sense.

While testing if an element is included or is not included in a set
with the in or notin operators is always a very fast operation, other
operations like building the intersection or union and set comparison
operations may be not that fast when we use the int16 or uint16 base
types, as for these operations the whole set, that is 2^10 words on a
64 bit CPU, has to be processed.

We will start our explanation with sets with character base type as
these sets are very easy to understand and at the same time very
useful. Let us assume that we have a variable of character type and
we want to test if that variable is alphanumeric, that is if it is a
lower or upper case letter or a digit. A traditional test would be (x
>= a and x <=+z) or (x +>= A and x <= Z) or (x >= 0 and x <= 9).
Using Nim's set notation we can write that in a simpler form:

const
  AlphaNum: set[char] = {'a' .. 'z', 'A' .. 'Z', '0' .. '9'}

var x: char = 's'
echo x in AlphaNum

Here we defined a constant of set[char] type which contains lower and
upper case letters and the decimal digits. We used the range notation
to save us a lot of typing ({'a', 'b', 'c', ... }). It works in this
case only, as we know that all the lowercase letters, the upper case
letters and the decimal digits built an uninterrupted range in the
ASCII table.

With that definition we can use a simple test with the in keyword.
These test is equivalent to the procedure call AlphaNum.contains(x).
In compiled languages (most) set operations are generally very fast
as they map well to CPU instructions.

Older languages like C have not a dedicated set data type, but as
sets are so useful and efficient, C emulates these operations by
using bit-wise and and or operations in conjunction with bit shifts.

Two important operations for sets are building the union and the
intersection:

const
  AlphaNum: set[char] = {'a' .. 'z', 'A' .. 'Z', '0' .. '9'}
  MathOp = {'+', '-', '*', '/'} # set[char]

  ANMO = AlphaNum + MathOp # union
  Empty = AlphaNum * MathOp # intersection

The constant ANMO would now contain all the characters from AlphaNum
and from MathOp, that is letters, digits and math operators. The
constant Empty would get all the characters that are at the same time
contained in set AlphaNum and in set MathOp. As there is not a single
common characters, the set Empty is indeed empty. Remembering the two
operators + and * for union and intersection is not easy. For the
intersection operator * it may help when we imagine the set members
as bits, and we assume that we multiply the bits of both operands
bitwise, that is we multiply the set or unset bits at corresponding
position each. The resulting bit pattern would get set bits only for
positions where both arguments have set bits.

We can use the functions incl() and excl() to add or remove single
set members:

var s: set[char]
s = {} # empty set
s = {'a' .. 'd', '_'}
s.excl('d')
s.incl('?')

The result is a set with letters a, b, c and the characters _ and ?.
Note that calling incl() has no effect when the value is already
included in the set, and calling excl() has no effect when the value
is not contained in the set at all.

Another operation is the difference of two sets -- a - b is a set
which contains only the elements of a which are not contained in b.
In Nim there is currently no operator for the complement or the
symmetric difference of sets available. We can produce a set
complement by using a fully filled set and then removing the elements
of which we want the complement. For a character set that would look
like {'\0'..'\255'} - s, where s is the set to complement. And the
symmetric difference of set a and set b can be generated by the
operation (a+b) - (a*b) or by (a-b) + (b-a).

As the not operator binds more tightly than the in operator, we have
to use brackets for the inverted membership test like not(x in a) or
we can use the notin operator and write x notin a. We can test for
equality of sets a and b like a == b and for subset relation a < b or
a <= b. a <= b indicates that b contains at least all members of a,
and a < b that b contains all members of a and at least one more
element.

Finally we can use the function card() to get the cardinality of a 
set variable, that is the number of contained members.

We should also mention that we can have character sets which are
restricted to a range of characters:

type
  CharRange = set['a' .. 'f']

# var y: CharRange = {'x'} #invalid

var y: CharRange = {'b', 'd'}
echo 'c' in y

In the code above the compiler detects the assignment to variable y
as invalid.

Set of numbers work in principle in the same way as sets of
characters. One problem is that in Nim integer numbers are generally
4 or 8 bytes large, but sets can contain only numbers with 1 or 2
byte size. So we have to specify the type of set members explicitly:

type
  ChessPos = set[0'i8 .. 63'i8]

var baseLine: ChessPos = {0.int8 .. 7.int8}
var p: int8
echo p in baseLine

In the code above we defined a set type which can contain int8
numbers in the range 0 to 63.

We can use also another notation for numeric sets when we define an
explicit range type like in

type
  ChessSquare = range[0 .. 63]
  ChessSquares = set[ChessSquare]

const baseLine = {0.ChessSquare .. 7.ChessSquare}
# or
const baseLineExplicit: ChessSquares = {0.ChessSquare .. 7.ChessSquare}
assert baseLine == baseLineExplicit

What may be a bit surprising is the fact that Nim's sets work also
for negative numbers:

type
  XPos = set[-3'i8 .. +2'i8]

var xp: XPos = {-3.int8 .. 1.int8}
var pp: int8 = -1

Enum sets are also very useful and can be used to represent multiple
boolean properties in a single set variable instead of using multiple
boolean variables for this purpose:

type
  CompLangFlags = enum
    compiled, interpreted, hasGC, isOpenSource, isSelfHosted
   CompLangProp = set[CompLangFlags]

const NimProp:  CompLangProp = {compiled, hasGC, isOpenSource, isSelfHosted}

Enum sets can be used to interact with functions of C libraries where
for flag variables often or'ed ints are used. For example for the
gintro C bindings there is this definition:

type
  DialogFlag* {.size: sizeof(cint), pure.} = enum
    modal = 0
    destroyWithParent = 1
    useHeaderBar = 2

  DialogFlags* {.size: sizeof(cint).} = set[DialogFlag]

Here the {.size.} pragma is used to ensure that the byte size of that
set type matches the size of integers in C languages.

When we define set of enums in this way to generate bindings to C
libraries, then we have to ensure that the enum values start with
zero, otherwise Nim's definition will not match with the C
definition. For example in the gdk.nim module we have

type
  AxisFlag* {.size: sizeof(cint), pure.} = enum
    ignoreThisDummyValue = 0
    x = 1
    y = 2
    pressure = 3
    xtilt = 4
    ytilt = 5
    wheel = 6
    distance = 7
    rotation = 8
    slider = 9

  AxisFlags* {.size: sizeof(cint).} = set[AxisFlag]

The first enum with ordinal value zero was automatically added by the
bindings generator script to ensure type matching. Nim devs sometimes
recommend to use plain (distinct) integer constants for C enums. That
may be easier, but integer constants provide no name spaces, so names
may be aFlagWheel instead of AxisFlag.wheel or plain wheel when there
is no name conflict for pure enums. And with integer constants we
have to combine flags by or operation like (aFlagWheel or
aFlagSlider) instead of clean {AxisFlag.wheel, slider}.

Can we print sets easily? As sets are an unordered type, it is not
fully trivial, but we can iterate over the full base type and check
if the element is contained in our set like

var s: set[char] = {'d' .. 'f', '!'}

for c in 0.char .. 255.char:
  if c in s:
    stdout.write(c, ' ')
echo ' '

! d e f

We will learn how the for loop work soon. Note that the sequence in
which the set members are printed is determined by our query loop,
not by the set content itself, as sets are unordered types.

Strings

The string data type is a sequence of characters. It is used whenever
a textual input or output operation is performed. Usually it is a
sequence of ASCII-only characters, but multiple characters in the
string can be interpreted as so called utf-8 unicode characters, that
allow displaying nearly unlimited symbols as long as all the needed
fonts are installed on your computer and you manage to enter
them -- unicode characters may be not accessible by a simple
keystroke. For now we will only use ASCII characters, as they are
simpler and work everywhere. String literals must be enclosed in
double quotes. Nim strings are similar to the Nim seq data types:
both are homogeneous variable-size containers. That means that a
string or a seq expands automatically when you append or insert
characters or other strings. Don't confuse short strings with only
one character with single characters: A string is a non trivial
entity with internal state like data buffer (the actual contained
characters), length and storage capacity why a variable of char type
is nothing more than a single byte interpreted in a special way. So a
string like "x" is fully different from 'x'.

var
  str: string = "Hello"
  name: string
echo "Please tell me your name"
name = readLine(stdin)
add(str, ' ')
echo str, name

In the above example code we declare a variable called str and assign
it the initial literal value "Hello". We use the echo() procedure to
ask the user for his name, and use the readLine() procedure to read
the user input from the terminal. To show how we can add characters
to existing string variables we call the add() procedure to append a
space character to our str variable, and finally call the echo()
procedure to print the hello message and the name to the screen. Note
that the echo() procedure automatically terminates each output
operation with a jump to the next line. If you want an output
operation without a newline, you can use the similar write()
procedure. But write() needs an additional first parameter, for which
we use the special variable stdout when we want to write to the
terminal window.

So we could substitute the last two lines of the above code by

write(stdout, str)
write(stdout, ' ')
echo name

The Nim standard library provides a lot of functions for creating and
modifying strings, most of these functions are collected in the 
system and in the strutils module. The most important procedures for
strings are len() and high(). The len() procedure returns the length
of a string, that is the number of ASCII characters or bytes that the
string currently contains. The empty string "" has length zero. Note
that the plain len() function returns the number of 8-bit characters,
not the number of unicode glyphs when the string should be
interpreted as unicode text. To determine the number of glyphs of
unicode strings you should use some of the unicode modules. The high
() function is very similar to the len() function, it returns the
index of the last character in the string. For each string s high(s)
== len(s) -1, so high("") is -1. Remember that Nim supports method
call syntax, so we can also write s.len instead of len(s).

The most important operators for strings are the subscript operator 
[] which allows access to individual characters of strings, and the 
.. slice operator which allows access to sub-strings. The first
character in a string has always the index zero. For concatenation of
string literals or string variables Nim uses the & operator.

var s = "We hate " & "Nim?"
s[3 .. 6] = "like"
s[s.high] = '!'

In the example above we define the string variable s by use of two
literal strings to show the use of the concatenation operator. In
line two we use the slice operator to replace the sub-string "hate",
that is the characters with index position 3 up to 6, with the string
literal "like". In this case the replacement has exactly that many
characters as the text to replace, but that is not necessary: We can
replace sub-strings with longer or shorter strings, which includes
the empty string "" to delete a text area. In the last line of above
example we use the subscript operator [] to replace the single
character '?' at the end of our string with an exclamation mark. For
subscript and slice operators Nim supports also a special notation
which indicates indexing from the end of the string. Python and Ruby
use negative integers for this purpose, while Nim uses the ^
character. So [^1] is the last character, [^2] the one before the
last. So we could have written s[^1] = '!' for the last line of our
code fragment above. The reason that Nim does not use negative
integers for this purpose is that Nim arrays don't have to start at
index zero, but can start with an arbitrary index including negative
indices, so for negative indices it may be not always clear if a
regular index or a position from the end of the string is desired.
The term s[^x] is equivalent to s[s.len - x]. We will learn some more
details about the slice operator in a later section when we have
introduced arrays and sequences.

Another important operator for strings is the "toString" or stringify
operator $. It can be applied to variables of nearly all data types
and returns its string representation which can then be printed. Some
procedures like echo() apply this operator on its arguments
automatically. When we define our own data types then it can make
some sense to define the $ for them, in case we need a textual
representation of our data -- maybe only for debugging purpose. Note
that applying the $ operator on a string directly has no effect and
is ignored, as the result would not change.

Strings can contain all characters of the char data type including
the control characters. The most important control character for
strings is the newline character '\n' which is used at the end or
sometimes also in the middle of strings to start a new line. For
strings Nim also supports the virtual character "\p" to encode an OS
dependent line break. When compiled for Windows, "\p" is
automatically converted to "\r\n", and to a plain '\n' on Linux. Note
that "\p" can be used in strings, but not as single character, as it
is two byte on Windows. "\p" is only needed to support very old
Windows version or maybe other exotic operating system, as modern
Windows recognizes plain '\n' well.

As strings support utf-8 unicode, an escape sequence starting with "\
u" is supported to insert unicode codepoints. The "\u" follows
exactly 4 hexadecimal digits or an arbitrary number of hex digits
enclosed in curly braces {}.

As string literals are enclosed in quotation marks, it follows that
strings can not directly contain this character, we have to escape it
as in "\"Hello\", she said".

Maybe we should mention that Nim strings use copy semantics for
assignment. As we have not yet introduced references or pointers,
copy semantics is what you should expect -- strings behave just like
all the other simple data types we used before like integer or float
numbers or enums and characters:

var
  s1: string
  s2: string
s1 = "Nim"
s2 = s1
s1.add(" is easy!")
echo s1 & "\n" & "s2"

The output is

Nim is easy!
Nim

The assignment s2 = s1 creates a copy of s1, so the subsequent add()
operation does only modify s1 but not s2. Probably not surprising for
you, but other programming languages may behave differently, i.e. the
assignment may not copy the textual content but create only a
reference to the first string, so that modifying one of then also
effect the other. We will learn more about the concept of references
when we introduce the object data type.

Entering Unicode Characters

UTF-8 is a variable-width character encoding. To cite the introducing
section from https://en.wikipedia.org/wiki/UTF-8:

UTF-8 is capable of encoding all 1,112,064[nb 1] valid character code
points in Unicode using one to four one-byte (8-bit) code units. Code
points with lower numerical values, which tend to occur more
frequently, are encoded using fewer bytes. It was designed for
backward compatibility with ASCII: the first 128 characters of
Unicode, which correspond one-to-one with ASCII, are encoded using a
single byte with the same binary value as ASCII, so that valid ASCII
text is valid UTF-8-encoded Unicode as well. Since ASCII bytes do not
occur when encoding non-ASCII code points into UTF-8, UTF-8 is safe
to use within most programming and document languages that interpret
certain ASCII characters in a special way, such as "/" (slash) in
filenames, "\" (backslash) in escape sequences, and "%" in printf.

In Nim we have four ways to enter unicode characters: As hexadecimal
digits following the "\x", as unicode codepoint following the "\u" or
we can type the unicode sequence directly on our keyboard, as one
single keystroke when our keyboard layout supports that, or as a
special OS dependent sequence of keystrokes:

echo "\xe2\x99\x9A \xe2\x99\x94"
echo "\u265A \u2654"
echo "\u{265A} \u{2654}" # {} is only necessary for more than 4 hex digits
echo " "

The code above shows three ways to print the symbol for a black and a
white king of a chess game. In the first line we typed the unicode
sequence directly as hexadecimal digits, this method is rarely used
today. In the second line we used "\u" to enter the codepoint
directly, we get the code from https://en.wikipedia.org/wiki/
List_of_Unicode_characters. And finally we entered the glyph directly
in an editor. For some Linux editors like gedit we can hold down
shift and control key and then type u, release all keys and type the
unicode digits like 265a and a space. See https://en.wikipedia.org/
wiki/Unicode_input for details and other operating system.

The CString data type

In the C programming language strings are just pointers to sequences
of characters of fixed length.^[20] The end of such a C string is
generally marked with the character '\x0' -- a null byte with all bits
cleared. C functions like printf() needs these "\x0" characters to
determine the end of the C string. While Nim strings are complex
entities that store its current size and other properties, and can
grow dynamically, the character sequence of Nim strings has also a
hidden terminating '\x0' character at the end to make them compatible
with C strings. Nim has also the data type cstring, called
"compatible string" in modern Nim, which matches the strings in C
language if we compile as usual with the C backend. Cstrings are used
in bindings definitions for C libraries, but as cstrings can not grow
and do support only few string operations, they are only used in rare
cases in ordinary Nim source code. The Nim compiler passes
automatically the zero terminated data buffer of Nim strings to C
libraries whenever we call a C library, so there is no expensive type
conversion involved. But the other way is much more expensive: When
you have an existing cstring and need a Nim string with same content,
then a simple conversion is not possible as a Nim string is a
different, more complex entity. So we have to create a Nim string and
copy the content, you can use the stringify operator $ for this like
in myNimStr = $myCString. Generally string creation is an expensive
operation compared to plain operations like adding two numbers, so
when performance matters one should try to avoid unnecessary string
creation and also unnecessary string operations. This is mostly
important in loops, which are executed often. We will explain more
about the internal of strings and why string creation and dynamically
allocating memory is expensive in later sections of the book.

When we access text ranges with the slice operator or single
characters with the subscript operator we should never access indices
below the currently last index, which is the index mystr.high or ^1.
If we do that we get an exception, as that index would contain
undefined data or would not exist at all. We said earlier that Nim
strings grow automatically if we insert or append data. But that does
not mean that we can use the subscript or slice operator to access
characters after the current end of the string. Such an operation
would really make not much sense: Imagine we have a string var str =
"Nim" and now use the subscript operator and assign a character at
position 10 with str[10] = '!'. What should become the content of
characters 4 .. 9? Well maybe spaces would make some sense, but in
fact such an access after the currently last valid character of the
string is forbidden. You could do str.add(" !") for this purpose.

Another operation you should avoid is inserting the '\x0' null byte
character somewhere in an existing Nim string. Nim stores the actual
length of strings explicitly and additional terminates the end of the
actual data with a '\x0' to make the string compatible with C strings
and allow passing the data buffer directly to C library functions. A 
'\x0' character somewhere in the middle of a Nim string would
generate an inconsistency, as C library functions like printf() would
regard '\x0' as the string end marker, while pure Nim functions may
assume still a longer string. Intermediate '\x0' bytes in strings can
in very rare cases be a problem when we get the actual byte sequence
from C libraries. For the same reason a Nim string is not identical
or fully compatible with s seq[char], as a seq[char] may contain
multiple zero byte data, while Nim strings should not.

Escape Sequences in Strings

We learned about control characters already in the section about
characters, and earlier in this section we mentioned that strings can
also contain control characters. As the use of control characters may
be not really easy to understand, we will explain their use in
strings in some more detail and give a concrete example.

The most important control characters for strings is the newline
character, which moves the cursor in the terminal window to the
beginning of the next line. The echo() procedure prints that
character automatically after each output operation. Indeed it can be
important to terminate each output operation with that character, as
the output can be buffered, and writing just a string without a
termination newline may not appear at once on the screen, but can be
delayed. That is bad when the user is asked something and should
respond, but the message is still buffered and not yet visible.

The problem with special characters like backspace or newline is that
we can not enter them directly with the keyboard.^[21] To solve that
problem, escape sequences were introduced for most programming
languages. An escape sequence is a special sequence of characters,
that the compiler can discover in strings and then replace with a
single special character. Whenever we want a newline in a string we
type it as "\n", that is, the backslash character followed by an
ordinary letter n, n for newline.

echo "\n"
echo "Hello\nHello\nHello"

The first line prints two empty lines -- two because the \n generates
a jump to next the line, and because echo() always adds one more jump
to the next line. The second line prints three lines which each
contains the word Hello, and the cursor is moved below the last Hello
, because echo() automatically adds one more newline character.

Older Windows versions used generally a sequence of two control
characters to start a new line, one '\r' (carriage-return) to move to
the start of the line, and one '\l' (linefeed) to move down. You may
still find these two characters in old Windows text files at the end
of each line. Old printers used these combination too, so it was
possible to send that text files to old printers directly. Nim also
has the special escape sequence "\p" which is called platform
dependent newline and maps to "\c\l" on Windows. That is when we
compile our program on Windows, then the compiler replaces "\p" in
our strings with a carriage-return and a linefeed character, and when
we compile on Linux then the compiler replaces "\p" only with a
newline character. But modern Windows supports '\n', so we generally
can use that.

Raw Strings and multi-line Strings

In rare situations you may want to print exactly what you have typed,
so you do not want the compiler to replaces a '\n' by a newline
character. You can do that in two ways: You can escape the escape
character, that is you put in front of the backslash one more
backslash. When you print the string "\\n" you will get a backslash
and the n character in your terminal. Or you can use so called raw
strings, that is you put the character r immediately in front of your
string like

echo r"\n"
echo "\\n"

Multi-line strings are also raw strings, that is contained
escape-sequences are not interpreted by the compiler, and additional
multi-line strings, as the name implies, can extend over multiple
lines of the source text. Multi-line texts starts and ends with three
quotes like in

echo """this is
three lines
of text"""

echo "this is\nthree lines\nof text"

Both echo() commands above generates exactly the same machine code!

Comments

Comments are not really a data type, but they are also important.
Ordinary comments starts with the hashtag character # and extend to
the end of the line. The \# character itself and all following
characters up to the line end are ignored by the compiler. You can
also start the comment with \\##, then it is a documentation comment.
It is also ignored by the compiler, but can be processed when you use
later tools to generate documentation for your code. Documentation
comments are only allowed at certain places, often they are inserted
at the beginning of a procedure body to explain its use. There are
also multi-line comments, which starts with the two characters #[ and
ends with ]#. These form of comment can extend over multiple lines
and can be nested, that is multi-line comments can contain again
plain or multi-line comments.

# this is comment
## important note for documentation
#[ a longer
but useless comment
#]

Multi-line documentation comments also exists and can also be nested.

proc even(i: int): bool =
  ##[ This procedure
  returns true if the integer argument is
  even and false otherwise.
  ]##
  return i mod 2 == 0

You can also use the #[ comment ]# notation to insert comments
everywhere in the source code at places where a white-space character
is allowed, but these form of in source comments is rarely used.

Other data types

There exists some more predefined types like the container types 
array and seq, which can contain multiple elements of the same type,
or the tuple and the object type which can contain data of different
types. Nim tuples and objects are similar to C structs, they are not
so verbose as Java classes. We will learn more about all these types
in later sections of the book.

Nim Source Code

You have already seen a few examples of simple Nim source code. The
code is basically a plain text file consisting of ASCII characters,
that is the ordinary characters which you can type on your keyboard.
Generally Nim source code can also contain unicode utf-8 characters,
so instead of using names consisting of ASCII characters for your
symbol names, you could just use single unicode characters or
sequences of unicode characters. But generally that makes not much
sense, entering unicode is not that easy with a keyboard, and it is
displayed only correctly on the screen or in the terminal when the
editor or terminal supports unicode properly and when all necessary
fonts are installed. That may be the case for your local computer,
but what when someone other may edit your source code?

Starting with Nim version 1.6 we got some support for unicode
operators, which may be useful for some applications. For details
please see the Nim compiler manual.

Nim currently does not allow to insert tabular characters (tabs) in
your source code, so you have to do the indentation of blocks by
spaces only. Generally we use two spaces for each indentation level.
Other numbers work also but you should stick to a fixed value.

Names in Nim, as used for modules, variables, constants, procedures,
user defined types and other symbols may contain lower and upper case
letters, digits, unicode characters and additional underscores. But
the names are not allowed to start with digits or to start or end
with an underscore, and one underscore may not follow directly after
another underscore.

var
  pos2: int # OK
  leftMargin: int # OK
  next_right_margin: int # OK
  _privat: int # illegal
  custom_: int # illegal
  strange__error: int # illegal

Generally we use camel case like leftMargin for variable names, not
snake case like left_margin.

Current Nim has the special property that names are case insensitive
and that underscores are simple ignored by the compiler. The only
exception is the first letter of a name, that letter is case
sensitive. So the names leftMargin, leftmargin and left_margin are
identical for the compiler. But LeftMargin is different to all the
others, because it starts with a capital letter. This may sound a bit
strange at first, but works well in practice. One advantage is, that
a library author may use snake_case in his library for names, but the
users of the library can freely decide if they prefer camelCase. But
still you may think that all this generates confusion. In practice it
does not, it prevents confusion. Imagine a conventional programming
language, fully case sensitive and not ignoring underscores: In a
larger program we may then have names like nextIteration and 
next_Iteration or keymap and keyMap. What when both names are visible
in current scope, and we type the wrong one. The compiler may not
detect it when types match, but the program may do strange things.
Nim would not allow that similar looking names, as the compiler would
regard them as identical and would complain about a name
redefinition.

You may ask why the first letter is case sensitive. That is to allow
for user defined types to use capital type names and then write
something like var window: Window. So we can declare a variable named
window of a user defined data type named Window. That is a common
practice.

The case insensitivity and the ignoring of underscores may be not the
greatest invention of Nim, but it does not really hurt. The only
exception is when we make bindings to C libraries, where leading or
trailing underscores are used, that can make some renamings
necessary.

The only minor disadvantage of Nim's fuzzy names is when you use
tools like grep or your editor search functionality: You could not be
sure if a search for "KdTree" would give you all results, you would
have to try "Kd_Tree" or "KDTree" and maybe some more variants too.
For that task Nim provides a tool called nimgrep that does a case-
and style-insensitive search. And maybe your editor supports that
type of search also. You can also enforce a consistent naming scheme
when you call the compiler with the command line argument 
--styleCheck:error or --styleCheck:hint.

Languages like C uses curly braces to mark blocks, while other
languages like Pascal uses begin/end keywords for this purpose. At
the same time blocks are generally indented by tabs or spaces to make
it easier for the programmer to recognize the extent of the block.
This is some redundancy which is not always helpful -- block marks and
indentation range can contradict each other and can generate strange
bugs. Like Python or Haskell Nim does not need additional block
markers, the indent level is enough to mark the block extend for the
compiler and the human programmer. This style looks clean and compact
and was used in pseudo-code of textbooks for decades already. Some
people still argue that this style is less "safe", as the behavior of
the code depends on invisible white-space. But this is a strange
argumentation -- the white-space is always visible due to the fact
that there are visible characters on the right. Of course changing
the indention of the last line of a block would effect the behavior
of the code. But that change is well visible. And program code
contains many locations where changing one character breaks it. All
numeric literals would suffer from adding a digit or deleting a
digit. Or the operators like ++ or += from C -- the code may compile
well after deleting the leading +, but it would be wrong. Computer
programming is working carefully! Indeed use of curly braces for
blocks has some advantages, e.g. many editors can highlight such
blocks well, editor may support jumping back and forth between the
braces, and for really large blocks it may be indeed simpler to
discover the whole block range. But practice has shown that marking
blocks with indentation only works fine, most people who have used it
for some time just prefer it.

Blocks, Scopes, Visibility, Locality and Shadowing

Like most other programming languages, Nim has the concept of code
blocks or scopes. The body of procedures, functions, iterators and
templates, as well as the body of various loop constructs or code
following conditional statements builds an indented block and creates
a new scope. In this new scope we can define variables, named
constants, or types with the var, let, const and type keywords which
are local to this block. These symbols are only visible in this
scope, and local variables that need storage are actually created
when the program executes the block, and destroyed when the block is
left. Well, in principle, and at least for ordinary stack allocated
value variables, for references and pointer variables, things are a
bit more complicated, we will discuss that in more detail when we
introduce references. Here we have used the term code block, to
clearly separate them from the const, var, type and import sections
which are a different form of indented blocks. Remember that the
compiler processes our program code from the top to the bottom, so we
have always to define symbols before we can actually use them. When
we define an entity in a code block, and a symbol with that name was
already declared before outside of this block, then that symbol is
shadowed, that is the prior declaration gets temporary invisible.

proc doSomething =
  type NumType = int
  const Seven = 7
  var a: NumType = Seven
  var b: bool = true
  if b:
    echo a, ' ', b # variables of outer scope are visible
    var a, sum: float # now outer a is shadowed
    a = 2.0
    sum = a * a + 1
    echo a, ' ', sum # local data only visible in if block

  echo a # initial int variable with value 7 become visible again

doSomething()

While we have not officially introduced procedures as units to
structure our program code yet, we have put the above code this time
by intend into the body of a proc called doSomething().

This way we can guarantee that the two variables a and b defined in
that proc are really stack allocated. Actually in real life programs
nearly all of the program code is embedded in procs. We will discuss
the peculiarity of global code later. In the proc body from above the
two variables a and b are local to the proc doSomething() -- they are
created on the stack when the procedure is called, that is when we
ask to start it execution by a statement like doSomething(). These
two variables are never visible in code outside of this proc, and the
storage for these two variables is automatically released when
execution of that procedure ends, in this case when the last line of
the proc is reached. In the body of the proc we define although a new
custom type and a named constant -- just to show that it is possible.
Both symbols are also local to this proc and invisible outside.

The indented block following the if b: statement is sometimes called
a "if then" block or just if block -- in that block we define two
other variables called a and sum of float type, which are also stack
allocated. If these two variables are already allocated when the proc
starts its execution, or only when the then block following the if
statements is executed, is actually an implementation detail. As the
variable a of float type in the if then block has the same name as
the outer variable of int type, that integer variable is shadowed in
the if block -- the outer value gets temporary invisible as soon as
the new symbol is declared. Other symbols of outer scopes remain
visible. In the if then block as well as in most other indented code
blocks we could also define named constants or custom types, these
would be visible only in this block. Indented code blocks can be
nested -- in one block we can have more indented block, for which all
declared symbols are again local and invisible outside. The last echo
() statement in our code example from above is already below the if
then block, so the initial variable a of integer type becomes again
visible.

Global code

In the introducing sections of the book we have generally used
program code at a global level, not embedded in a proc body. We did
that for simplicity and as we had not already introduced procs.
Global code is sometimes used in small scripts or for special
purposes, like program initialization. But for larger programs most
of the code is generally grouped in procedures. For variables defined
in global code it is not that well defined where they are stored, it
may depend on the actual Nim compiler implementation and the compiler
backend. The performance of global code can be worse than code
enclosed in proc bodies, so when performance maters we should put our
code in procs. One reason for the not optimal performance of global
code is, that global variables are not located on the stack,but in
the global BSS segment of the program, and that the backend can not
optimize global code that well, e.g. global variables may not be
cached in CPU registers. Note that variables that have to exists and
keep it value for the whole runtime of the program, and not only for
the duration of the execution of a single proc, has to be defined as
globals. The same holds obviously for global variables that are used
from code of different procs, like the stdout and stdin variables of
the system module. An alternative to the use of global variables when
a variable used in a proc should keep its value between different 
proc calls is to attach the {.global.} pragma to a proc local
variable. This way that variable is still only visible in that proc
where the variable is declared, but the variable is stored in the BSS
segment instead on the stack and so its value is preserved between 
proc calls.

Note that structured named constants, e.g. constant strings, are
stored also in the BSS segment, even when they are only defined local
to a proc. So large structured constants can increase the executable
size, as the BSS segment is a part of the program executable.

White space, punctuation and operators

The space character with decimal ASCII value 32 is used in Nim
program code to indent code blocks and to separate different symbols
from each other. Nim keywords are always separated with leading and
trailing white-space from other symbols, while other symbols are most
often separated by punctuation and an additional optional space
character. Whenever the syntax allows a space, we can also insert
multiple spaces or a comment enclosed in #[ ]# in the source code.
Tabulator characters are not allowed in the Nim source code, but we
can use them in comments and of course in string literals. We
mentioned already, that spaces can make a difference how operators or
function parameters are handled. In expressions like a+b or a + b the
+ operator is regarded as an infix operator, but in a + -b the minus
sign is regarded as unary operator bound to b. This way asymmetric
expressions like a +b or a <b would be invalid, as the operators are
interpreted as unary ones attached to b, and then there is no infix
operator between the two variables. A proc call like echo(1, 2) is
interpreted as a call of echo() with two integer literal arguments,
while a call like echo (1, 2) with a space after the proc name is
interpreted in command invocation syntax as a call with a tuple
argument. While in C code it is not uncommon to always insert a space
between the function name and it parameter list, we should avoid that
in Nim for the described reason. We will learn more about proc calls
and the tuple data type later.

Operators

Nim uses the following punctuation characters as operators:

=, +, -, *, /, <, >, @, $, ~, &, %, |, !, ?, ^, ., :, \

These symbols can be used as single entities or in combination, and
we can define our own operators or redefine existing operators. All
these symbols can be used as infix operators between two arguments,
or as unary prefix operators, but Nim does not support unary postfix
operators, so a notation like i++ as known from the C language is not
possible in Nim. There exists a few combinations of these punctuation
characters that have a special meaning, we will learn more about that
and how we can define our own operators later in the book.

In Nim these keywords are also used as operators:

and, or, not, xor, shl, shr, div, mod, in, notin, is, isnot, of, as, from.

Operators have different priorities, e.g. * and / have a higher
priority than + and -. In most cases the priority is just as we would
expect, maybe with a few exceptions. If we are unsure, we can group
terms with brackets, or consult the compiler manual for details.

Since version 1.6 Nim also allows to define and use a few unicode
operators, but these are still considered experimental. For details
you should consult the compiler manual.

Order of Execution

Global program code or code in called procs is generally executed
from top to the bottom and from left to the right, as long as control
structures do not enforce a different order. To demonstrate this, we
use here a set of four different procs, which contain an echo()
statement each, and return a numeric expression. Well, we have not
yet formally introduced procedures, so if you should feel too
uncomfortable with the code below, just skip this section for now and
come back when you have read the section about procs:

proc a(i: int): int =
  echo "a"
  i * 2

proc b(i: int): int =
  echo "b"
  i * i

proc c(i: int): int =
  echo "c"
  i * i * i

proc d(i: int): int =
  echo "d"
  i + 1

echo a(1); echo b(1)
echo b(2) + d(c(3)) # (2 * 2) + ((3*3*3) + 1)
echo "--"
echo a(1) < 0 and b(1) > 0
echo a(1) > 0 or b(1) > 0

It should be no real surprise, that the first three echo() statements
produce this output:

a
2
b
1
b
c
d
32

For the term d(c(3)) it is obvious that the inner expression c(3) has
to be evaluated first, before that result can be used to call proc d
().

The last two lines demonstrate the so called short-cut-evaluation for
expressions with the boolean and or or operators. As the expression a
() and b() is always false when a() is false, for this case b() has
not to be evaluated at all. In a similar way, as the expression a()
or b() is always true when a() is true, for that case b() has not to
be evaluated at all. So in the last two lines of above code b() is
never called at all, and the output is just

a
false
a
true

Note that in Nim as in most other programming languages the
assignment operator = behaves different compared to ordinary
operators like + or *, as in assignments like let a = b + c()
obviously the right side has to be evaluated before the result can be
actually assigned to variable a.

Control Structures

Larger computer programs generally consists not only of code that is
executed in a linear fashion, but contain code for conditional or
repeated execution.

The most important control structures of Nim are the if statement for
conditional execution, the related case statement and the while and 
for loops for repetitions. All these statements controls the actual
program execution at program runtime. Syntactically very similar to
the if statement is Nim's when statement, which is already evaluated
at compile time, and can be used to adapt our program code for
various operating system or to compile our code with special options,
e.g. for debugging or testing purposes.

All these control structures can be nested in arbitrary ways, so we
can have in one if branch other if conditions or while loops, and in 
while loops again other control structures including other loops.

If Statement and If Expression

The if statement with multiple optional elif branches and an optional
else branch evaluates a sequence of boolean conditions at program
runtime. As soon as one condition evaluates as true the corresponding
statement block is executed, and after that the program execution
continues after the whole if construct. That is at most one branch is
executed. If none of the conditions after the if or elif keywords
evaluates to true, then the else branch is executed if it exists. A
complete if statement consists of one if condition, an arbitrary
number of elif conditions and one optional else part:

if condition1:
  statement1a
  statement1b
  ...
elif condition2:
  statement2a
  statement2b
  ...
elif condition3:
  statement3a
  statement3b
  ...
elif ...:
  ...
else:
  statementa
  statementb
  ...

The most simple form of an if statement is

if condition:
  statement

if age > 17:
  echo "you may drink and smoke, but better avoid it!"

Note that the branches are indented by spaces, we use two spaces
generally, but other numbers work as well. And note that it is elif,
not elsif like in Ruby, and that there is a colon after the
condition. Instead of a single statement we can use multiple in each
branch, all on its own line and all indented in the same way.

No, the terminating colon is not really necessary for the compiler,
the compiler could determine the end of the condition without it, as
the following statement is indented. But it looks better with colon,
the colon makes it for humans easier to understand the structure of
the whole if statement. So the compiler expects the colons and
complains otherwise currently.

When there is no elif and no else part, then we can also write the
conditional code direct after the colon, like

if age > 17: echo "you may drink and smoke, but better avoid it!"

With an elif and an else branch the example from above may look like

var age: int = 7
if age == 1:
  echo "you are really too young to drive"
elif age < 6:
  echo "you may drive a kids car"
elif age > 17 and age < 75:
  echo "you can drive a car"
else:
  echo "drive carefully"

Note that we perform the age tests in ascending order -- it would not
make much sense to first test for a condition age < 6, and later to
test for age < 4, as the if statement is evaluated from top to
bottom, and as soon as one condition is evaluated as true, that
branch is executed and then the program execution continues after the
whole if construct. So a later test age < 4 would be useless, when
that condition is already covered by a prior test age < 6.

As the various conditions of the if statement are processed from top
to bottom until one condition evaluates to true, it can be a good
idea to put the most likely conditions first for optional
performance, as then the unlikely conditions have not to be evaluated
in most cases. Another strategy for larger if/elif constructs is to
put the most simple and fast tests to the top when possible.

We can also have if/else expressions which returns a value like in

var speed: float = if time > 0: delta / time else: 0.0 # prevent div by zero error

In C for a similar construct the ternary ? operator is used.

In languages like C or Ruby the assignment operator "=" is an
expression which returns the assigned value, so in C we can write
code like

while (char c = getChar()) { process(c)}

In Nim the assignment operator is not an expression with a result,
but we can group multiple statements in round brackets separated by
semicolon, and when the last statement in the bracket is an
expression, than the whole bracket has the same value. So we can use
conditional terms like

while (let c = getChar(); c != '\0'):
  process(c)

If we declare a variable in this way using the var or let keyword
then that variable is only visible in the bracket expression itself
and in the following indented block.

Note that if-expressions must always return a well defined value, so
they must always contain an else branch. A plain if, without an else,
or an if/elif without an else does not work. And as Nim is a
statically typed language and all variables have a strictly well
defined type, the if-expression must return the same type for all
branches!

var a: int
var b: bool
a = if b: 1 elif a > 0: 7 else: 0 # OK
a = if b: 1 elif a > 0: 7 # invalid
a = if b: 1 # invalid
a = if b: 1 else: 0.0 # invalid, different types!

The When Statement

The when statement is syntactically very similar to the if statement,
but while all the boolean conditions are evaluated during the program
run time for the if statement, for the when construct all the when/
elif/else conditions have to be constant expressions and are already
evaluated at compile time. In ordinary program code the when
statement is not used that often, but it is useful when we write
bindings to C libraries and low level code. Common use cases for the 
when statement are the isMainModule condition test and the test for
defined symbols like defined(windows):

when not defined(gcDestructors):
  echo "You may try to compile your code with option --gc:arc"
when isMainModule:
  doAllTheTests()

The value isMainModule is only true for a source code file, when that
file is compiled directly as main module, that is when it is not
indirectly compiled because it is imported by other modules. This way
we can include easily test code to our library modules -- that test
code is ignored when the module is used as library, but active when
we compile the module direct for testing.

A when defined() construct can be used to test for predefined or our
own custom options, e.g. we may give the optional option 
-d:gintroDebug to the compiler and test in the code of that module
for this option, like when defined(gintroDebug):.

One difference of the when to the if statement is, that the "then"
branches do not open a new scope, so variables which we define there
are still visible after the construct has been processed:

when sizeof(int) == 2:
  var intSize = 2
  echo "running on a 16 bit system!"
elif sizeof(int) == 4:
  var intSize = 4
  echo "running on a 32 bit system!"
elif sizeof(int) == 8:
  var intSize = 8
  echo "running on a 64 bit system!"
else:
  echo "cannot happen!"

echo intSize # variable is visible here!

Another peculiarity of the when statement is, that it can be used
inside object definitions -- we will show an example for that in a
later section of the book when we introduce the object data type. In
the same way as the if construct, when can also be used as an
expression.

The Case Statement

The case statement is not used that often, but it can be useful when
we have many similar conditions:

case inputChar
of 'x': deleteWord()
of 'v': pastWord()
of 'q', 'e': quitProgram()
else: echo "unknown keycode"

To enable optimizations the case construct has some restrictions
compared to a more flexible if/elif statement:

The variable after the case keyword must have a so called ordinal
type like int, char or string, while float would not work. And the
values after each of keyword must be constant, that is a single
constant value, multiple constant values or a constant range like 'a'
.. 'd' for the 4 first lower case letters. Of course these constants
must have a type compatible to the type of the variable after the
case keyword. A case statement must cover all possible cases, so most
of the time an else branch is necessary.

For Nim version 1.6 the case statement can contain also optional elif
branches with arbitrary boolean conditions. This was not the case in
the wirthian languages Pascal, Modula and Oberon, and makes Nim's
case construct now very similar to the ordinary if/elif/else.

Unless the similar switch statement in C the case statement needs no
break after each branch. If a condition after a of keyword is true,
then the corresponding statement or statement sequence is executed,
and after that program execution continues after the whole case
construct.

The case construct can also be used as an expression like in

var j: int
var i: int =
  case j
    of 0 .. 3: 1
    of 4, 5: 2
    of 9: 7
    else: 0

Here an else is necessary to cover all cases. And as you see we can
also indent the block after the case keyword if we want.

The While Loop

The while loop is used when we want to do conditional repetitions,
that is, if we want to check a condition and want to execute a block
of statements only as long, as the condition is true. If the
condition is false in advance, or becomes false after some
repetitions, then the program execution proceeds after the indented
loop body block.

A basic while loop has this shape:

while condition:
  statement1
  statementN
firstStatementAfterTheWhileLoop

var repetitions = 3
while repetitions > 0:
  echo "Nim is easy!"
  repetitions = repetitions - 1

That loop would print the message three times. Like the condition in
the if clause the condition is terminated with a colon. Note that the
condition must change during execution of the loop, otherwise, when
the condition is true for the first iteration, it would remain true
and the loop would never terminate. We decrease the loop counter 
repetitions in the loop, so at some point the condition will become 
false and the loop will terminate and program execution will continue
with the first statement after the loop body. Note how we decrement
the loop counter: The right site of the assignment operator is
evaluated, after that is done, the new value is assigned to the
counter.

There exists two rarely used variants of a while loop: the loop body
can contain a break or a continue statement, which each consists only
of this single keyword. A break in the body stops execution of the
loop immediately and continues execution after the loop body. And a 
continue statement in the body skips the following statements in the
body and starts at the top again, the while condition is evaluated
again.

var input = ""
while input != "quit":
  input = readLine(stdin)
  if input == "":
    continue
  if input == "exit":
    break

Above code used the == and the != operators. The == operator does a
test for equality, and != test for inequality. Both operator work for
most data types like integer, floats, characters and strings. The
literal value of an empty string is written "". In line 2 we test if
the variable named input has not the value "quit", and in line 4 we
test of that variable is empty, that it contains no text at all.

Using of break and continue destroys the expected flow in loops, it
can make understanding loops harder. So we generally avoid their use,
but sometimes break or continue are really helpful. For example when
an unexpected error occurs, maybe by invalid user input.

There in no repeat loop as in Pascal in Nim, which does the first
check at the end of the loop when it was executed already for the
first time. Repeat loops are not used that much in Pascal, and they
are some sort of dangerous, because they check the condition after
the first execution of the body, so maybe the body is executed with
invalid data for the first iteration. Later, we will see how we can
use Nim macros to extend Nim by a repeat loop that can be used as it
would be part of Nim core functionality.

The Block Statement

The block statement can be used to create a new indented block, which
creates a new scope, in the same way as a when true: statement would
do:

block: # create a new scope
  var i = 7
echo i # would not compile, as variable i is undefined

Blocks can be useful to structure large code segments, when there are
no better ways, as splitting the code in multiple procs, available.
For testing purposes blocks can be useful too, to keep symbol in a
local scope. But actually most useful are blocks, when the blocks get
attached names, and we use the break statement in a while or for loop
to break out of a nested loop:

let names = ["Nim", "Julia", "?", "Rust"]
block check:
  for n in names:
    for c in n:
      if c notin {'a' .. 'z', 'A' .. 'Z' }:
        echo "invalid character in name"
        break check
echo "we continue"

The break check statement would immediately leave the nested loops
and continue with the first statement after the block, which is the
last line in the code segment above. Using break in such a way is not
very nice, as it may make it harder to understand the code structure,
but sometimes it can be very useful.

For Loops and Iterators

These are very useful and important in Nim and other programming
languages. For loops are most often used to iterate over containers
or collections. We have not discussed the important array and seq
containers yet, but we know already the string container.

A string contains characters, the characters are numbered starting
with 0, and we can access single characters of a string with the
subscript operator [], which gets the position of the desired
character as argument. So we could print the single characters of a 
string, in this way:

var
  s = "Nim is not always that easy?"
pos = 0
while s[pos] != '?':
  echo "-->", s[pos]
  inc(pos)

It is obvious that the pos variable is some sort of annoying
here -- we want to process all the characters in the string in
sequence, so why would we have to use a position variable to do that.
And this way is susceptible to errors, maybe we forget increasing the
pos variable in the loop body. So most modern languages provide us
with iterators for this purpose:

var
  s = "Nim is not always that easy?"
for ch in items(s):
  echo "-->", ch

That is obvious shorter. The for construct may appear a bit strange,
and it is indeed, but it is a common way to write iterators, it is
used in Python too. Ruby uses something like s.each{|ch| ... } instead.

For loops in Nim iterates over containers or collections, and pics
each element in sequence in this process. The variable after the for
keyword is used to access or to reference the single elements. That
variable has automatically the right type, which is the type of the
elements in the container, and get in each iteration the value of the
next element in the container, starting by the first element in the
container and stopping when there is no element left. Items() is here
the actual iterator, which allows us to access the individual
characters in sequence. It exists the convention in Nim that an items
() iterator is automatically called in a for loop construct when no
iterator name is explicitly given, so we could also write shorter for
ch in s: in this use case.

You may recognize that the output of the above for loop is not
identical to the output of the previous while loop. The while loop
stops when the last character, that is '?', is reached, while the for
loop processes this last character still. That is intended for the 
for loop, its general purpose is to process all the elements in
containers or collections.

The above for loop does a read access to the string, that is, we get
basically a copy of each character, and we can not modify the actual 
string in this way. When we want to modify the string, there is a
variant available.

var
  s = "Nim is not always that easy?"
for ch in mitems(s):
  if ch == '?':
    ch = '!'

Here we use mitems() instead of the plain items(), the leading "m"
stands for mutable. In the loop body we can assign different values
to the actual content.

Objects

We have worked with basic data types like numbers, characters and
strings already. Often it makes sense to join some variables of these
basic data types to more complex entities. Assume you want to build
an online store to sell computers, and you want to built a database
for them. The database should contain the most important data of each
device type, like type of CPU, RAM and SSD size, power consumption,
manufacturer, quantity available, and actual selling price.

We can create a custom object data type with fields containing the
desired data for this purpose:

type
  Computer = object
    manufacturer: string
    cpu: string
    powerConsumption: float
    ram: int # GB
    ssd: int # GB
    quantity: int
    price: float

We have to use the type keyword to tell the compiler that we want to
define a new custom type. Writing the type keyword on its own line
begins a type section where we can declare one or more custom data
types. All type declarations in a type section must be indented. In
the next line we write our type name, an equal sign and the keyword 
object. That indicates that we want to declare a new object type
named Computer. Here Computer is a type name, in Nim we use the
convention that user defined type names start with a capital letter.
In the following indented block we specify the desired fields, each
line contains the name of a field, and a colon followed by the needed
data type. That is similar like a plain variable declaration.

Objects in Nim are similar to structs in C. Unlike classes in Java
Nim objects contain only the fields, sometimes also called member
variables, but no procedures, functions or methods, and no
initializers or destructors as in C++. In Nim we keep the data
objects, and the procedures, functions, methods and also optional
initializers and destructors that work with that data objects
separated.

Now that we have defined our own new object type, we can declare
variables of that type and store content in its fields.

var
  computer: Computer

computer.manufacturer = "bananas"
computer.cpu = "x7"
computer.powerConsumption = 17
computer.ram = 32
computer.ssd = 1024
computer.quantity = 3
computer.price = 499.99

Of course in real applications we would fill the fields not in this
way, but we would maybe read the data from a file, from terminal or
maybe from a graphical user interface.

It may look a bit ugly that we have to write computer. before each
field when we access the fields. Indeed in recent Nim versions that
is not necessary, you may use the with construct now instead.

import std/with
var
  computer: Computer
with computer:
        manufacturer = "bananas"
        cpu = "x7"
        powerConsumption = 17
        ram = 32
        ssd = 1024
        quantity = 3
        price = 499.99

We can use the fields like ordinary variables:

computer.quantity = computer.quantity - 1 # we sold one piece
echo computer.quantity

As you already know, the right side of the assignment operator is
evaluated first, then the result is stored in the variable on the
left side. But we can also just write computer.quantity -= 1 or dec
(computer.quantity).

Generally a computer store would offer many different types of
computers, so it would make sense to store all the different devices
in a container like a sequence, called short seq in Nim.

Arrays and Sequences

Sequences and arrays are homogeneous containers, they can contain
multiple other elements of the same data type, while a plain variable
like a float or an int only contains a single value. In some way we
could regard objects also as containers, because objects contain
multiple fields. The same holds for tuples -- tuples are a very
simple, restricted form of objects and also contain fields. But more
typical container data types are the built in arrays and sequences,
or for example hash tables which are provided by the Nim standard
library. Arrays, sequences and hash tables can contain multiple
elements, but all elements must have the same data type, which we
call the base type.^[22] The data type of the base type is not
restricted, it can be even again array or sequence types, so we can
built multidimensional matrices in this way. Arrays have a fixed,
predefined size, they can not grow or shrink during runtime of our
program. Sequences and hash tables can grow and shrink.

Arrays and sequences appear very similar, a sequence appears even
more powerful because it can change its size, that is the number of
elements that it contains, at runtime, while an array has a fixed
size. So why do we have arrays at all? The reason is mostly
efficiency and performance. An array is a plain block of memory in
the RAM of the computer, which can be accessed very fast and needs
not much care by the runtime system. Sequences take much more effort,
especially when we add elements and the sequence has to grow. When we
create sequences, we can specify how many elements should fit in it
at least and the runtime system reserves a block of RAM of the
appropriate size. But when our estimation was too small, and we want
to append or insert even more elements, then the runtime system may
have to allocate a larger block of memory first, copy the already
existing elements at the new location, and then release the old, now
unnecessary memory block. And this is an relative slow operation. The
reason why this process can be necessary is, that the initially
allocated memory block may not increase in size because the
neighborhood in the RAM is already occupied by other data. Now let us
see what we can do with arrays and sequences:

var
  a: array[8, int]
  v = 1
for el in mitems(a):
  el = v
  inc(v)
for el in mitems(a)
  el = el * el
for square in a:
  echo square

In the second line of the code above we declare a variable named a of
array type -- we want to use an array with exactly 8 elements, and
each element should have the data type int. To declare a variable of 
array data type we use the array keyword followed in square brackets
by the number of the elements, and separated by a comma, the data
type of the elements. We can also specify the range of the indices
explicitly by specifying a range like array[0 .. 7, int] or array[-4
.. 3, int]. The first specification is identical to the one in above
example program, and the second one would allow us to access array
elements with index positions from -4 up to 3.^[23] or [int, 8]. It
may help to remember that for plain variables the data type comes
last also like in var i: int.]

The first for loop of above program fills our array -- that is for
each of the 8 storage places in the array we fill in some well
defined data. We use the mitems() iterator here, because we want to
modify the content of our array -- we fill in numbers 1 .. 8. In the
next for loop we square each storage location, and finally we print
the content. In the last for loop we do not modify the content, so a
plain items() instead of mitems() would work, but we already learned
that we have not to write the plain items() at all in this case.

Sequences work very similar like arrays, but they can grow:

var
  s: seq[int]
  v = 0
while v < 8:
  inc(v)
  add(s, v)
for el in mitems(s)
  el = el * el
for square in s:
  echo square

We start with an empty seq here, and use the add() procedure to
append elements. After that we can iterate over the seq as we did for
the array.

In the same way as we access single characters of a string with the
subscript operator [], we can use that operator to access single
elements of an array or a seq like in a[myPos]. The slice operator is
available for arrays and sequences too and can be used to extract
sub-ranges or to replace multiple elements. As arrays have a fixed
length, the slice operator can only replace elements in arrays, but
not remove or insert ranges. The first element position is generally 
0 for arrays and sequences. Arrays can even be defined in a way that
the index position starts with an arbitrary value, but that is not
used that often. Whenever you use the subscript or slice operator you
have to ensure that you access only valid positions, that is
positions that really exists. a[8] or s[8] would be invalid in our
above example -- the array has only places numbered 0 .. 7, and for
the seq we have added 8 values which now occupy positions 0 .. 7
also, position 8 in the seq is still undefined. We would get a
runtime error if we would try to access position 8 or above, as well
when we would try to access negative positions. You might think that
an assignment for a seq like s[s.length] = 9 is the same as s.add(9),
but only the add() operation works in this case.

Note that in some languages like Julia arrays start at position 1.^[
24] Nim arrays can have an arbitrary integral start position,
including negative start positions, but start position as well as
highest subscript position are determined in the program source code
and can not change at runtime. We say that arrays have fixed
compile-time bounds. Sequences starts always at position 0, we can
specify an initial size, and we can always add more elements at
runtime.

Arrays and sequences allow fast access to its elements: All the
elements are stored in a continues memory block in RAM, and the start
location of that memory block is well known. As all the elements have
the same byte size, it is an easy operation to find the memory
location of each element. The compiler uses the start location of the
array or seq, and adds the product of subscript index and element
byte size. The result is the memory location of the desired element,
which was selected by the index used in the subscript operator. When
the array should not start at position 0, then the compiler would
have to adjust the index, by subtraction of the well known start
index. This operation takes not much time, but still arrays starting
at position 0 may be a bit faster. We said that the compiler has to
do a multiplication of index position and element size -- that is an
integer multiplication, which is very fast. When the element size is
a power of two, then the compiler can even optimize the
multiplication by using a simple shift operation, which may be even
faster, depending on your CPU.

It should be not surprising that the internal structure of sequences
are a bit more involved than arrays. Arrays are indeed nothing more
than a block of memory, generally allocated on the stack for local
data or allocated in the BSS segment for global data. Don't worry
when you have not yet an idea what the stack, the heap and a BSS
segment is, we will learn that soon. The Nim seq data type has a
variable size, so it is clear that it needs not only a storage
location for its elements, but also a counter to store how many
elements it currently contains, and another counter how many it could
contain at most. The element counter must be updated when we add or
delete elements, and when the counters tells that there is currently
no more space available for more elements, then a new block of memory
must be allocated, and the existing elements must be copied from the
old location into the newly allocated memory region, before the old
memory region can be released.^[25] Due to this additional effort
appending elements to a seq by using the add() procedure is not
extremely fast. You may wonder why we have not to save a size
information for arrays. Well arrays have fixed size, so it is obvious
that we never have to adjust something like a size counter, simple
because size would never change. But would we have to save the
desired initial size of the array? Well, in some way yes. But it is a
constant value. During the compile process the compiler can catch
some errors already for us -- when we have an array as above with size
8, then the compiler would be able already at compile time to
recognize some invalid access to array elements -- a[9] would be a
compile time error for sure. But at runtime, when we execute our
program, access to not existing index position may occur, for example
by constructs like var i = 9; a[i] = 1 when the array is declared as 
var a: array[8, int]. For catching that type of error the compiler
has to store the fixed array size somewhere and to check against that
value when an array access by using the subscript operator with a non
constant argument occurs, as the a[i] above. One related remark:
Accessing array elements is as fast as ordinary variable access when
we use a constant value as index, that is a constant literal or a
named constant. The reason for this is, that when the index is a
constant, then the compiler just knows the exact position of that
array element in memory, just as it knows the address of plain
variables, so there is no need for address calculations at runtime.
Actually, to access an array element with a given constant index
position, the compiler only has to add a constant value to the
current stack pointer, as arrays are stored on the stack. To access a
constant position in a seq, the compiler would have to add a constant
to the base address of the memory block that contains the seq data.

We said that appending elements to sequences is not extremely
fast -- indeed it is a few times slower than access to an array
element by its index using the subscript operator. So when we know
that our seq will have to contain at least an initial amount of
elements, then it can be useful for maximum performance, that we
allocate the seq from the beginning for this size and than fill in
the content by use of the subscript operator instead that we append
all the elements one by one. Here is one example:

var s: seq[int] = newSeq[int](8)
var i: int
while i < 8:
  s[i] = i * i
  inc(i)

We use the newSeq() procedure to initialize the sequence for us, the
content of the square brackets tells the newSeq() procedure that we
want a sequence with base type int, and the number 8 as argument
tells it that the newly created sequence should have 8 elements with
default value (0). This procedure is a so called generic procedure,
it needs additional information, which is the data type that the
elements should have. Don't confuse the square bracket in the newSeq
[int]() call with the subscript operator a[i] which we have used for 
array access, both are completely unrelated. Note that the
initialization of the seq above does not restrict its use in any way,
we can still use it like an uninitialized seq, that is we can use the
add() operator to add more elements, we can insert or delete elements
and all that.

Deleting elements from an array or from a sequence can be very slow.^
[26] It is slow when we use the naive approach and move all the
elements located after the element that should be removed one
position forward. This would obtain the order in the container, so
sometimes this is the only solution, but of course moving all the
entries is expensive for large containers. Nim's standard library
provides the delete() function for this order maintaining delete
operation. A much faster way to delete an entry in a seq or array is
to remove the last entry and replace the one that should be deleted
with that last entry. This operation moves the last entry to the
front, so order of elements in not maintained. Nim's standard library
provides the del() function for this faster, but order changing
delete operation. Of course, whenever the order is not important, we
should use del(). The delete() and del() functions are actually only
available for sequences, as arrays have a fixed size -- but in
principle we could do similar operations with arrays as well, we have
just to store the actual used size somewhere. ^[27]

In the section about strings we said that strings have value
semantic, that is that an assignment like str1 = str2 creates a copy
of str2 and that after that assignment str1 and str2 are fully
independent entities -- modifying one does not change the content of
the other one. Array and sequences behave in the same way, both have
value semantic too. Indeed arrays are true value types in Nim, as
they live on the stack in the same way as plain variables like
integers, floats or characters. Sequences have a dynamic data buffer
which is allocated on the heap, so it would be possible that an
assignment like seq1 = seq2 would not copy the data buffer but reuse
the old one. In that case both sequences would be not independent, 
seq2 would be an alias for seq1. This is called reference semantic,
some languages like Ruby behave in this way. But in Nim arrays,
strings and sequences have value semantic, an assignment creates an
independent copy. We will learn more details about reference semantic
and the use of the stack or heap to store data soon when we discuss
references to objects.

Some details

Let us investigate at the end of this section some internal details
about arrays and sequences. Beginners not yet familiar with the
concept of pointers should better skip this subsection, and maybe
come back later. We could consult the Nim language manual or the
compiler source code to learn more details about arrays and
sequences. Or we can write some code to test properties and behavior.
Let us start investigating an array:

proc main =
  var a: array[4, uint64]
  echo sizeof(a)
  a[0] = 7
  echo a[0]
  echo cast[int](addr a)
  echo cast[int](addr a[0])

  var a2 = a
  a[0] = 3
  echo a2[0]

main()

When we run this program we get this output:

32
7
140734216410384
140734216410384
7

The size of the whole array is 32, as we have 4 elements each 8 byte
in size. And the address of the array itself as well as the address
of its first element is identical. Remember that the actual address
values will be different for each run of our program, and they may be
totally different on different computers, as it is some random choice
of the OS which free memory area is used to run our program. This
result is expected as the array is a plain block of memory stored on
the stack. And indeed the array has copy semantic, when we create a
copy called a2 and later modify a, then the content of a2 is
unchanged. That was not really surprising, so let us investigate a
sequence:

proc main =
  var dummy: int
  var s: seq[int64]
  echo sizeof(seq)
  echo sizeof(s)
  s.add(7)
  echo s[0]
  echo cast[int](addr dummy)
  echo cast[int](addr s)
  echo cast[int](addr s[0])

  var s2 = s
  s[0] = 3
  echo s2[0]

main()

When we run above code we get:

8
8
7
140732171249104
140732171249112
140463681433696
7

The first two lines of the output may confuse us, as a size of only 8
byte may indicate a plain pointer value on a 64 bit system. Indeed
the sequence is not a large object that contains size and capacity
fields, but only a tiny object that contains a single pointer to the
data storage of that sequence. We know that it is not a plain pointer
or ref by the fact that we can not assign nil or test for nil for
sequences. (But an object which contains only a pointer is basically
identically to a plain pointer, as Nim objects have no overhead as
long as we do not use inheritance and when no padding to word size is
needed for tiny fields like int8.) Capacity and length are stored
also in the memory block that is allocated for the elements as long
as the sequence is not empty. So empty sequences don't wast much
memory when we have a lot of them, i.e. arrays or sequences of
sequences (matrices). We use the dummy int variable in the code above
as we know that plain ints are stored on the stack, and when we
compare the addresses of our dummy variable and our sequence, then we
see that the addresses indicate close neighborhood, so the seq object
is also stored on the stack. But the address of s[0] is very
different, indicating that the data buffer is stored in a different
memory region, which is the heap. If we would continuously add
elements to the seq, then the address s[0] would change at some
point, while the address of s would remain always unchanged. That is
because the capacity of the data buffer would become exhausted at
some point and a new data buffer with a different address would be
used. Finally we see again that the sequence has also copy semantic,
as the content of the copy s2 remains unchanged when we modify the
initial sequence s. We could try to discover some more details of the
internals of Nim's sequences, i.e. we could try to detect where the
capacity and size is stored. But that are internal details which
should not really interest us and which may change with new compiler
version or different compilers.

But OK, you may still not believe what we said, so let us go one
layer deeper. We strongly assume that a seq needs a length and a
capacity field. And we assume that its data type should be int. We
said that both fields should be adjacent to the buffer of the seq
elements, that means at the start or at the end. Obviously we can not
access the end as long as we do not know the capacity, so capacity
field should be at the start, and then length field also. We may find
out which one is which by observing the content when seq grows. So
let us write some code:

proc main =
  var
    s: seq[int64] = newSeqOfCap[int64](4)
    s2: seq[int64]
    p: ptr int

  var h = cast[ptr int](addr s2) # prove that an uninitialized seq is indeed a pointer with nil (0) value
  echo cast[int](h) # address on stack
  echo h[] # value (0)
  echo ""

  for i in 0 .. 8:
    s.add(i)
    echo cast[int](addr s[0])
    p = cast[ptr int](cast[int](addr s[0]) - 8) # capacity
    echo p[]
    p = cast[ptr int](cast[int](addr s[0]) - 16) # length
    echo p[]

main()

Output when we run the program is:

140725732630192
0

140251431497824
4
1
140251431497824
4
2
140251431497824
4
3
140251431497824
4
4
140251431506016
8
5
140251431506016
8
6
140251431506016
8
7
140251431506016
8
8
140251431510112
16
9

Don't worry when you do not understand the program and the output
yet. You will better understand it when you have read the sections
about references, pointers and memory management. The first two
output lines shows us that an uninitialized seq is just a pointer
pointing to nil. And the remaining output lines show us the address
of the first seq element, the capacity and the length of the seq
whenever we add an element. We started with a seq with initial
capacity of 4, so address and capacity is constant while we add the
first 4 elements. Then the capacity of the allocated buffer is
exhausted. A new buffer with different address and doubled capacity
is allocated, the already contained elements are silently copied to
the start of the new buffer and so on.

Slices

Nim slices are objects of type Slice that contain a lower a and an
upper bound b. The system module defines also the HSlice object
called heterogeneous slice for which the lover and and upper bound
can have different data types:

type
  HSlice*[T, U] = object   ## "Heterogeneous" slice type.
    a*: T                  ## The lower bound (inclusive).
    b*: U                  ## The upper bound (inclusive).
  Slice*[T] = HSlice[T, T] ## An alias for `HSlice[T, T]`.

As the Slice and Slice objects are not a builtin type, their Names
start with capital letters. Slices are not used that often directly,
but mostly indirectly with the range operator, e.g. to access
sub-ranges of strings and other containers.

One example for its direct use from the system module is

proc contains*[U, V, W](s: HSlice[U, V], value: W): bool {.noSideEffect, inline.} =
  result = s.a <= value and value <= s.b

Slices are used by functions of the standard library or by user
defined functions to access sub-ranges of strings, arrays and
sequences. Applied on these container data types slices look
syntactically like sub-ranges:

var
  m = "Nim programming is difficult."
m[19 .. 28] = "not easy."
echo m
echo "Indeed " & m[0 .. 18] & "is much fun!"

In line three we use the slice to replace the sub-string "is
difficult." which starts at position 19 with another string. Note
that the replacement can be a longer or a shorter string, that is the
slice supports not only overwriting characters, but also inserting or
deleting operations. In the last line we use the slice to access a
sub-string and create a new string with it. As we learned earlier in
the strings section already, we can use the ^ operator to access
elements counted from the end of the container, so we could have
written line three also as m[19 .. ^1] = "not easy.".

Slices can be used in a similar way for arrays, strings and
sequences. But we have to remember that slices are only objects with
a lower and an upper bound, so there must be always a procedure that
accepts the container and the slice as arguments to do the real work.

When we care for utmost performance, then we have to be a bit
carefully with slices, as slices can generate copies. Consider this
example:

type
  O = object
    i: int

proc main =
  var
    s = newSeq[O](1000000)
  for i in 0 .. (1000000 - 1):
    s[i] = O(i: i)

  var sum = 0
  for x in s[1 .. ^1]:
    sum += x.i

Here we use the slice operator to exclude the first element from our
summing operation. Unfortunately in current Nim v1.6 use of the slice
operation in this way creates a copy of our sequence, which increases
the run-time and memory consumption. We may try to use the new
toOpenArray() expression and try a construct like

  for x in items(s.toOpenArray(1, s.high)):

but that does currently not compile.

One option is currently that we create a custom iterator like

iterator span*[T](a: openArray[T]; j, k: Natural): T {.inline.} =
  assert k < a.len
  var i: int = j
  while i <= k:
    yield a[i]
    inc(i)

and use

for x in s.span(1, s.high):

Or we may do the summing in a procedure and pass that proc an
openArray created with toOpenArray() like

proc sum(x: openArray[O]): int =
  for el in x:
    inc(result, el.i)

echo sum(s.toOpenArray(1, s.high))

But this is work in progress, so the situation may improve, see

+ https://forum.nim-lang.org/t/4823

+ https://forum.nim-lang.org/t/4582#28715

Value Objects and References

We have already used different types of variables -- integers, floats,
or the custom Computer object, and some more. We said that variables
are named memory regions, where the content of our variables is
stored. We call this type of variables also value objects.

Value objects always implies copies when we do an assignment

var i, j: int
i = 7
j = i
i = 3
echo i, j

Here we have 3 assignments, first we assign the integer literal 7 to
variable i, then we assign the content of variable i to variable j,
and finally we overwrite the old content of variable i with the new
literal value 3. The output of the echo() statement should be 3 and 7
, because in line 3 we copy the content of variable i, which is
currently the value 7, into variable j. The new assignment in line 4
in no way touched the content of variable j.

Maybe that is not too surprising, but when we would have references
instead of plain variables, then the situation would be different, as
we will see soon.

Whenever possible we should use this simple form of variables, as
they are fast and easy to use.

But there exist situations where we need some sort of indirection,
and then references and pointers come into play. For example when the
data entities depend in some form on each other, the elements may
build linked lists, trees or other structures. The entities may have
some neighborhood relation, also called some one to many relation.

Indeed value objects and references occur in real life also:

Imagine you have baked a cake for your family, and you know that your
friendly neighbor loves cakes too. As you have still a lot of all
necessary ingredients and because the oven is still hot, you make one
more identical cake to give it later to your neighbor. We can think
of the cake as a value type, and your second cake can be considered
as a copy. When you give the copy to your neighbor, then you have
still your own, and when you or the neighbor eats the cake, then the
other one still exist.

Now imagine that you know a good car repair shop. You can give the
telephone number or location of that car repair shop to your
neighbor, so he can use that shop too. So you gave him a reference to
the shop, but you gave him not a copy. You can also give some of your
other friends each a reference to that shop, which is nearly no
effort for you. While backing a cake for all of them would be some
effort.

You can regard names of persons as some sort of reference too.
Imagine you have a list with the names of all the people you intend
to invite to your birthday party, and another list with names of
people who owe you money. Some names may be on both list, this is it
refers to the same person.

In computers the dynamic storage, called RAM, consists of
consecutive, numbered storage locations, called words. Each
individual word has its address, which is a number generally starting
at zero and extending to a value which is defined by the amount of
memory available in your computer. These addresses can be used to
access the storage locations, that is to store a value at that
address, or to read the content again. Reading generally does not
modify the content, you can read it many times and will always get
the same value. When your write another value to that storage
location, then reads will give you that new value.

Basically for all the data that you use in your program you need in
some form its address in the RAM, without the address you can not
access it. But what is with all the plain, value object variables we
have used before, we have never used addresses? That is true -- we
used only names to access our variables, and the compiler mapped our
chosen name to the actual address of the variables in memory whenever
we accessed the variable. For most simple cases this is the best way
to access variables. Now let us assume we have such value object type
of variable declared in our program, can we access it without using
its name? When we have declared it, it should reside somewhere in the
RAM when the program is executed. Well, when we do really not want to
access it by variable name, then there is still one chance: We can
search in the whole RAM for the desired content. In practice we would
never do that, as it is stupid and would take very long, but we could
do. But how can we detect our variable? How can we be sure that it is
indeed ours? Generally we can not. Even when we are sure that the
variable must reside somewhere in the RAM, generally the variable is
marked in no way of course. Even when we would know the value which
is stored in that variable, we would only know what bit pattern it
should have, so for most words of the RAM with a different bit
pattern we could say for sure that it can not be our variable, but
whenever we find the expected bit pattern than it can be just a
coincidence, there can be many more words in RAM with that content.
In some way it is as you would search a person and you know that that
person lives in a long road with numbered houses. If you only know
that the person wears brown shoes but you know not the number of the
house nor the name of the person and no other unique property of that
person, then you have not much luck.

References and Pointers

Introduction to Pointers

In Nim references are some form of smart or managed pointers, we will
learn more about references later. The plain pointer data type is
nothing more than a memory address, it is similar to a (unsigned)
integer number. We say that a pointer points to an entity when the
pointer contains the memory address of that entity.

Beside the pointer data type, which is only some RAM address, we have
also the ptr entity. Ptr is not a datatype for its own, it is always
used in conjunction with another data type:

var
 p: pointer
 ip: ptr int

Here the variable p is of type pointer, we could use it to point to
some arbitrary memory address. The variable ip is of type ptr int,
which indicates that it should only point to memory addresses where a
variable with data type int resides. So a ptr is a pointer that is
bound to a specific data type. Generally we speak only about
pointers, if we are referring to an untyped pointer or a typed ptr is
generally clear from the context.

When we only declare pointers but do not assign a value then the
pointers have the value nil, what indicates that they are regarded to
point to nothing. Exactly speaking a pointer can never point to
nothing in the same way as an integer variable can not contain no
number. As an integer variable always contains a bit pattern, a
pointer also always contains a bit pattern. But we are free to define
a special pattern as nil, and whenever a pointer has this special
value, then we know that it does not really point to something
useful. In C instead of nil NULL was chosen for the same purpose. In
practice nil and NULL are generally mapped to 0, that is a word with
all bits cleared. But that is more or less an arbitrary decision.

So how can we give our pointers above a useful value?

One possibility would be to use Nim's addr() function, which gives us
the memory address of each ordinary variable.

var
 number: int = 7
 p: pointer
 ip: ptr int
echo cast[int](p)
echo cast[int](ip)
p = addr(number)
ip = addr(number)
echo cast[int](p)
echo cast[int](ip)

First we declare an ordinary integer variable called number which
will reside somewhere in memory when we execute the program, and then
we use the addr() function to assign the address of that variable to 
p and ip. The addr() function is a low level function provided by the
compiler, it can be used to determine the memory address of variables
and some other entities known to the compiler.^[28] We used the echo
() procedure to show us the numeric decimal value of the addresses in
the terminal. As it generally makes not too much sense to print
addresses, echo() would refuse to print it, so we have used the
construct cast[int](someValue) to tell that echo() should regard our
pointers as plain integer and print it. That operation is called
casting, we generally should avoid it, as it destroys type safety,
but for learning purposes it is OK to use it. We will learn more
about casts and related type conversion later.

The first two echo statements should print the decimal value 0, as
the pointers have the initial default value nil.

The echo()s in the last two lines should print a value different from
0, as we have assigned the valid address of an ordinary variable that
resides somewhere in the RAM when the program is executed. Both
outputs should be identical, as we have assigned for both pointers 
addr(number) each.

Maybe a funny fact is, that when you run the program multiple times
the output of the last two echo() statements print different values.
But that is not really surprising -- whenever you launch the program,
then for our variable number a storage location in RAM is reserved.
And that location can differ for each new program execution. For your
next holiday in the same hotel, you may get a different room also.

So when we have the pointer ip pointing to a valid address, can we
recover the content of that memory region? Sure, we use the
de-reference operator [] for that purpose. Whenever we have a typed
pointer x we can use x[] to get the content of the memory location
where the pointer is pointing to. Note that the operator [] is not
really related to the subscript operator [pos] which we used earlier
for array, seq and string access. Nim uses ASCII characters for its
operators, and that set is not very large. And maybe it would even be
confusing when we would have a different symbol for each operator. We
can consider [] as some form of content access operator -- mystring
[pos] gives us the character at that position, and ip[] gives us the
content of the memory location where ip points to.

var
 number: int = 7
 ip: ptr int
echo cast[int](ip)
ip = addr(number)
echo cast[int](ip)
echo ip[]

What do you expect as output for the last echo() statement? Note that
for the last echo() statement we do not need a cast, as ip[] has a
well defined type: ip has type ptr int, so ip[] is of well defined
type int and echo() can print the content.

Now let us investigate how we can use pointers to modify the content
of variables:

var
 number: int = 7
 ip: ptr int
ip = addr(number)
echo ip[]
ip[] = 3
echo ip[]
echo number

What do you expect for the output of the last echo() statement? Well
remember, ip points to the location where variable number is stored
in RAM. So echo ip[] gave us the content of number. Now ip[] = 3 is
an assignment, the right site of the assignment operator is the
literal number 3, which is a value type. Earlier we said that for
value types an assignment is a copy operation, the right site of the
assignment operator is copied into the variable on the left site. Now
ip[] stands exactly for the same content as the name number, and so
assigning to ip[] is the same as assigning to number.

Pointer Arithmetic

In low level programming languages pointer arithmetic can be useful.
For example old C code often iterates with pointer arithmetic over
arrays by use of constructs like sum += *(myIntPtr++). This was done
to maximize performance. Modern C compiler generally understands
statements like sum += el[i]; i++ well and generates very good
assembly instructions for it. So pointer arithmetic is not necessary
in C that often today.

Nim does not provide math operations for pointers directly, but we
can always cast pointers to integers and do arbitrary math. And of
course we could define our own operators for that purpose, but
generally we should avoid that, as it is dangerous, error prone and
generally not necessary. As an example let us sum up some array
elements:

proc main =
  var
    a: array[8, int] = [0, 1, 2, 3, 4, 5, 6, 7]
    sum = 0
  var p: ptr int = addr(a[0])
  for i in a.low .. a.high:
    echo p[]
    sum += p[]
    echo cast[int](p)
    var h = cast[int](p); h += sizeof(a[0]); p = cast[ptr int](h)
    #cast[var int](p) += sizeof(a[0]) # this compiles but does not work currently

  echo sum
  echo typeof(sizeof(a[0]))

main()

When we do pointer arithmetic or similar math to calculate the
address of variables in the computer memory, then memory addresses
are used like integer numbers, and so it makes same sense that Nim's
integers have the same byte size as pointers.

References:

  * https://github.com/kaushalmodi/ptr_math

Allocating Objects

In the previous section we learned the basics about pointers. We used
the addr() operator to initialize the pointer by assigning the
address of an already existing object. This is in practice not that
often done, and it can be a bit dangerous, as it is not always
guaranteed that the variable on which we applied addr() will exist as
long as our pointer exist. So the pointer may point later to a memory
location that is already freed or used by a totally different object
already. So the use of addr() is more reserved for advanced
programmers who know well what they do, and most of the time addr()
is not necessary at all or is only necessary for really low level
code, maybe when interfacing with external libraries written in C.
Instead of using addr() to assigning to pointers a valid address,
often procedures like alloc() or create() are used to reserve a block
of memory:

var ip: ptr int
ip = create(int)
ip[] = 13
echo ip[] * 3
var ip2: ptr int
ip2 = ip
echo ip2[] * 3
dealloc(ip)

Here the procedure create() is used to reserve a block of memory, the
int parameter ensures that the block has the size of an integer
value. After ip has a valid value, we can store a value in that
memory location and read it again. Note that multiple pointers can
point to the same memory location: We declared one more int ptr
called ip2. But for that pointer we do not allocate a new block, but
we assign the old block that we allocated for ip to ip2. Now both
pointers points to the same object, the int value 13. We may call ip2
an alias, as it is a different way to access the same object.

When we use alloc() or create() to allocate memory blocks, then we
have to deallocate them when we need them not any more. Otherwise
that memory blocks couldn't be reused. If we would continuously
allocate memory blocks and never deallocate, that is free them, then
at some point in time all memory would be occupied -- not only for our
own program, but for all programs running currently on the same
computer. We had to terminate our program -- when a program is
terminated then all resources get freed automatically by the OS.

The use of procedure pairs like alloc() and dealloc() is common
practice in low level programming languages like C, but it is
inconvenient and dangerous: We can forget to call dealloc() and waste
resources, or we may even deallocate memory blocks but still use it
by our pointers. The later would at some point of time crash our
program, as we would use memory blocks which are already released and
may be used for other variables -- from our own program or from other
programs. Note that in the source code above there is only one single
dealloc() call. The reason for that is, that we only allocated one
single memory block in one single create() call, ip2 is only one more
pointer that points to that block. If we would have used an
additional dealloc(ip2) call, then that would be a so called double
free error.

As you see, using pointers is inconvenient and dangerous. But still
there are situations where plain value type variables do not suffice.
The solution of many higher level programming languages to this
problem is a Garbage-Collector (GC). The GC does the dangerous and
inconvenient task of deallocating unused memory blocks for us
automatically.

To distinct the GC managed "pointers" cleanly from the manually
managed ones, we call them in Nim references, in some other languages
they are called traced pointers. References are always typed like ptr
, there is no equivalent to the untyped pointer type for references.

For References we have still to do the allocation our self, then we
can use the references, and when we are not using them any more, then
the GC frees the corresponding memory block. A typical scenario is
that we use references in a procedure or in an otherwise limited
block of code: We declare the reference in that code block, allocated
and use it, and when the code block is left the GC frees the
allocated memory for us. You may think that the fact that we still
have to allocate the memory for our references our self is still a
problem, as we may forget that step. Well it is not that dangerous,
when we forget the allocation step, we would use a reference with
value nil, which would immediate result in a runtime error. So we
would see the problem immediately. Other pointer errors, like missing
de-allocation or use after free are not that obvious and more
dangerous.

With references we can rewrite our previous example code in this way:

var ip: ref int
new(ip)
ip[] = 13
echo ip[] * 3
var ip2: ref int
ip2 = ip
echo ip2[] * 3

We have replaced ptr by ref, and instead of alloc() or create() we
are using the new() proc which gets the uninitialized ref as a
parameter and allocates a managed memory block for it, that is after
the new() call ip has a well defined value referring to a managed
memory block that can store an integer value. Again, we can use one
more ref and assign that ref the value of the other, so now both
references the same memory block. The advantage here is that we don't
have to care about freeing that block, the GC will do that when
appropriate.

To verify that in the example code above both references really
reference the same object in memory, we could add two more lines of
code:

ip2[] = 7
echo ip[]
echo ip2[]

Here we are using the reference ip2 to assign to the memory block the
literal value 7. After that assignment both echo statements would
display that new content.

Using references and pointers to store basic data types like integers
is not done that often, in most cases we work with larger objects,
and we create some relations between the objects. We will try that in
the next section.

References to Objects

You should still wonder for what references are really useful -- they
seem to be only a more complicated version of plain value type
variables.

Now let us assume we want to create a list of things or persons,
maybe a list of our previously used Computer data type, or maybe a
list of persons we will invite to our next party. We will create the
party list for now, as the Computer data type we used before has
already many fields, and filling all the fields would be some effort,
so let us use a new Friend data type which should store only the
friends name for the beginning -- we may add more fields later when
necessary. So we may have

type
  Friend = object
    name: string

With that declaration we could declare a few friends variables like

  var harry, clint, eastwood: Friend

But that is not what we want, we would need a list with all of our
friends that we would like to invite to our party, we would want to
add friends to the list, and maybe we would want to delete friends
also. You may think we could use Nim's sequence data type for that,
and you are right. But let us assume we could not use that predefined
Nim data type for some reason. Then we could create a list of linked
references to Person.

type
  Friend = ref object
    name: string
    next: Friend

Now our Friend data type is a reference to an object , and the object
itself has an additional next field which is again of type Friend.

That is some sort of recursion. If that should appear as too strange,
then imagine you have some numbered paper cards, each with two
fields: One field name, one field next: In the name field you can
fill in a name of a friend, in the next field you fill in the number
of the next card. The last card in the chain gets no entry in the
next field.

In languages like Nim or C lists, also called linked lists, are
dynamically created data structures consisting of elements (called
nodes), where each node has a field which is a references or pointer
to its successor or predecessor. When the nodes have only a successor
field, then we call the list a single linked list, and when it also
has a predecessor field, then we call is a double linked list.
Contrary to arrays and Nim's sequences lists do not allow access to
arbitrary elements, we can only traversal the list starting from its
first element for single linked lists, or also from its last elements
for double linked lists. The first element of a list is also called
its head, and the last element is called its tail. Often the head and
the tail elements are just plain nodes, but the head can be also an
extended node object with additional fields carrying information for
the whole list, maybe an additional string field for the list name.
In this section we use the simplest form of a list, which is a single
linked list where the head is just an ordinary node. If the head has
the value nil then the list is empty.

Now we create a small Nim program which reads in names of our friends
from the terminal, creates a list of all friends, and finally prints
the list.

type
  Friend = ref object
    name: string
    next: Friend

var
  f: Friend # the head of our list
  n: string # name or "quit" to terminate the input process

while true:
  write(stdout, "Name of friend: ")
  n = readline(stdin)
  if n == "" or n == "quit":
    break
  var node: Friend (1)
  new(node)
  node.name = n
  node.next = f
  f = node

while f != nil:
  echo f.name
  f = f.next

1 The actual name for this temporary variable is arbitrary, we could
  have used el for element maybe.

This example code seems to be not that easy. But it is not really
difficult, and when you have understood it, you can call yourself a
Nim programmer already. Maybe you should think about the code above
for a few minutes before reading the explanations below.

First let us summarize what our program should do: It should read in
some names of our friends which we would like to invite to our next
party. Of course when entering the names, we would need a way to tell
that we are done. In our program we can do that in two ways, we can
enter an empty name by just pressing the return key, or we can enter
the text "quit" to stop the loop. Unfortunately that means that we
can never invite a friend with that name to our parties. When we have
terminated the input loop, then the next loop prints all the entries
to the terminal.

Let us start with the type and variable declarations: We use a user
defined type named Friend which is a reference to an object , that
object type has a field name of type string, and a field next which
is again a reference to the same data type.

We are using two variables, one called n of type string to read in a
name or the quit command from terminal, and a variable called f of
type Friend. The variable f seems to match only to one single friend,
but as the type of f has a next field it can be a whole list of
friends, with f being the start or head of that list.

In the code above we are using a special while loop -- special because
the construct while true: and because the loop contains a break
statement. Earlier we said that we should avoid the break statement
in loops, because it interrupts the control flow and can make it more
difficult to understand and proof the flow. But in this case that
form makes some sense: For the first loop we have to first read in a
name from terminal and then we can decide what to do, so we can not
really evaluate a condition after the while statement at the top. So
we use the simple constant condition true, which would never
terminate the loop. We need a break inside the loop body to terminate
the loop.

Let us investigate the second loop first as it is really easy: In the
while condition we check if current value of f is nil, that is if
there are no more entries in our list. For that case we terminate the
loop, as we are done. If f has not the value nil, than f points to a
valid content, that is there is at least a valid name, which we
access by the field access operator and print it with echo f.name.
Note that in Nim the field access operator . works in the same way
for value objects types as well as for ref objects types. For ref
objects types we could also write f[].name instead of plain f.name,
that is we first apply [] to f to get the content, and then use the .
operator to access the name field. In some other languages like C we
would have to use a special operator -> to access fields of pointer
or reference types.

The most interesting statement in the output loop is f = f.next. We
assign the content of f.next to f and proceed with that new content.
The content could be a valid reference to one more Friend object, or
it could be nil, indicating that our loop should terminate.

The input loop is also not that complicated: To make the process of
adding more friends to the list easy, we always add the new names at
the beginning. First we ask the user to enter a name. We use write
(stdout) for this, as echo() always generates a newline, but we want
to read in the name on the same line. If the name is empty or has the
special value "quit" then we terminate the input loop. In the loop we
are using a temporary variable called node of type Friend, we
allocate a memory block for it with new().^[29] Then we assign the
read in friend's name n to the name field. The last two statements of
the loop body are a bit demanding: First we assign to node.next the
value of f. Now node is basically the start of our list, and its next
field refers to the first element of the current list. Fine, but we
said that the node variable is only a temporary variable, we do not
intend to use it longer as necessary. But currently node is so
useful, it is the head of our list. On the other hand, the former
list start f is now useless, current f is identical with node.next.
So the trick is, we just assign to f the value of node. Now f is the
complete list, and we do not need node any more. The node variable
can be used in the next loop iteration again, but we have to allocate
a new memory block for the node reference, as the previous memory
block is still in use, it contains the name which we just entered and
also a reference to the next object in the list.

Note that we add the new elements at the top of the list in this way.
We have done it that way because it is very easy in this way. For
adding at the end of the list, we would have to use one more
reference variable which allows us always access to the current end
of the list, or we would have to traversal the list from head to tail
whenever we would like to add elements at the tail.

For one more exercise let us consider deleting entries in our list.
Basically that operation is very easy, we would just skip one entry.
Lets adds this code to the program above:

while f != nil:
  write(stdin, "Name to delete: ")
  n = readline(stdin)
  if n == "" or n == "quit":
    break
  if f.name == n:
    f = f.next
  else:
    while f.next != nil:
      if f.next.name == n:
        f.next = f.next.next
        break
      f = f.next

Here we are using again an outer while loop to read in the names
which we want to delete. That loop uses the condition while f != nil:
because when the list is empty we should stop of course.

In the loop body we have an if statement, and in the else branch of
the if statement we have one more loop. The reason why we need the if
statement is, that the case that our name to delete is the first in
the list is some sort of special. Let us investigate the inner loop
first. That loop assumes that there are at least 2 elements in the
list, f and f.next. We compare the name of the next entry with n. If
they match then we would have to skip the next entry. We can do that
by the statement f.next = f.next.next. That is we replace the
reference from the current element f to the next list entry, that is 
f.next, by the next entry of the next element, which is (n.next).next
. We do not have to write the parenthesis. The n.next.next entry can
be nil, in that case it is the end of the list. If we found a
matching name then we terminate the inner loop with a break
statement, and we are done. Otherwise we assign to f the value of 
f.next and continue the loop execution. Now to the special case that
the name to delete is the first in the list. We need the first if
branch for that -- if already the first element matches the name to
delete than we just skip the first element by setting the head of the
list to the next entry, which may or may not be nil.

This is one way to solve the task, for operations on lists there
exist in most cases various solutions, some optimized to easy or
short code, some for performance. You may copy the code segment above
to the end of the former code, and maybe add one more copy of our
printing loop at the end again. Then you should have a program that
reads in a list, prints the contents, then ask for names to delete,
and finally prints the resulting list. Maybe you can improve the
code, or maybe you can detect special corner cases where it may fail.
What is for example when some of your friends have the same name? May
the program fail in that case? Or you may add more fields to your 
Friend data type. Maybe a textual field with content male or female,
and you can report the ratio of male to female. And maybe remove
males from the list when we have more males then females?

For references to objects the assignment operator = copies the
references, but not the object. In the same way the operator == for
equality test compares the references, but not the content of the
objects to which the references point. If you want to compare the
content of the objects, you can apply the dereference operator [] on
both references:

type
  RO = ref object
    i: int

var
  ro1 = RO(i: 1)
  ro2 = RO(i: 1)
  ro3 = ro1

echo ro1 == ro2 # false
echo ro1[] == ro2[] # true
echo ro1 == ro3 # true

Procedures and Functions

Procedures and functions, called proc and func in Nim, are the most
common way to structure, or break larger programs into smaller,
dedicated tasks which are to be performed.

The terms procedure and function were used in Pascal and the other
languages of Wirth already, while C uses the term function only, and
Fortran generally uses the term subroutine instead. And finally,
Python and Ruby are using the really strange terms def and fun for
it.

Nim's procedures are basically similar, but much more advanced than
its equally named cousins in the wirthian languages, or the plain
functions in the C language: Nim's procedures support generics and
overloading, named parameters and default values, the special
parameter types varargs and openArray, various ways to return a
result and finally multiple calling conventions including the method
and command calling conventions.

Introduction

We call or invoke a proc by just writing its name followed by a
parameter list enclosed in parenthesis. The parameter list can be
empty. When we call a proc, then the program execution continues with
that procedure, and when the execution of the procedure terminates,
then the next statement after that proc call is executed. Sometimes
we say that we jump into a procedure and jump back when that
procedure terminates.

In Nim functions are a special form of procedures that return a
result and do not modify the current state of the program. Modifying
a global variable or performing an input/output operation would be
examples for modifying the state. We have already used some
predefined procedures like echo() for output operations, add() for
appending single characters to strings, and readLine() for reading in
textual user input. And we talked about math functions like sin(), 
cos(), pow() -- these are functions as they accept one or two
arguments and return a result but do not change a state -- calling
them again with the same arguments would always give the same result.
ReadLine() is only a proc, not a function, as the result may be
different for each call, and as we pass a file variable as argument,
which may change its state for each call, maybe because the file end
is reached. A function is only a special sub-type of a procedure, the
func keyword indicates to the reader of the code and to the compiler
some special properties, that is that a result is returned and that
global state is not changed. Whenever the func keyword is used a proc
would do as well, and in this text we generally speak about
procedures, even when a function would do.

Let us start with a very simple function called sqr() for square.

func sqr(i: int): int =
  i * i

A procedure declaration consists of the keyword proc, a user selected
name, a optional parameter list enclosed in parenthesis and an
optional colon followed by the result data type. For a function
declaration we use the keyword func instead of proc, and as functions
does always returns a result, we have always to specify the result
data type.

Note that this is only a declaration so far -- the compiler could
recognize the construct, its parameters and its result type.
Sometimes we call this construct a procedure-header.

Generally we do not only declare a function, but we define it, that
is we add an equal sign to the procedure header and add an indented
procedure body that contains the code that is performed for each
invocation.

Pure proc declarations can be necessary in rare situations, maybe
when two procedures call each other. In this case the procedure
defined first would call the other procedure, which is not already
defined, so the compiler may complain about a unknown procedure. We
could solve that problem by first declaring the second procedure, so
that the compiler would know about it existence. We would then define
that second procedure later, that is closer to the end of the program
file.

The sqr() proc above accepts an integer argument and returns its
square of same data type. We would call that proc like

var j: int
j = 7
echo sqr(j)

Earlier in this book we said that the compiler processes our source
code from top to bottom, and that the final program is executed from
top to bottom too. The first statement is indeed true, for that
reason it can be necessary to declare a function at the top, and
define it below, as we can not call a proc before it is declared or
defined.

For the program execution we have to know that procs are only
executed when we call them. That is, when we write a proc at the top
of our source code, then that proc is processed by the compiler, but
it is not executed during program runtime before we call it.
Actually, as the Nim compiler supports "death code removal", code of
procedures that we never call would not make it in to our final
executable at all.

The procedure body builds a new scope. We can declare entities like
variables, constants, types or other procedures and functions in that
scope. These entities are only visible in the procedure body but not
outside of the proc. Will will learn more about scopes and visibility
soon.

Parameter lists of procedures consist of one or more lists of
parameter names, separated with commas, followed by a colon and the
data type of the parameters. The sub-lists with same data type are
separated by semicolons:

proc p(i, j, k: int; x, y: float; s: string)

While the wirthian languages would require semicolons to separate the
parameter blocks, in Nim we could also use plain commas for that. For
the data types of proc parameters all of Nim's data types are
allowed, including structured types, ref, pointer and container
types, and we can pass literal values, named constants or variables.

When we call such a proc with multiple arguments, we have to specify
the arguments in the order as they are listed in the proc header,
separated with commas, and the arguments must have compatible data
types:

var i: int = 7; x: float = 3.1415
p(i, 13, 19, x, 2.0, "We call proc p() with a lot of parameters")

Here compatible data types means, that for the i, j, and k parameters
which are specified as int type in the proc definition, variables of
smaller int types like int16 would work. For the two parameters of 
float type, we would have to pass floating point variables or a float
literal. As a special case an int literal would work also, as the
compiler knows the desired data type and automatically converts the 
int literal into a float for us, as long as that is possible without
loss of precision. We could pass 2 instead of 2.0, but passing a very
long int literal with more than 16 digits may fail at compile time:

proc p(i, j, k: int; x, y: float; s: string) =
  echo s

var
  n: int16
  m: int # int64 would not compile
  z: float32
p(n, n, m, 1234567890, z, "")

Actually float32 types and int literals up to ten digits seems to
work for float parameters, but even on 64 bit systems the int64 data
type is not allowed for int parameters. As you see from the example
above, it is possible to pass the same variable multiple times as a
parameter, and empty string literals are of course allowed too.

Nim does also support default values for proc parameters and named
parameters, that is that we can leave parameters unspecified and use
the default value, or use the actual parameter names like in an
variable assignment when we call a proc:

proc p(i: int; x: float; s: string = "") = echo i.float * x, s
p(x = 2.0, i = 3)

Here we used named parameters when calling the proc p(), this way we
can freely order the parameters, and as parameter s has a default
value, we can let it unspecified and just use the default value.

Functions always return a result, and procedures can return a result,
but they don't have to. In the C language function results can just
be ignored, but in Nim whenever there is a result, then we have to
use it at the call site, that is we have to assign the returned value
to a variable or we have to use it in an expression. Nim enforces
this, as generally the returned value is important, the returned
value may be the actual result as in a sin() call, or it may give us
additional information, like the number of read characters when we do
text processing or maybe an error indication, like end of file. For
the rare conditions when we really intend to ignore the result of a
function call, we can call that function as discard myProcWithResult
(a, b,... ). Another solution is to apply the {.discardable.} pragma to
the function definition, we will learn more about pragmas later. When
a procedure should not return a result, then we can use the void
return type or just leave the return type out -- the later is
recommended, void types are used only rarely in Nim. When the proc
has no parameters at all, then we can even leave out the empty
parameter list in the proc definition:

proc p1() =
  echo "Hello and goodbye"

proc p2 =
  echo "Hello and goodbye"

proc p3: void =
  echo "Hello and goodbye"

Calling Procedures

When we call a procedure or a function, that is when we intend to
execute it, then we have always to specify a parameter list enclosed
in brackets, but the parameter list can be empty:

var i = myFunc(7)
var j = myF()
var p = myF # not a function call, but assignment of the proc to variable p

Note that the last line in above code is not a call of myF(), but an
assignment of that function to the variable p. Will will discuss this
use case soon.

We have already learned, that we can also use the method call syntax,
like 7.myFunc instead of myFunc(7), and we can use the command
invocation syntax like in echo "Hello" and that we should avoid
putting a space between the proc name and the opening bracket as that
would be interpreted as a call with a tuple argument. When the
function or proc expects multiple arguments, then we separate the
arguments with commas, and we put generally a space after each comma:

proc p(i, j: int) = i + j
echo p(1, 2) # ordinary proc call
echo 1.p(2) # method call syntax
echo p 1, 2 # command invocation syntax
echo p (1, 2) # argument looks like a tuple, so this would not compile

For the proc definition above we wrote the body statement directly
after the equal sign -- that is possible, and used sometimes for very
short procs. And indeed, here p() is a function.

In the examples above we have passed plain integers as parameters to
procedures, but of course proc parameters can have any type, we can
pass strings, array, objects and all that. The way we pass the
parameters to the procs is sometimes called "pass by value", an old
term introduced for the Pascal language, used to indicate that the
passed parameter seems to be copied to the proc, the proc is not able
to modify the original instance. In the next section we will learn
about the var parameter type, which is used when we want to allow the
proc to modify the original instance. In the wirthian languages the
procedure parameters actually get copied, so inside the proc we could
modify it, but we modified only the copy, the original instance
remained unchanged. In Nim it is a bit different. When we pass
parameters by value to a proc, we can not modify it at all in the 
proc body. When we need a mutable copy, we have to generate that copy
our self in the proc body. This allows some optimizations: Nim needs
not really to copy the proc parameters, as they are immutable, Nim
can just work with pointers to the original instances internally.
Actually there are rumors, that for parameters smaller than 3 *
sizeof(float) Nim copies the instances, but for larger instances Nim
works internally with pointers to the original value. But this is an
implementation detail -- data copied to the procs stack allow fastest
access, but on the other hand the initial copy process can be
expensive, so it is a compromise.

Procedure Parameters of Var Type

Our sqr() function above accepts only one parameter and that
parameter is a value type, which indicates that we can not modify it
in the procedure body. That fact is useful to know for the caller of
a proc, as one can be sure that the passed parameter is not modified
and is available unchanged after the proc call.^[30] But of course
there are situations where we may want that a passed parameter is
modified. Let us assume that we want to "frame" a passed string, for
example we want to pass in the string "Hello" and want to change it
to "* Hello *". Further let us assume that we may sometimes want to
use other characters instead of the asterisk, maybe a + sign.

proc frame(s: var string; c: char = '*') =
  var cs = newString(2)
  cs[0] = c
  cs[1] = ' '
  insert(s, cs)
  add(s, ' ')
  add(s, c)

# we can call that proc like
var message = "Hello World"
frame(message)
echo message

Note: In the wirthian languages we actually put the var keywords for 
proc parameters in front of the parameter name, that is we would have
to write proc frame(var s: string; c: char = '*') = for the procedure
header.

The frame proc above accepts two parameters and returns no result.
The first parameter has the type string, it is not a value parameter
but a var parameter, which is indicated by the var keyword between
the colon and the type of the parameter. Note that we use here again
the keyword var that we used earlier to declare variables. The main
reason that we use again the same keyword is that we do not want to
use a new one -- var proc parameters are different from var
declarations. Parameters of var type can be modified in the procedure
body, and that modification is visible after the proc call.^[31] The
second proc parameter is a plain value type, it is a character which
has the default value '*'. To specify a default value for a
parameter, we write an equal sign after the parameter type followed
by the actual default value, like we would do it in an assignment.
Indeed as in an assignment, we can even leave out the colon with the
data type in this case, at least for the case that the compiler can
infer the correct data type from the assigned default value. Default
values are useful for parameters that have in most cases the same
value, but can be different sometimes. The advantage is, that when
calling that proc we can just leave that parameter out. For default
values we have to be a bit careful, only value parameter can have
default values, and when we call a proc with many parameters with
default values it may be not always clear which parameter we pass and
for which parameter we want a default value.

To generate the frame around the passed in string we have to insert
two characters at the front of the string, and to append two more
characters. Inserting in strings is not a very cheap operation, as it
involves moving all following characters. So we try not to insert two
single characters, but we first create a short string consisting of
the passed c character and a white-space character, and then insert
that two character string at the front of the passed string. We use
the standard procedure newString() with parameter 2 to create a new 
string of length 2 with undefined content, and then fill in the
content by using the subscript operator. We could have used the add()
proc to add that two characters to an empty string, but that is a bit
slower. Then we use the standard procedure insert() to insert our two
character string at the front of our passed string. Finally we add a
white-space and the c character to the passed string. The passed
string is now modified, it is 4 characters longer. That modification
is noticeable for the caller of that proc, that is echo() will print
the modified version.

Passing mutable arguments to procedures by use of the var keyword was
sometimes called "pass by reference" in the old wirthian languages
like Pascal. This leads to confusion for some people unfortunately.
Of course proc var parameters are not really related to Nim's ref
type. Well, using Nim's ref data types would also allow to modify 
proc arguments, in the same way as using pointers would do it. But we
never use ref types in Nim just to be able to modify passed data in 
procs, and also not to avoid a possible expensive copy operation for
value types. We could create a ref instance with var intRef: ref int
= new int, pass that intRef to a proc and so allow to modify the
actual value where the intRef points to from inside the proc. But
that would be silly, as the var parameter is available for that. In
Nim, we use reference types, when we really need then, e.g. when we
really need reference semantics, or when we have to create highly
dynamic, many to one data types, like tree structures.

When we call a proc or function with multiple arguments, then we have
to pass the arguments in the same order as they are specified in the 
proc declaration.

Our frame() proc above modifies the passed string. We could have
instead decided that the proc should not modify the string, but
should return a new string consisting of the frame and the passed 
string in the center. Generally when creating procs we have to decide
what is more useful -- modifying a passed value or returning a
modified copy. And sometimes we have to regard efficiency too.
Returning newly created large data types like strings may be
expensive. A string is not a trivial structure, as it contains the
dynamic buffer for the string content, which has to be allocated. On
the other hand, for the passed var string we inserted characters,
which involves moving characters and is also not a really cheap
operation, and maybe when we insert a lot, the string buffer must be
even enlarged, which is again expensive. So for this use case it is
not really clear what approach is better -- we used the var parameter
mainly to introduce var parameters. OK, let us investigate how a
function that returns a modified string may look:

func framed(s: string; c: char = '*'): string =
  var res = newStringOfCap(s.len + 4)
  add(res, c)
  add(res, ' ')
  add(res, s)
  add(res, ' ')
  add(res, c)
  return res

# we can call that proc like
echo framed("Hello World")
echo framed("Hello World", '#')

Above code is one possible solution. We can use the keyword func
instead of proc here as we only return a result but modify no states.
We pass the initial string and the character for the frame both as
plain value parameters and return a newly created framed string. In
the function body we start with an optimized version of the procedure
newString() from the system module, called newStringOfCap(). Like 
newString() that procedure creates an empty string variable, but it
ensures that the data buffer of the new string has exactly the
specified size. That is an optimization, which makes sense in our use
case, as we know that our newly created string will have 4 characters
more than the passed string. So we can avoid that the result string
has to be enlarged while we add characters or the initial string, and
we ensure at the same time that no space is wasted -- the data buffer
size of the new string will be a perfect fit for the desired result.
The rest of the function body is not really interesting, we just add
() what is needed and return the result. Well, earlier we said that 
add() is not extremely fast. So when you have to frame millions of
strings each day you may consider avoiding add(), and you know
already enough about Nim to do it. Just try it. You may start with a 
string of right size containing undefined content created by 
newString(s.len + 4) and then you may copy in the required data in a
loop character for character. Or you may use the slice operator to
insert the passed string into the new string.

Click to see a possible solution

func framed(s: string; c: char = '*'): string =
  var res = newString(s.len + 4)
  res[0] = c
  res[1] = ' '
  res[2 .. s.high + 2] = s # we may insert the string by using the slice operator or
  # for p in 0 .. s.high: # we can use a for loop and
  #   res[p + 2] = s[p] # the subscript operator
  res[^2] = ' '
  res[^1] = c
  return res

The situation that we may need a procedure that works on a var
parameter in one case and returns a modified copy in another case is
not that rare. So for example Nim's standard library contains a
procedure called sort() which can sort container data types in place,
and a procedure called sorted() which returns a sorted copy. This
code duplication is not really that nice. Of course sorted() is the
more universal solution, as we can always replace sort(data) with 
data = sorted(data). But the later creates a temporary copy, which
may not be optimal for performance. Since Nim version 1.2 a dup()
macro is available from sugar module which creates copies of
variables and then applies one or multiple in place procs on the
copy. So the procs sorted() or our proc framed() would be
unnecessary. We can use dup() as in this example:

from sugar import dup

proc frame(s: var string; c: char = '*') =
  var cs = newString(2)
  cs[0] = c
  cs[1] = ' '
  insert(s, cs)
  add(s, ' ')
  add(s, c)

echo "Hello World".dup(frame)
echo "Hello World".dup(frame, frame)
echo "Hello World".dup(frame('#'))

Note that we apply frame() two times in the line before the last
one -- in the same way we could apply a sequence of different procs.
The result of above program is

* Hello World *
* * Hello World * *
# Hello World #

Returning from a Procedure and the implicit Result variable

The execution of a procedure terminates when the last statement of
the procedure body has been processed. We can also terminate a
procedure earlier when we specify a return statement somewhere.

Functions and procedures which return a result can also terminate
with the last expression of the procedure body, or earlier with a
return expression like return i * i. Functions and procedures with a
result declare automatically a mutable result variable for us, which
is of the function's return type and which we may use or just ignore.
So for our previous sqr() function we have various ways to write it:

func sqr1(i: int): int =
  i * i

func sqr2(i: int): int =
  result = i * i

func sqr3(i: int): int =
  return i * i

For short and simple procedures the first form is often used. For
longer procedures where the result is constructed in multiple steps,
like some string operations, using the result variable makes sense.
And finally, when there exist multiple points where we may jump back
using return statements may make sense. One use case is an early
error check, maybe we want to return -1 as some form of error
indication when we write a procedure that should calculate the square
root of an integer value. (Well in Nim we have other and sometimes
better ways to catch errors, we will learn about that later.)

Generally we should avoid writing something like

func sqr(i: int): int =
  result = i
  i * i

as it may be unclear in this case if the expression i * i is returned
or the result variable with value i. For Nim v1.6 we will get a
warning or an error message in such a case.

For the performance of our code, it may have a tiny benefit to only
use the result variable and fully avoiding return statements, as in
this case for a function call like var i = sqr(j) the result variable
may be just an alias for the actual result i here, so that the
compiler can optimize the code and avoid temporary copies. But that
are rumors, and may depend on the actual compiler version.

Var Return Type

A procedure, converter, or iterator may return a var type, which can
be modified by the caller. The Nim compiler manual provides this
basic example:

var g = 0
proc writeAccessToG(): var int =
  result = g
writeAccessToG() = 6
assert g == 6

This way we can call a proc and immediately assign a new value to the
result. In the above example this works, as the result is an alias
for the global variable g.

Actually used are var return types for iterators like mitems() or 
mpairs(), which allows to modify the yielded results. For details and
restrictions of the var return type you should consult the Nim
compiler manual:

References:

  * https://nim-lang.org/docs/manual.html#procedures-var-return-type

Proc name overloading

Note that we used the proc names sqr1, sqr2 and sqr3 above. Using the
same name with the same argument types multiple times would result in
a redefinition error, as the compiler could not know what proc body
should be executed when that proc name is called.

But Nim supports so called proc overloading, that is we can use the
same name when the parameter list is different, as the compiler can
select from the parameters in the proc call which proc has to be
called:

func sqr(i: real): real =
  i * i

We have only changed the parameter and result data type. Now there is
no conflict with the proc with same name which we defined for
integers. Note that Nim use only the parameter list for overload
resolution, but not the result type of a proc or function. The reason
for that is that Nim supports type inference, and that would not work
when we would have two procs with same name each accepting an int
parameter but one returning an int and one returning a float number.

Nim does also support named arguments in proc calls, that is we could
invoke the proc above with sqr(i = 2.0). Named arguments can be
useful when procs or functions have many arguments, maybe some with
default values, and we do not remember the order of parameters or
when we want to specify only a few.

Objects and Ref Objects as Procedure Parameters

In the previous section we learned that we have to pass var
parameters when the procedure should be able to mutate the variable
permanently. This is also valid when the parameters are objects. When
a procedure should modify fields of an object parameter, then we have
to pass that object as a var parameter. In the following example proc
t1 gives a compiler error because that procedure tries to modify a
field of an object while the object instance is not passed as a var
parameter. If we remove proc t1 then we can compile and run the
example:

type O = object
  i: int

proc t1(o: O) =
  o.i = 7 # Error: 'o.i' cannot be assigned to

proc t2(o: var O) =
  o.i = 13

proc main =
  var x = O(i: 3)
  echo x.repr
  t2(x)
  echo x.repr

main()

The output is:

[i = 3]
[i = 13]

Proc t2 gets a var parameter and can modify fields of the passed 
object. Here we used the expression echo x.repr to print the whole
object. Strings and sequences are value objects in Nim, so you have
to pass them as var parameters when you want to change their length
or when you want to modify elements. This code would give you compile
errors, unless you add the var keyword to make the proc parameters
mutable:

proc t1(s: string) =
  s.setLen(7)
  s[0] = 'x'

proc t2(s: seq[int]) =
  s.setLen(7)
  s[0] = 13

This was not really surprising. But what when we use a reference to
an object and pass that to procedures as value and as var parameter?
In the code below proc t1 gets a variable of type ref object and the
procedure can modify fields of the passed instance. That can be
indeed surprising. In this case passing the ref object without use of
the var keyword means only that we can not mutate the ref value
itself in the procedure, but we are allowed to modify the fields of
the object. For proc t2 we pass a var parameter. As always we can
modify a var parameter in the procedure, so we can assign it a newly
created instance.

type O = ref object
  i: int

proc t1(o: O) =
  o.i = 7

proc t2(o: var O) =
  o = O(i : 11)

proc main =
  var x = O(i : 3)
  echo x.repr
  t1(x)
  echo x.repr
  t2(x)
  echo x.repr

main()

When we compile and run above code we get:

ref 0x7f054a904050 --> [i = 3]

ref 0x7f054a904050 --> [i = 7]

ref 0x7f054a904070 --> [i = 11]

For a ref object the repr() function gives us the address of the 
object instance in memory and the contents of its fields. The first
two echo() statements shows the same address, indicating that proc t1
has modified only a field of our instance, the instance itself (its
address in memory) was not changed. But proc t2 has created a new
instance and assigned that value to the variable x in the main()
procedure. We notice this as the address of variable x has changed.
The old instance variable with address 0x7f054a904050 is now unused
and will be freed by the Nim memory management.

Special Argument Types: OpenArray and Varargs

The openArray and varargs data types can be used only in parameter
lists. OpenArray is a type which allows to pass arrays and sequences
to the procedure or function. Allowing that makes sense as arrays as
well as sequences store their content in a block of memory, which can
be processed uniformly. Although arrays generally do not have to
start with index number 0, when passed as openArray the first element
is mapped to index 0, and the index of the last element is available
by using the high() function on the passed array parameter. Whenever
we write a procedure that accepts an array or a sequence, we should
consider using the openArray parameter type to allow passing in both
data types. Strings can be passed also to procs accepting openArrays
with char base type. Note that a proc with openArray parameter type
can not change the length of a passed seq, as for the openArray
parameter type sequences are handled like arrays. So in the code
below proc t1 generates a compiler error while t2 compiles and works
fine.

proc t1(x: var openarray[int]) =
  x.setLen(7)

proc t2(x: var seq[int]) =
  x.setLen(7)

The varargs parameter type is similar to the openArray type, but it
additional allows passing an arbitrary number of single arguments.
The compiler automatically collects the single arguments into an
array for us, so in the proc body we can use it like an array, e.g.
iterating over it.

proc print(s: varargs[string]) =
  for el in s:
    stdout.write(el)
    stdout.write(", ")
  stdout.write('\n')

print("Hello", "World") # compiler builds the array for us
print(["Hello", "World"]) # we generate the array our self

There exists a variant of the varargs argument type that performs a
type conversion automatically by applying a proc on all arguments.
For example varargs[string, $] would apply the stringify operation on
the passed arguments automatically. That is what echo() does.

Varargs arguments may be only allowed for the last argument in a
parameter list.

Finally we may wonder if it makes sense to specify a parameter of
type var varargs. If we try to pass a constant string this will
obviously not work, and if the compiler generates an array for us it
does also not work, the automatically generated array seems to behave
like a constant array. But may we pass an array variable? Let us try:

proc print(s: var varargs[string]) =
  s[0] = "Goodbye"
  for el in s:
    stdout.write(el)
    stdout.write(", ")
  stdout.write('\n')

var msg = ["Hello", "World"]
print(msg)

Surprisingly that does not compile, while it works when we replace 
varargs with openArray.

Procedures bound to a Data Type

In some other programming languages like Python or Ruby we can define
class methods or static methods which are bound to a class or type
and can be called as MyType.myProc. In Nim we can do something
similar by use of the typedesc proc parameter type:

type
    Factory = object
        name: string

proc start(t: typedesc[Factory]) =
    echo "Factory.start"

Factory.start

Here we used the method call syntax instead of start(Factory). We
will learn more about the typedesc data type later.

Scoping , Visibility and Locality

Scoping, visibility and locality is an important concept in computer
programming to keep the source code clean. Imagine that a variable
which we declare at some point in our program would be visible
everywhere. That would even for medium size programs generate a lot
of confusion -- whenever we would need a variable we would have to
carefully check which names are already in use. And for performance
it would be bad also, as all variables declared somewhere would
reside permanently in memory.

So most programming languages including Nim support the concept of
locality -- names declared inside of a procedure body or inside
another form of block are only visible there and can only be used
there. We say that they are only visible in that scope. For Nim we
can say that whenever Nim's syntax requires a new level of
indentation, that is a new statement block, then all symbols declared
in that block are only visible in that block and in sub-blocks of
this block, but not outside of that block. Nim has another important
concept of visibility, which is called modules and allows separation
of our code in logically separated text files with well defined
visibility rules, we will discuss modules later.

Visibility is really a simple concept, let us regard this useless
example:

var e: float = 2.7

proc p1 =
  var x: float = 3.1415
  if x > 1.0:
    var y = 2.0 * x
    echo y # OK
  echo x # OK
  echo y # compile error, y is not visible
  echo e # OK, e is declared globally, so it is visible everywhere

echo e # OK
echo x # ?
echo y # ?

In line one we declare a so called global variable, that one is
visible after declaration, that is below the line where it is
declared, in the whole program. The variables declared in the proc p1
are called local variables, they are not visible outside of that proc
p1. The variable x is declared at the start of the proc body and is
visible in the whole proc everywhere, while variable y is declared in
the if block and is visible only there. So it should be clear if the
last two echo statements for x and y compile fine? Remember that
symbols that we define inside of a new scope may shadow symbols that
were visible outside of the actual block, e.g. by defining a variable
named e of arbitrary type in the proc p1 from above would shadow the
global variable e, that is the global variable e would become
invisible until execution of proc p1 terminates. We discussed
shadowing already in the introducing section Scopes, Visibility,
Locality and Shadowing.

Related to visibility of variables is their lifetime, that is the
duration how long they exist and how long they can store a value.
Global variables exist for the whole program runtime -- when you have
assigned a value to it that value can be used everywhere as long as
the program runs, and as long as you do not assign a different value
of course. Global variables are generally stored in a special memory
region that is called the BSS region.

Variables of value type defined locally inside a procedure or
function do exist only for the execution time of that proc, that is
they are created when the proc is invoked and vanish when the proc
terminates, that is when execution continues with the statement
following on the proc call.

Local variables declared in a proc reside in a special memory region
of the RAM which is called the stack. The stack is nothing more than
an arbitrary part of the hole RAM that is used in some clever
fashion: The memory words in it are used in consecutive order. A so
called stack pointer is used to indicate the address of the first
free area in that stack. So when a proc is called, which may have n
bytes of local variables, then the compiler can use the area where
the stack pointer points to for that variables, and when the proc is
called then the stack pointer is increased by that size. So the stack
pointer points again to the next free area of the stack, and another 
proc can be called in the same way from within the current proc.
Whenever a proc terminates, the stack pointer is set back to the
value which it had when the proc starts execution. This method of
memory management is simple and fast, but it does only work when the
total amount of memory that the local variables in a proc needs is
known at compile time, so that the compiler can adjust the stack
pointer accordingly. It does not work for dynamically sized data
types like strings or sequences.

Note that pointers and references are value types itself, we can
regard pointers and references as a plain integer variable
interpreted in a special way -- as a memory location. But the memory
blocks to which the pointers and references may point and that is
allocated by alloc() or new() is different: That memory blocks are
not allocated on the stack, but in the ordinary RAM which we call
heap to separate it from the stack.

So why can the stack not be used for memory blocks which alloc() or 
new() provides for us: An important fact for the use of the stack to
store variables is that the total size which is needed by a proc for
all the static variables must be a compile time constant. The stack
pointer is adjusted by that amount when the proc starts and all the
local variables are accessed with a fixed offset to that stack
pointer then. When we use alloc() or new() in a proc, then we may
call that multiple times like we did in our previous list example,
and for alloc() an additional fact is that the byte size that alloc()
should reserve can be a runtime value. So the total amount of RAM
that alloc() or new() would allocate is a runtime value, and we can
not use the stack for it. Instead alloc() and new() allocates block
of memory in a more dynamic fashion, which is basically that they ask
the OS for a free block of right size somewhere in the available RAM.
That block is later given back to the OS for reuse by functions like 
dealloc() or automatically by the GC.

Let us at the end of this section investigate some special cases:

While in languages like C we have always a well defined main()
function and all program code is contained in this function or in
other functions which are called from this main function, in Nim we
have also global code as in scripting languages Ruby or Python:

var i: int
while i < 100:
  var j: int
  j = i * i
  echo j
  inc(i)

It should be clear that the global variable i resides in the BSS
segment. But what is with the variable j declared in the body of the 
while statement? It is clear that that variable is only visible
inside of the body of the while statement. But does j reside on the
stack? There seems to be no proc involved so there may be no stack?
The variable j may reside in the BSS segment too? That is not really
clear and may be different for different Nim compilers maybe. But why
should we care for that detail at all? Well it may be important for
performance. Local proc variables allocated on the stack are
generally optimal for performance, and they are optimized by the
compiler very well. We will learn more about the reasons for that
later when we discuss the data cache. For now we should only remember
that it may be a good idea to avoid global code and put all code in 
procs. We may have an arbitrary named main() proc then and call that
from global scope only. At least for the current Nim v1.6 that seems
to be a good idea, maybe later versions or other implementations will
automatically move all global code into a hidden proc for us.

 For optimal performance put all your code in procedures or functions
 and avoid global code and when possible global variables.

Let us discuss above while loop again, but this time in the body of a
proc:

proc p =
  var i: int
  while i < 100:
    let j: int = i * i
    echo j
    inc(i)

When we carefully investigate that proc within the while loop we may
wonder about two points. First we said earlier that we can and should
use the let keyword instead of var when there is only one assignment
to a variable, so the variable can be regarded as immutable. But the
loop is executed 100 times, so how can we say there is only a single
assignment to variable j? The trick is, that j is locally to the 
while loop, and that j is virtually newly created and initialized to 
0 for each iteration. So let is OK and the compiler does not
complain.

We can test that fact with this simple program:

proc main =
  var i: int
  while i < 10:
    var a: int
    a = a + 1
    echo a
    inc(i)
main()

The output is 1 for each loop iteration, as variable a is virtually
newly created for each loop iteration.

We said virtually newly created, because we can not be sure how the
compiler may handle it internally. Is storage for variable a already
allocated when the proc is invoked, that is in the same way as
storage for the loop counter variable i is allocated on the stack
when the proc is called. Or is storage for variable a reserved for
each loop iteration by increasing the stack pointer at the start of
the loop and resetting it at the end of the loop. We can not be sure
without reading the compiler source code, but finally we should not
care, as it does not really matter.

Generics

In the previous section we defined a sqr() proc for ints and one for 
float numbers. Both procs look nearly identical, only the data types
differ. For that case we can use so called generic procedures.

func sqr[T](v: T): T =
  var p: T
  p = v * v
  return p

echo sqr(2)
echo sqr(3.1415)

We put a square bracket after the function name which includes a
symbolic name, and that name is then used instead of concrete types
in the proc header or in the proc body.

We can now call that proc with parameters of different types
including int and float types. You may wonder why that works -- Nim is
a statically typed language, so how can the parameter of function sqr
() as well accept an integer as a floating point number? Is there a
hidden type conversion involved? No, the trick is that whenever we
call that generic proc with a different type, then a new proc or
function is instantiated. As we called the generic sqr() proc with an
int and a float parameter, during compile time the compiler creates
machine code for two separate functions, one which is called when an 
int is passed as parameter, and one which is called when a float is
passed. If we would call that proc name again with an int or float
parameter, than one of the two existing procs would be used. But for
a different, still unused data type like float32 again a new proc
would be instantiated. In this way generics procs can lead to some
code bloat. Note that calling the generic function with a data type
like a character or a string would fail, as that types do not support
multiplication with itself.

A slightly different notation is available by so called or types:

func sqr(v: int or float): auto =
  var p: typeof(v)
  p = v * v
  return p

echo sqr(2)
echo sqr(3.1415)

Here we have limited the parameter types to the int type or the float
type. We could have defined also a custom type first, like type MyNum
= int or float and use that type for the type of our sqr() proc.
These or types are also called type classes. Instead of keyword or
the | character can be used for defining type classes. Again the
compiler would instantiate two separate functions for the both data
types. As we had not the symbolic type T available here, we have used
the keyword auto as return type, and for the type of variable p we
used the macro typeof(). The type auto for the return type works as
long as the function returns a well defined type. Note that we can
not decide at runtime what a type the function should return, so a
construct like if cond: return 2 else: return 3.1415 would not work,
at least not when the values are variables of different type. For the
literal value it may work, as the compiler may be smart and guess
that we want to return the float literal 2.0.

A bit care is needed when we define procs for mutable or types:

# proc t(s: var seq[uint8] | var seq[char]) =
proc t(s: var (seq[uint8] | seq[char])) =

Here we try to define a proc called t which should accept a mutable 
seq[uint8] or a mutable seq[char] as parameter. While the first line
compiles fine, the seq[char] would be immutable. The correct notation
is shown in the second line. This behavior was labeled "won't fix" in
github issue tracker, so we have to remember this case, see https://
github.com/nim-lang/Nim/issues/15063#issue-665553657.

Let us assume that you want to define a proc that accepts two numbers
of type int or float and that returns a float. You may write it in
one of this ways:

proc sqrsum(x, y: int | float): float =
  (x * x).float + (y * y).float

proc sqrsum2[T](x, y: T): float =
  (x * x).float + (y * y).float

proc sqrsum3[T1, T2](x: T1; y: T2): float =
  (x * x).float + (y * y).float

var i: int = 2
var x:float = 3.0

echo sqrsum(i, x)
#echo sqrsum2(i, x)
echo sqrsum2(x, 2)
#echo sqrsum2(2, x)
echo sqrsum3(i, x)

The commented out lines would give you a compiler error. The reason
for this is that the proc sqrsum2[T] defines a generic proc, but the
compiler enforces that both parameters have the same type. The
expression sqrsum2(x, 2) compiles fine, as due to the first parameter
x the compiler instantiates a proc for a float parameter type, and
then converts the second parameter, which is an integer literal, to 
float automatically. This automatic conversion is only done for
literal numbers, not for variables. The expression sqrsum2(2, x) does
not compile, as due to the first parameter, which is an integer
literal, a proc for integer parameters is instantiated, and the
second x parameter of [.type]float type is not compatible with the
instantiated proc.

Generics can become a bit complicated, as we may use multiple
different generic types for different proc parameters. And we can use
generics also for object types, we may for example create lists like
we did for our names list that work not only for strings, but that
can work with other data types like numbers or sequences in a very
similar way. We may explain that in more detail later.

Example for the use of Generics

Generics are used a lot in Nim's standard library. Most container
types like sequences or tables accept generic types, and generic
procedures like sort() are provided which can easily sort arbitrary
data types and objects. We have only to provide a cmp() proc for our
user defined data types which sort() can call to compare the values
during the sorting process.

We will demonstrate the use of generics for library modules with a
few tiny examples: Assume we create a library which should be able to
store and process arbitrary data types. The stored values may have
well defined relations, which enables ordering or much more
complicated spatial relations. Triangulation of spatial data points
or grouping the data in structures like RTrees for fast point
location as well as geometric processing with algorithm like finding
the convex hull are some examples. To make our example simple and
compact, we define a generic container type which can store only two
values of arbitrary data type. The container allows to sort the
elements by size. The following code example defines a generic
container called MyGenericContainer, a proc to add() data objects
into the container instance and a sortBySize() proc to sort the two
elements:

type
  MyGenericContainer[T] = object
    storage: array[2, T]

proc add[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

# sort by direct field access
proc sortBySize[T](c: var MyGenericContainer[T]) =
  if c.storage[0].size > c.storage[1].size:
    swap(c.storage[0], c.storage[1])

# a simple stringify proc for our container data type
proc `$`[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])

type
  TestObj1 = object
    name: string
    size: int

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", size: 162)
  var b = TestObj1(name: "Bob", size: 184)

  add(c, b, a)
  echo c
  c.sortBySize
  echo c

main()

The sortBySize() proc of the above examples accesses the size field
of our data objects directly, so we can use the container for
arbitrary data types as long as the data types have a size field and
as long as a > proc is defined for the data type of the size field.
In the above example we have defined a $ procedure to convert
instances of our container to a string, which allows us to call the 
echo() function on it. The output of our program looks like

(name: "Bob", size: 184), (name: "Alice", size: 162)
(name: "Alice", size: 162), (name: "Bob", size: 184)

We can avoid the restriction of a matching field name when we provide
getter and setter procedures which the library procs can use to
access the important fields:

type
  MyGenericContainer[T] = object
    storage: array[2, T]

proc add[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

proc sortBySize[T](c: var MyGenericContainer[T]) =
  if c.storage[0].size > c.storage[1].size:
    swap(c.storage[0], c.storage[1])

proc `$`[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])

type
  TestObj1 = object # arbitrary field names
    name: string
    length: int

# this getter proc enables sorting
proc size(t: TestObj1): int =
  t.length

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", length: 162)
  var b = TestObj1(name: "Bob", length: 184)

  add(c, b, a)
  echo c
  c.sortBySize
  echo c

main()

In the example above our TestObj1 data type has no field with a name
matching for the sortBySize() proc, but we define a size() proc for
our data type which that library function can use. This solution is
more flexible, and when we add the inline pragma to the used size() 
proc or when we compile with link time optimization (LTO) enabled,
then the overhead should be negligible. The two examples above are
located in a single file each, but of course for practical use we
would use separate modules for the library and the application part
as in

#module t3.nim
type
  MyGenericContainer*[T] = object
    storage: array[2, T]

proc add*[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

proc sortBySize*[T](c: var MyGenericContainer[T]) =
  if c.storage[0].size > c.storage[1].size:
    swap(c.storage[0], c.storage[1])

proc `$`*[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])

import t3

type
  TestObj1 = object # arbitrary field names
    name: string
    length: int

proc size(t: TestObj1): int =
  t.length

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", length: 162)
  var b = TestObj1(name: "Bob", length: 184)

  add(c, b, a)
  echo c
  c.sortBySize
  echo c

main()

The example with direct field access would look for different modules
like this:

# module t4.nim
type
  MyGenericContainer*[T] = object
    storage: array[2, T]

proc add*[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

proc sortBySize*[T](c: var MyGenericContainer[T]) =
  if c.storage[0].size > c.storage[1].size:
    swap(c.storage[0], c.storage[1])

proc `$`*[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])

import t4

type
  TestObj1 = object
    name: string
    size: int

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", size: 162)
  var b = TestObj1(name: "Bob", size: 184)

  add(c, b, a)
  echo c
  c.sortBySize
  echo c

main()

You may wonder why we do not have to export the size field of our 
TestObj1 (or maybe the object itself also) as it is used from code
defined in a different module. The reason why we do not need export
markers is that the sortBySize() is defined in the library module,
but as it is a generic procedure, it is instantiated and executed in
the application module. For the same reason we had not to export the 
size() getter procedure before.

Finally one more way to use generic library modules is by passing
procedure variables to the library functions. The passed in
procedures may provide access to properties or attributes of the
stored objects, or they may offer relations between the objects. The
later is often used for sorting purposes:

# module tx.nim
type
  MyGenericContainer*[T] = object
    storage: array[2, T]

proc add*[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

proc sortBy*[T](c: var MyGenericContainer[T]; smaller: proc(a, b: T): bool) =
  if smaller(c.storage[1], c.storage[0]):
    swap(c.storage[0], c.storage[1])

proc `$`*[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])

import tx

type
  TestObj1 = object
    name: string
    size: int

proc smaller(a, b: TestObj1): bool =
  a.size < b.size

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", size: 162)
  var b = TestObj1(name: "Bob", size: 184)

  add(c, b, a)
  echo c
  c.sortBy(smaller)
  echo c

main()

Here we have modified the sort() proc of our library module in a way
that it takes an additional procedure parameter. In this case we use
a procedure signature that takes two object instances and returns a
boolean value indicating if the first parameter is smaller than the
second. In our application module we define a matching procedure and
pass that one to the sortBy() procedure. Again we get the desired
sorted output:

(name: "Bob", size: 184), (name: "Alice", size: 162)
(name: "Alice", size: 162), (name: "Bob", size: 184)

This last method is used often in Nim's standard library, e.g. for
sorting sequences with custom objects. Unfortunately this way can
introduce some performance regression, as the procedure variable has
to be passed to the called proc and so inlining of that passed proc
is not possible for the compiler. ^[32]

Method Call Syntax

A useful coding style introduced by OPP languages is the method call
syntax, which was initially used in OOP programming style for
objects, and later applied by languages like Ruby to all data types.
Ruby in some way regards all data as objects.

Method call syntax means, that for example for a variable s of data
type string we do write s.add(c) instead of add(s, c). Or for an
integer variable i we may write i.abs instead of abs(i). That is we
put the first parameter of the proc parameter list in front of the 
proc name, and separate that parameter from the proc name by a
period. The compiler regards both notation as equivalent. The
advantage of the method call syntax is that we may save a character
and that it is more clear with what "object" we are working, as it
stands in front of the expression.

Most OOP languages allows that notation only for a class, for example
the string class may declare all possible operations that can be done
with strings, and the method call syntax is used for that operations.
One problem is, that it can be difficult to add more operations which
can be used in that style, as often all that operations are defined
in the class scope. Ruby fixed that restriction by allowing so called
reopen of classes, that is user can later add more operations.

Nim simple allows that notation generally, as did the D language, but
D used the term Uniform Function Call Syntax (UFCS) for it.

Procedure Variables

Procedures and functions are not always fully static entities. We can
assign procedures and functions to variables, and we can pass whole
procedures or functions as parameters to other procedures or
functions. And functions can even generate and return new functions.
Let us investigate how procedure variables work:

var
  p: proc(i: int): int

proc p1(i: int): int =
  i + i

proc p2(i: int): int =
  i * i

p = p1
echo p(7)
p = p2
echo p(7)

The output of the two echo statements should be 14 and 49 -- we called
in both cases the same proc variable with the same parameter, but the
proc variable p was in the first call an alias for p1, and in the
second call an alias for p2. Note, when we assign a proc to a proc
variable we do only write the name of that proc, there is no ()
involved. That is because we assign that proc to the proc variable,
but we do not call the proc in this case. Of course, when we assign a
proc to a proc variable then the proc signatures have to match, that
is the parameter list and the result have to be compatible.

Now we use a function as a proc argument.

type
  EchoProc = proc (x: float)

proc t(ep: EchoProc; x: float) =
  echo "The value is"
  ep(x)

proc ep1(x: float) =
  echo "==> ", x

proc ep2(x: float) =
  echo x

t(ep1, 3.1415)
t(ep2, 3.1415)

A common use case for a function as a proc parameter is sorting. We
can use the same sort procedure for different data types when we
provide a cmp() proc that can compare that data type.

from algorithm import sort

proc cmp(a, b: int): int =
  if a < b:
    -1
  elif a == b:
    0
  else:
    1

proc main =
  var a = [2, 3, 1]
  a.sort(cmp)
  for i in a:
    echo i

main()

The sort() procedure is provided by the algorithm module. The sort() 
proc accepts an array or a sequence, and a cmp() proc that gets two
parameters of the same type as the elements in the passed array, and
that returns -1, 0, or 1 as the result of the comparison. We could
easily sort other data types like strings or our custom objects by an
arbitrary key, as long as we can provide a matching cmp() proc. For
the cmp() proc it is important that it returns a well defined result
based on the input, and when both parameters are equal it should
really return 0. If you would exchange the return values 1 and -1 in
the cmp() proc above, you would invert the sort order.

Nested Procedures and Closures

While in C all functions must be defined in top level scope and
nesting of functions is not allowed, in Nim procedures can contain
other procedures. A special case occurs when the sub-procedures do
access variables of the outer scope. In this case the sub-procedure
is called a closure:

proc digitScanner(s: string) =

  var pos = 0
  proc nextDigit: char =
    while pos < s.len and s[pos] notin {'0' .. '9'}:
      inc(pos)
    if pos == s.len:
      return '\x0'
    result = s[pos]
    inc(pos)

  var c: char
  while true:
    c = nextDigit()
    if c == '\x0':
      break
    stdout.write(c)
  stdout.write('\n')

digitScanner("ad5f2eo73q9st")

When you run this program the output should be

52739

This program is not that easy, but when you think about it a bit you
should be able to understand it. The task is to extract from a string
all the digits and to ignore the other characters.

To get the digits, we use a local procedure that uses the pos
variable of the enclosing procedure, and also access the parameter s
of the enclosing procedure. The closure nextDigit() checks if the
position in the string is still valid, that is if it is still smaller
than the length of the string, and also checks if the current
character is a digit. The first check uses the standard procedure len
() which return the length of a passed string parameter, that is how
many characters the string contains. We have used the method call
syntax here instead of using the ordinary proc call len(s). The next
check test if the current character is not a decimal digit. For that
test we could use a series of compares like if c == '0' or c == '1'
or ...  or c == '9'. But to make such tests easier and faster, Nim
offers one more data type, the set type. And the notin operator tests
if an value is not contained in a set constant. An important point
for the expression after the while statement is, that it is processed
from left to right. That fact is here important, because we have
first to check if pos is still a valid position, before we can use
the subscript operator [] to access the current character and test if
it is not contained in the set. If the check for the valid position
would not come first, then we may access an invalid position in the
string and we would get a runtime range error.

While the position is still valid but the current character is not a
digit we increase the position. The while loop can end by two
conditions: Either the current character is a digit, or we have
reached the end of the string and we have to stop. For the last case
we use a special stop mark, we return a special character which we
have entered in escape notation as '\x0'. That is a very special
character, that is used in C to mark the end of strings. It is the
first character in the ASCII table and has the decimal value 0. We
said earlier that characters are encoded in 8 bit and correspond to
the unsigned integer numbers 0 up to 255. '\x0' is just a special
notation for the first character which corresponds to integer value 0
. Well, when the end of the string is reached then we return that
character. Otherwise we return the current character. Remember, from
the while condition we know that the string end is reached, or
current character is a digit. As we tested for the string end before,
we can only have the case that the current character is a digit now.
But can we return that character immediately now? If we would, s[pos]
would be a digit, and we would get exactly the same character for the
next proc call! So we have to move to the next character by
increasing pos before we return that character. For this the
pre-declared result variable is useful. We assign the current
character to the result variable, and then increase pos. As the last
statement in our proc is not a expression but a plain inc()
statement, the content of the result variable is returned. The other 
while loop in the outer procedure is very simple, we just call the
closure in the body of the while loop and terminate the loop when we
get the special Null character.

And finally an example where one proc returns another procedure:

proc addN(n: int): auto = (proc(x: int): int = x + n)

let add2 = addN(2)
echo add2(7)

The output of echo() would be 9 in this case. This construct is
sometimes named currying.

Anonymous Procedures

In the section Module sequtils in part III of the book we will
introduce a few functions which are often used in the functional
programming style like map() or filter(). These functions get
procedures as arguments that determine how container data types are
converted. We can pass a regular named procedure as second argument
to procs like map() and filter, or in simple cases we can just pass
an anonymous proc or use the = operator provided by the sugar module:

import sequtils, sugar

proc primeFilter(x: int): bool =
  x in {3, 5, 7, 13}

var s = (0 .. 9).toSeq # @[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

echo s.filter(primeFilter) # @[3, 5, 7]
echo s.filter(proc(x: int): bool = (x and 1) == 0) # @[0, 2, 4, 6, 8]

echo s.map(proc(x: int): int = x * x) # always @[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
echo s.map(x => x * x) # from sugar module

Here we use the toSeq() template to create our initial sequence with
numbers 0 up to 9 -- just to have not to type all the number in, we
will explain templates soon. Then we apply the filter() proc to that
sequence. The filter() proc expects as second argument a function
with an argument of the seq's base type returning a boolean value. We
can pass the named function primeFilter(), or we can just pass an
anonymous proc explicitly.

In the last two lines of our example, we use the map() function to
convert the data of our sequence. Map() expects as second argument a 
proc with a parameter of the seq's base type returning a result of
the same type. In the line before the last one, we specify an
anonymous proc as parameter, while in the last line we use the =
operator from the sugar module to just specify the actual conversion.

Compile time proc execution

When a function is called only with constant arguments, then the
compiler can execute it already at compile time:

func genSep(l: int): string =
  debugecho "Generating separator string"
  for i in 1 .. l:
    result.add('=')

const Sep = genSep(80) # function is executed at compile time

echo Sep

Here we used a function called genSep() to create a string constant
at compile time. When we compile above program, we get the message
"Generating separator string". As that proc is not executed at
program runtime, it is not included in the final executable program.
Here we had to use the debugEcho() proc instead of the ordinary echo
(), because echo() is not really a pure function, and the compiler
would complain when we use echo() in a pure function. DebugEcho() is
not really pure as well, but the compiler ignores that fact, which is
OK for debugging purposes. We could even make gesSep() a plain proc
and then use echo(), the compiler would not complain. But it would
complain, if for instance we would access global variables from
inside the genSep() proc.

Inlining Procedures

Calling procedures and functions is always some overhead -- proc
parameters may have to put on the stack or loaded into CPU registers,
some CPU or FPU registers may have to be saved, the stack pointer and
the program counter have to be updated and finally the instruction
cache has to be filled with new instructions.

So for tiny procs actually calling the proc may take more time than
processing the actual code inside of the proc. To avoid this
additional effort, procedures and functionss can be inlined. The
compiler may do this automatically for us, but we can support it by
applying the {.inline.} prama to tiny procs.^[33] For inlined procs
the code is just inserted directly at the callsite. This may increase
the total executable size, when the proc is used often. So we should
use the inline pragma with some care. Another option is to just
compile the whole program with link time optimization passing the
option -d:lto to the compiler, that way the C backend can
automatically inline all proc code, even procs from imported modules.
One more option is to use templates instead of tiny procs -- templates
do always a plain code substitution, so templates may behave very
similar to inline procs. We will discuss templates later. The
following example show how we can apply the inline pragma to
procedures and functionss:

proc max(a, b: int): int {.inline.} =
  if a < b: b else: a

Note that functions from shared libraries can not be in lined, so
calling external C functions, directly or indirectly, may be slower
than expected.

Recursion

Procedures and functions can call them self in a repetive manner,
which is called recursion. Obviously there must exist some condition
that finally stops the recursion, without the proc would call itself
again and again, for each call some data would have to be stored on
the stack, at least the proc return address, so finally the stack
would overflow and the program would crash. Generally recursion
should be used only for cases where it really helps to simplify the
algorithm. In part V of the book, in the section about various
sorting algorithm, we will discover some useful applications for
recursion. In most cases a plain iterative algorithm is faster than a
recursive one, because all the overhead with many proc calls is
avoided for plain iterative solutions. But sometimes recursive
algorithm are easier to understand, or programming an iterative
solution may be really complicated.

As one of the most simple algorithm we will present here the
recursive fac() function:

proc fac(i: int): int =
  if i < 2:
    1
  else:
    i * fac(i - 1)

That function should terminate, as we only call itself again with a
decreased argument. Of course, using recursion here is not really
smart, it should be easy for you to convert the proc into an
iterative solution without recursion. Note that recursive procs can
never be inlined!

Converters

Nim's converters are a special variant of functions that are called
automatically by the compiler when argument types do not match.

converter myIntToBool(i: int): bool =
  if i == 0:
    false
  else:
    true

proc processBool(b: bool) =
  if b:
    echo "true"
  else:
    echo "false"

var i = 7
processBool(i)
if i:
  echo "true"
else:
  echo "false"

With above converter we can pass an integer to a proc that expects a
boolean parameter, and we can even use an integer as a logical
expression in an if condition in the same way as it is done in C
language. Converters do work only in a direct way, that is automatic
chaining is not supported: If we have one converter from character to
integer and one from int to boolean, that does not mean that we can
pass a character to a proc that expects a boolean. We would have to
declare one more converter that directly converts character to
boolean.

Whenever we do consider using converters we should think
twice -- converters may be confusing, may have some strange effects
and may increase compile time.

Maybe you wondered why we wrote above converter in such a verbose
way? Well it was done intentionally, but you are right of course, we
can write it just as

converter myIntToBool(i: int): bool =
  i != 0

Object-Orientated Programming and Inheritance

Object-Orientated Programming and Inheritance became very popular in
the early nineties of the last century. Java is a prominent
representative of that programming paradigm, but most languages
created in the nineties of the last century support it, like C++,
Ruby and Python.

The idea of OOP is that objects and procedures working on that
objects are grouped to classes, and that classes can be extended with
additional data fields and with additional procedures. In OOP
procedures and function are often called methods and data fields are
called members. Sometimes the members are completely hidden and are
accessed only by so called getter and setter methods. That is called
encapsulation. Encapsulation allows hiding implementation details, so
that that details may change when necessary, without that the change
of internal details become visible to users of that class, so that
the users can use the class without noticing the change. Getters and
setters also help hiding internal details and they ensure that the
class is always in a consistent and valid state.

An important property of OOP is dynamic dispatch: When we create
various sub-classes of a common parent class, and we have defined
methods for all the sub-classes, then we can have collections of
instances of different sub-classes, and the compiler can
automatically ensure that always the matching method for each
instance is called.

A classical example is a drawing program, where we have different
geometrical shapes like rectangle, circle and many more. All the
geometrical objects are stored is some form of list, and when we want
to draw all of them on the screen then we have to call only an
unspecific draw() method, and the compiler ensures that for each
shape the matching draw method is called. In Nim that may look like

type
  Shape = ref object of RootRef

  Rectangle = ref object of Shape
    x, y, width, height: float

  Circle = ref object of Shape
    x, y, radius: float

  LineSegment = ref object of Shape
    x1, y1, x2, y2: float

method draw(s: Shape) {.base.} =
  # override this base method
  quit "to override!"

method draw(r: Rectangle) =
  echo "drawing a rectangle"

method draw(r: Circle) =
  echo "drawing a circle"

method draw(r: LineSegment) =
  echo "drawing a line segment"

proc main =
  var l: seq[Shape]
  l.add(Rectangle(x: 0, y: 0, width: 100, height: 50))
  l.add(Circle(x: 60, y: 20, radius: 50))
  l.add(LineSegment(x1: 20, y1: 20, x2: 50, y2: 50))

  for el in l:
    draw(el)

main()

The output of that program is

drawing a rectangle
drawing a circle
drawing a line segment

So we can have a sequence of the base type, add various sub-types and
then iterate over the list to draw all the various sub-types. Of
course in the same way we could do many more task like moving,
rotating or storing all the objects in one call. The compiler does
the right dynamic dispatching for us, we have just to provide all
necessary methods. The need of the base method seems to be a bit
strange, some other OOP languages do not need that. The base method
is marked by a {.base.} pragma, we will discuss the purpose of
pragmas later. In the example we have used only one level of
sub-classing, but of course we can use many levels, for example we
can again subclass the Circle by a FilledCircle with a color field.

The OOP coding style can be very convenient for some tasks, one
important use case could be graphical user interfaces where the
graphical elements like labels, buttons, frames built in natural way
a hierarchical structure. Another typical use case are drawing
applications with code similar to our basic example.

Note that the OOP style only works with ref objects, but not with
value objects. The obvious reason is that we can have collections of
different sub-types stored in arrays or sequences only for ref
objects, as in arrays and sequences all element types have to have
equal size. For references that is the case, as references are
basically pointers. But different value types would have different
size. Linked lists would be no better solution, as again we can not
built lists with value objects.

For maximum performance OOP code with ref objects is generally not
optimal, as the dispatching itself needs some time, and as the ref
objects are not contained in a single block of memory, but are
distributed in the whole RAM, which is not cache friendly.

Other Builtin Data Types

Tuple Types

Tuples are heterogeneous container types similar to the struct type
in C. As Nim's object type creates no overhead as long as we use no
inheritance and so also directly corresponds to the C struct type,
tuples are very similar to Nim's objects. The biggest advantage of
tuples is, that we can create anonymous tuples, and that Nim supports
the automatic unpacking of tuple variables into ordinary unstructured
variables.

Compared to objects, tuples do support no inheritance at all, all the
tuple fields are always visible, and different tuple types are
regarded as identical, when all the field names and field data types
match. Remember that two different object types are always distinct
in Nim, even when the actual type definition looks identical.

We can define tuple types in the same way as we define objects, or we
can use the tuple[] constructor. Additional we can define anonymous
tuples just by enclosing its field types in round brackets. The
fields of tuple types can be accessed by field names as we do it for
objects, or we can access the fields with constant indices starting
at zero.

type
  Move = tuple # the object definition syntax
    from: int
    to: int
    check: bool

type Move = tuple[from: int, to: int, check: bool] # equivalent tuple constructor syntax

proc findBestNextMove(): (dest: int; check: bool) =
  ...

let (dst, check) = findBestNextMove()

In the code example above we show two equivalent ways to define a
tuple type, but actually we do not use that type at all but return an
anonymous tuple from our proc, that is a pair of an int and a bool.

Using automatic tuple unpacking and type inference our dst and check
variables gets the data types int and bool.

Tuples are also useful when a function has to return a value and also
an error state, or if it may not be able to return something at all
in a special case. For reference types we could return nil then, but
for results of value type like int or float we may not have a well
defined error indicating constant, so we can return a tuple with an
additional bool indicating success or error. But of course we could
use exceptions instead, or we could use Nim's option type instead. We
will learn more about that later.

Here are two examples which uses a tuple as a proc parameter:

proc p1(x: tuple[i: int, j: int]): int =
  x.i + x.j

echo p1((7, 7))

proc p2(x: (int, int)): int =
  x[0] + x[1]

echo p2((7, 7))

Proc p1() creates a tuple type using the tuple constructor syntax
with named fields, so in the proc body we can access the fields by
its names, while proc p2() uses an anonymous tuple, and so has to
access the fields by constant indices. Both procs are called with an
anonymous tuple parameter.

Object Variants

Nim's object variants, sometimes also called sum types or abstract
data types (ADT), are an advanced and type save variant of the union
type known from C. The basic idea is that we may use value types that
may store similar, but not identical data. Untyped languages like
Ruby or Python allow that of course, and we can do it in Nim with ref
types and inheritance too, as we showed in a previous section with
our Shape base type and various geometric shapes. We could store that
ref types in arrays or sequences or linked list and use dynamic
dispatch for processing the various sub-types. That is convenient but
gives not maximum performance due to dynamic dispatch at runtime and
due to bad cache use. So we may like to have a value type with
different content, so that we can store all the value types in a seq
and all entities reside in a compact block of memory for good cache
use.

type
  ShapeKind = enum
    line, rect, circ

  Shape = object
    visible: bool
    case kind: ShapeKind
    of line:
      x1, y1, x2, y2: float
    of rect:
      x, y, width, height: float
    of circ:
      x0, y0, radius: float

proc draw(el: Shape) =
  if el.kind == line:
    echo "process line segment"
  elif el.kind == rect:
    echo "process rectangle"
  elif el.kind == circ:
    echo "process circle"
  else:
    echo "unknown shape"

var
  s: seq[Shape]
s.add(Shape(kind: circ, x0: 0, y0:0, radius: 100, visible: true))
for el in s:
  draw(el)

Objects variants can have common fields like the boolean state
visible above, but the other fields are not allowed to use the same
names, so we had to use x0 and y0 for the names of the center
coordinates in the circle variant.

As you can see we can store all the different object variants as
value objects in a sequence and iterate over it. Note that object
variants may waste some storage, as all variants are silently
enlarged to have the exact same size, so that all variant types can
be stored in an array or sequences and can be passed as proc
parameters in the same way to the same proc.

Iterators

In section For Loops and Iterators we used a for loop to iterate over
the individual characters of a string. For loops are useful for
various iteration purposes, e.g. to iterate over container types like
strings, arrays, and sequences, or over a numeric range, and other
countable entities. We could do the same with a while loop, but using
a for loop is often more convenient and less error prone -- we do not
have to care for increasing a loop variable and for the stop
condition.

Nim's for loops are built on iterators, that is whenever a for loop
is executed an iterator is used under the hood. Some iterators are
used explicitly in for loops, e.g. countup() of Nim's standard
library, others like items() or pairs() are executed implicitly when
no explicit iterator name is specified.

The creation and use of iterators is very easy in Nim. Before we will
discuss all the details and some restrictions of iterators, and the
important differences between inline and closure iterators, we will
give a small example:

We have already used some of Nim's standard iterators to iterate over
the characters of a string or the content of a sequence.

In an earlier section of the book we declared a procedure that
extracts all the decimal digits from a string. We can do the same
with an iterator:

iterator decDigits(s: string): char =
  var pos = 0
  while pos < s.len:
    if s[pos] in {'0' .. '9'}:
      yield(s[pos])
    inc(pos)

for d in decDigits("df4j6dr78sd31tz"):
  stdout.write(d)
stdout.write('\n')

The definition of an iterator is very similar to the definition of a 
proc or function, but while a function returns a result only once to
the caller, an iterator uses the yield statement to give data back to
the call site multiple times, instead of returning just once.

Whenever a yield statement is reached in the body of the iterator,
the yielded data is bound to the for loop variable(s), the body of
the for loop is executed, and at the end of the for loop body control
returns to the iterator, that is execution continues direct after the
yield statement. The iterator's local variables and execution state
are automatically saved between calls. The iteration process
continues until the end of the body of the iterator declaration is
reached and the iterator terminates.

Iterators are used in for loops to iterate over containers, ranges or
other data. After the for keyword we specify one or more arbitrary
variable names, which we then can use in the body of the for loop to
access the yielded value(s). The data type of this iteration variable
(s) is inferred from the iterators return type, and its scope is
limited to the body of the for loop.

Nim's standard library defines for container types like strings, 
array and sequences iterators named items() and pairs() -- items() is
the default name when a for loop with only one variable is used, and 
pairs() is the default name when two variables are used, e.g. the
index position and the character when iterating over a string.

In Nim's standard library you may find items() and pairs() iterators
like these two:

iterator items(a: string): char =
  var i = 0
  while i < len(a):
    yield a[i]
    inc(i)

iterator pairs(a: string): tuple[key: int, val: char] =
  var i = 0
  while i < len(a):
    yield (i, a[i])
    inc(i)

var s = "Nim is nice."
for c in items(s):
  stdout.write(c, '*')
echo ""
for i, c in pairs(s):
  echo i, ": ", c

In the example above we specified the iterator names items() and 
pairs() explicitly in the for statement, but as these names are the
defaults, we can just write for c in s: and for i, c in s:.

The two iterators in the example code from above use as argument a
value type and return the single characters as value type. This way
we can not modify the string content. When we intent to modify the
content of a container by use of an iterator, we have to pass the
container as var parameter, and return the elements as var also. By
convention for iterating over mutable containers the iterator names 
mitems() and mpairs() are used, where the leading m stands for
mutable. We have to specify these names explicitly:

iterator mitems(a: var string): var char =
  var i = 0
  while i < len(a):
    yield a[i]
    inc(i)

iterator mpairs(a: var string): tuple[key: int, val: var char] =
  var i = 0
  while i < len(a):
    yield (i, a[i])
    inc(i)

from strutils import toLowerAscii
var s = "NIM"
for i, c in mpairs(s):
  if i > 0:
    c = toLowerAscii(c)
echo s # Nim

Whenever we iterate over a container, we should not delete, insert or
append elements to the container, as that may confuse the loop inside
of the iterator body. Iterators of Nim's standard library check the
length of the container and generate an exception when the length
changes during the iteration.

Nim differentiate between inline and closure iterators. When a for
loop uses an inline iterator, then the actual iterator loop is
inlined in the for loop body in a way that for each yield statement
in the iterator body the body of the for loop is executed. Actually
the for c in items(s): stdout.write(c, '*') in our example from above
is rewritten by the compiler into a code block like

var i = 0
while i < len(a):
  var c = a[i]
  echo c, '*'
  inc(i)

That is, the body of the for loop is inlined into the loop of the 
iterator.

This results in very fast code with no overhead, but similar to the
use of templates, this increases the total code size. Actually, when
the iterator should use multiple yield statements, then the code of
the body of the for loop is inserted for each yield statement.

Inline iterators are currently the default iterator type, so the 
iterators of the examples above are all inline iterators.

Closure iterators behave more like procedures, the iterator is
actually invoked, which costs some performance. We can use all the 
iterators of the examples from above as closure iterators by applying
the closure pragma as in iterator items(a: string): char {.closure.}
=.

Closure iterators behaves like objects, we can assign instances of
inline iterators to variables, and then call the instances
explicitly:

iterator myCounter(a, b: int): int {.closure.} =
  var i = a
  while i < b:
    yield i
    inc(i)

for x in myCounter(3, 5): # ordinary use of the operator
  echo x

echo "---"
var counter = myCounter # use of an iterator instance
while true:
  echo counter(5, 7)
  if counter.finished:
    break

which gives us this output:

3
4
---
5
6
0

Here we have used the finished() function to check if the iterator is
done.

Actually finished() returns true only, when the iterator has already
failed to yield a valid value, and not already when the last valid
value was yielded. That is, why we get in the example above as last
value the invalid value zero.

We can avoid this behaviour, when we rewrite the loop as

var counter2 = myCounter
while true:
  let v = counter2(5, 7)
  if counter2.finished:
    break # v is invalid
  echo v

Closure iterators are resumable functions and so one has to provide
the arguments to every call. To get around this limitation one can
capture parameters of an outer factory proc:^[34]

proc mycount(a, b: int): iterator (): int =
  result = iterator (): int =
    var i = a
    while i < b:
      yield i
      inc(i)

var c1 = mycount(5, 7)
for i in c1():
  echo i

echo "---"

var c2 = mycount(2, 5)
while true:
  let v = c2()
  if c2.finished:
    break # v is invalid
  echo v

In this example from the compiler manual the proc mycount() captures
the bound for the counter. When we compile and run the code above we
get:

5
6
---
2
3
4

At the end of this section we will list some properties of iterators:
Iterators have their own name space, so we can freely use for procs
and iterators the same names. Iterators have no predefined result
variable, and do not support recursion. Inline iterators can be used
only inside of for loops, and can not be forward declared, because
the compiler must be able to inline an iterator. (This restriction
will be gone in a future version of the compiler.) And iterators do
not support recursion. Closure iterator are not supported by the JS
backend, and cannot be executed at compile time. Inline iterators are
second class citizens; They can be passed as parameters only to other
inlining code facilities like templates, macros, and other inline
iterators. In contrast to that, a closure iterator can be passed
around more freely.

Templates

Nim templates are very different from C++ templates! In C++ templates
are used for generic programming -- a style of computer programming in
which algorithms are written in terms of types to-be-specified-later
that are then instantiated when needed for specific types provided as
parameters.^[35] This is just called generics in Nim and other
programming languages, we learned about Nim's generics earlier in
this book.

Nim templates are a simple, parameterized code substitution
mechanism, and are used in a similar way as procedures. The syntax to
invoke a template is the same as calling a procedure. But while
procedures built a single block of code that is then called multiple
times, templates work more like C macros as a (textual) code
substitution. Wherever we invoke a template the template source code
is inserted at the call site. In this way Nim templates have indeed
some similarity to C macros. But while C macros are executed by the C
pre-processor and can do only plain source text substitutions, Nim 
templates operates on Nim's abstract syntax trees, are processed in
the semantic pass of the compiler, integrate well with the rest of
the language, and share none of C's pre-processor macros flaws.

In some way Nim templates are a simplified application of Nim's
powerful macro and meta-programming system which we will discuss in
detail in part VI of the book.

In C we could use the "#define" pre-processur directive to define two
simple C macros.

#define PI 3.1416
#define SQR(x) (x)*(x)

The C pre-processor would then replace the symbol PI in the C source
code with the float literal 3.1416. And as the C pre-processor can
recognize some simple form of parameters, it would replace SQR(a + b)
with (a+b)*(a+b).

In Nim we would define a const for PI and use a generic proc or a 
template for SQR():

const PI = 3.1416
proc sqr1[T](x: T): T = x * x
template sqr2(x: typed): typed = x * x

Here the sqr2() template uses the special typed parameter, which
specifies that the parameter has a well defined type in the template
body, but that arbitrary data types are accepted. So sqr1() and sqr2
() would work for all numeric types and also for other data types for
which we have defined a operation. When there is no operator defined
for the passed data type, the compiler will give an error message.

Nim templates accept like procs all of Nim's ordinary data types, and
additional the abstract meta-types typed and untyped. The abstract
data types typed and untyped can be used only for the types of 
template and macro parameters, but not for parameters of procedures,
functions, iterators, or to define variables.

We will explain the differences between typed and untyped in detail
later in this section -- the short version of the explanation is, that
typed template parameters must have a well defined data type when we
pass it to the template, while untyped parameters can be passed as
still undefined symbolic name also.

So we can in principle replace each procedure or function definition
with a template. The important difference between procedures and 
templates is, that ordinary procs are instantiated only once, generic
procs may be instantiated for each data type with which they are
used, but templates are instantiated for each invocation of the 
template. The compiler creates for each defined proc some machine
code, which is executed whenever the proc is called. But for 
templates the compiler does some code substitution -- the source code
of the template is inserted where the template is invoked. This
avoids the need for an actual jump to a different machine code block
when a procedure is called, but increases the total code size for
each use of a template. So we would generally avoid templates that
contain a lot code and are used often.

For each ordinary proc one block of machine code instructions is
generated, and when the proc is called, program executions has to
jump to this block, and back when the proc execution is done. This
jumping involves some minimal overhead, which is noticeable for tiny 
procs called often. To avoid this overhead we may use a template
instead, or we may use inlined procs, which we discussed in the
previous section. Proc inlining can be done by the compiler
automatically when the proc is defined in the source code file where
it is used, or when we mark the proc with the inline pragma.
Additional, when we compile our program with -d:lto, the compiler can
inline all procedures and functions. Generally the compiler should
know well when inlining makes sense, so in most cases it makes not
much sense to just use templates instead of (small) procs just to
avoid the proc call overhead.

Templates can be used as some form of alias. Sometimes we have nested
data structures, and would like to have a shorter alias for the
access of fields:

type
  Point = object
    x, y: int

  Circle = object
    center: Point

template x(c: Circle): int = c.center.x

template `x=`(c: var Circle; v: int) = c.center.x = v

var a, b: Circle

a.center.x = 7
echo a.center.x

b.x = 7
echo b.x

The two templates simplify the access of field x, and as templates
are pure code substitution, the use of them costs no performance.
Since version 1.6 Nim has also the with macro, which can be also used
to save some typing. Note that in the second template we have called
the second int parameter v -- calling them x would give some trouble:

Error: in expression 'b.center.7': identifier expected, but found '7'

Nim's system module uses templates to define some operators like

template `!=` (a, b: untyped): untyped =
  not (a == b)

This way != is always the opposite of ==, so when we define the ==
operator for our own custom data types, != is available for free.

In some situations using templates instead of procs can avoid some
overhead. Let us investigate a log() template, that can print
messages to stdout when a global boolean constant is set to true:

const
  debug = true

template log(msg: string) =
  if debug: stdout.writeLine(msg)

var
  x = 4
log("x has the value: " & $x)

Here log() is called with the constructed argument ("x has the value:
" & $x) which implies a string concatenation operation at runtime. As
we use a template, the invocation of log("x has the value: " & $x) is
actually replaced by the compiler with code like

  if debug: stdout.writeLine("x has the value: " & $x)

So when debug is set to false, absolutely no code is generated. For
an ordinary, not inlined proc the situation is different, the
expensive string concatenation operation would always have to be
performed, but the log() proc would immediately return for debug ==
false. What exactly would happen when log() is an inlined proc may
depend on the actual used compiler backend.

Note that the delayed (lazy) parameter evaluation for template
parameters can have disadvantages: When we modify the log() template
like

template log(msg: string) =
  for i in 0 .. 2:
    stdout.writeLine(msg)

var x = 4
log("x has the value: " & $x)

the expensive string concatenation operation would be done in
principle three times in the template body.^[36] While for a proc the
already evaluated parameter would be passed. So when we access a
parameter multiple times inside of a template, it can make sense to
assign the parameter to a local variable and then use that variable
only.

Templates can inject entities defined in the template body into the
surrounding scope. By default variables defined in the template body
are not injected in the surrounding scope, but procs are:

template gen =
  var a: int
  proc maxx(a, b: int): int =
    if a > b: a else: b

gen()
echo maxx(2, 3)
# echo a

The call echo maxx(2, 3) compiles and works, while echo a complains
about an undefined symbol.

A very special property of templates and macros is, that we can pass
to them code blocks when we use untyped for the type of the last
parameter:

template withFile(f: untyped; filename: string; actions: untyped) =
  var f: File
  if open(f, filename, fmWrite):
      actions
      close(f)

withFile(myTextFile, "thisIsReallyNotAnExistingFileWithImportantContent.txt"):
  myTextFile.writeLine("line 1")
  myTextFile.writeLine("line 2")

The template withFile() from the above example has three
parameters -- a parameter f of untyped type, a filename of string
type, and as last parameter one more untyped parameter which we call
actions. For these last untyped actions parameter we can pass an
indented code block.

When we invoke the withFile() template, we pass the first two
parameters in the well know way by putting them in a parameter list
enclosed in round brackets. But instead of passing this way also the
final actions parameter, we put a colon after the parameter list, and
pass this way the following indented code block as last untyped
parameter. In the body of above template we have a open() call which
opens a file with the specified filename and the fmWrite mode, then
executes the passed code block, and finally closes the file. The
first parameter of our withFile() template has also a special
property: As we use untyped for the f parameter, we can pass the
still undefined symbol myTextFile to the template. In the template
body this symbol is used as variable name, and our two writeLine() 
proc calls can use it to refer to the file variable.

As Nim templates are hygienic, the instance of the file variable
created in the body of our template can be used by the passed code
block, but it actually exists only in the template, and pollutes not
the global name space of our program.

We can use templates to create new procs. An example is lifting procs
like math.sqrt() that accepts a scalar parameter and returns a scalar
value, to work with arrays and sequences. The following example is
taken from the official tut2 tutorial:

from std/math import sqrt

template liftScalarProc(fname) =
  proc fname[T](x: openarray[T]): auto =
    var temp: T
    type outType = typeof(fname(temp))
    result = newSeq[outType](x.len)
    for i in 0 .. x.high:
      result[i] = fname(x[i])

liftScalarProc(sqrt)   # make sqrt() work for sequences
echo sqrt(@[4.0, 16.0, 25.0, 36.0])   # => @[2.0, 4.0, 5.0, 6.0]

The template called liftScalarProc() creates a generic proc, that
accept as parameter an openArray[T] and returns a seq[T]. Well, we
should be able to understand the basic ideas used in that code, but
it is still fascinating that it really works.

Typed vs untyped parameters

Parameters passed to templates can have all the data types that we
can use for procs including special types like openarray and varargs,
and we can use as types additional the symbols untyped, typed or 
typedesc.

The typedesc type can be used to pass type information to the 
template, e.g. when we want to create a variable of a special data
type. The "meta-types" typed and untyped are used, when we want to
create some form of generic template, that can accept different data
types. Actually the distinction between typed and untyped parameters
is not that difficult and important for templates as it is for macros
, in most cases it is just clear if we need the typed or untyped
parameter type for a template, or both work fine. We discuss the
differences between typed and untyped in much more detail in part VI
of the book, when we discuss macros and meta-programming.

The following example demonstrates the use of the untyped and the 
typedesc parameter:

template declareInt(n: untyped) =
  var n: int

declareInt(i)
i = 3
echo i

template declareVar(n: untyped; t: typedesc) =
  var n: t

declareVar(x, float)
x = 3.0
echo x

As the parameter n is untyped, the compiler allows us to pass an
undefined symbol to the template. If we would change the parameter
type to typed the compiler would complain with a message like Error:
undeclared identifier: 'i'

For the second template called declareVar() we use an additional
parameter of typedesc type, so that the template can create for us a
variable of just the passed data type.

Citing the manual: "An untyped parameter means that symbol lookups
and type resolution is not performed before the expression is passed
to the template. This means that undeclared identifiers, for example,
can be passed to the template. A template where every parameter is 
untyped is called an immediate template. For historical reasons, 
templates can be explicitly annotated with an immediate pragma and
then these templates do not take part in overloading resolution and
the parameters' types are ignored by the compiler. Explicit immediate
templates are now deprecated. For historical reasons, stmt was an
alias for typed and expr was an alias for untyped, but they are
removed."

Passing a code block to a template

In the withFile() example above we showed that we can pass a block of
statements as the last argument to a template following the special :
syntax. To demonstrate the difference between code blocks of typed
and untyped data type we will cite the compiler manual, see https://
nim-lang.org/docs/manual.html#
templates-passing-a-code-block-to-a-template:

Usually, to pass a block of code to a template, the parameter that
accepts the block needs to be of type untyped. Because symbol lookups
are then delayed until template instantiation time:

template t(body: typed) =
  proc p = echo "hey"
  block:
    body

t:
  p()  # fails with 'undeclared identifier: p'

The above code fails with the error message that p is not declared.
The reason for this is that the p() body is type-checked before
getting passed to the body parameter and type checking in Nim implies
symbol lookups. The same code works with untyped as the passed body
is not required to be type-checked:

template t(body: untyped) =
  proc p = echo "hey"
  block:
    body

t:
  p()  # compiles

Passing operators to templates

One more use case for templates with untyped parameters is the
generation of math operations for custom data types. Let us assume
that we have created a custom Vector object, for which we have to
define addition and subtractions operations. Instead of writing code
for both cases, we can use a template and pass the actual math
operator as untyped parameter:

type
  Vector = object
    x, y, z: int

template genOp(op: untyped) =
  proc `op`(a, b: Vector): Vector =
    Vector(x: `op`(a.x, b.x), y: `op`(a.y, b.y), z: `op`(a.z, b.z))

genOp(`+`)
genOp(`-`)

echo `+`(2, 3) # 5

var p = Vector(x: 1, y: 1, z: 1)
var p2 = p + p
echo p2 # (x: 2, y: 2, z: 2)

This works, because for operators math like 1+2 can be written as +
(1, 2) and because we can pass such an operator as untyped parameter
to a template.

Advanced template use

For the advanced template stuff you should consult the compiler
manual.

This includes the symbol binding rules, identifier construction in 
templates, lookup rules for template parameters, hygiene in templates
, use of the inject pragma, and limitations of the method call
syntax.

All this is explained well in the compiler manual, so it makes no
sense to repeat it here. And it makes more sense to read it when you
actually have problems with the (default) behaviour of templates in
special situations.

Casts and Type Conversion

While we have various types of casts in C++, we have only one cast
and type conversions in Nim. In Nim cast just reinterpret the same
bit pattern for another data type. For example the boolean value 
false is internally encoded as a byte with all bits cleared, and true
is encoded as a byte with all bits but the least significant one
cleared. We could cast a bool to an int8 of same size and would get a
number with decimal value 0 or 1. Casting is not a real operation at
all, as nothing is really done. We watch the same bit pattern just
from a different perspective. But casting is dangerous, it violates
the safe type system of the language, and it can go very wrong: Can
we cast between float64 and int64? Well they have same size, and both
are numbers. We can cast, but the result would be far away from what
we may expect. While int64 has the well known and simple value
encoding, that is rightmost bit stands for 2^0, next bit for 2^1 and
so far, the encoding of floating point numbers is much more difficult
and has not such a simple scheme. In floats some bits represent the
so called mantissa and some bits represent the exponent. When we cast
we may again get a number, but the value is not easy predictable. We
have to be very carefully when we cast between types of different
size. Nim may permit that but we have to think what may really
happen. When we cast between a bool and an int64, in one direction 7
bytes have to be ignored, and in the other direction for 7 missing
bytes there is some padding necessary. We do a cast by writing after
the keyword cast in square brackets the desired type followed by
parenthesis enclosing the source variable:

var i: uint8 = cast[unint8](myBoolVar)

Totally different to casting is type conversion. We can convert
integers to floating point numbers without problems, for the
conversion we use the type like a proc call, that is int(myfloat) or 
float(myInt) -- of course we could use method call syntax like 
myInt.float instead. Type conversion is some effort for the CPU, but
most advanced CPUs should have fast instructions for basic
conversion.

Nim generally only allows type conversions that involve not too much
effort. So we should not expect var i ="1234"; echo i.int * 7 to be
available. Such a conversion is expensive, at runtime it costs many
CPU cycles as we would have to extract the digits, multiply with it
weight and sum them up. So for that operation functions like parseInt
() are available from the Nim standard library that accept a string
as argument and return an int. There exists different variants of 
parseInt(), one may raise exceptions for invalid input, the other may
return a boolean.

Bitwise Operations

All systems programming languages, and most other languages, have
support for bit manipulation operations, which includes querying and
setting individual bits of variables, and combining the bits of two
or more variables. As the CPU hardware supports these operations
directly, these operations are very efficient. In the C programming
language operators like &, |, <<, >>, ^, ~ are used for bit-wise and
and or operations, for shifting all the bits of a variable to the
left or to the right, and for the process of inverting all the bits
and for applying the exclusive-or operation on the bits of two
operands. Actually, for the right shift operation, we have to
distinct between a logical and an arithmetic shift: For a logical
shift the bit pattern is only moved right, and the leftmost bit is
always cleared. But for an arithmetic shift, the leftmost bit may
stay set when it was set before, indicating a negative number in case
of a numeric variable. In C the actual behaviour for a >> shift right
operation may be implementation dependent.

Nim prefers to use textual operators instead of cryptic symbols, so
the logical operators and, or and not have overloads to work on the
actual bit pattern of integer variables instead of on boolean values,
and for logical left and right shifts the operators are called shr
and shl. For shl shifted in bits from the right are always cleared,
while shr shifts in cleared bits from the left for unsigned
arguments, but preserved the leftmost set bit for signed arguments,
which corresponds to an arithmetic shift operation. The Nim standard
library provides also an ashr() function for arithmetic shifts, but
that one seems to be a legacy.

from strutils import toBin
var i = 1.int8 # 0b00000001
i = i shl 7 # 0b10000000
i = i shr 2 # 0b11100000 as sign is preserved
echo i.toBin(8)
var j: uint8 = 0b11111111
j = j shr 2 # 0b00111111, div 4 for unsigned int
echo j.int8.toBin(8)

The bit-wise operators and, or and not behave very similar to the
boolean ones, but the operation is performed for all the bit values
instead for two boolean operands. The shift operators require a right
hand operand specifying how many positions the bit pattern of the
integer variable on the left should be moved. As the shr operator
preserves the leftmost sign bit for each individual shift when
applied to an signed integer argument, we gets a value with the three
leftmost bits set in above example. For showing the bit pattern, we
used the toBin() function in above code, the second parameter
determines how many bits are actually printed. Remember, that for
unsigned numbers shl by one position is a multiplication by two, and
shr by one position is a division by two. Negative numbers are not
allowed for the number of bits to shift -- i = i shl -1 does compile,
but the result is always zero. For all the shift operations, n shifts
each by one position would give the same result as one single shift
by n positions. For most modern CPU hardware, all the bit shifting
operations are very fast and generally take only one clock cycle,
independent of how many positions we move the bit pattern and
independent if it is a logical or an arithmetic shift operation.

We can use the and and the or operator to extract single bits, or to
set single bits:

var a = 3 # two rightmost bits, at position 0 and 1 are set
var b = a and 2 # extract bit 1, so b gets the value 2
b = a and 4 # extract bit 3, which is unset, so result is 0
b = a or (4 + 8) # result is \b00001111, decimal 15

This should be enough to teach you the most basic bit operations.
Actually we need these operations not that often, but we should be
aware of their existence. The overloading of the and, the or and the 
not operator for signed and unsigned integer numbers may be
convenient, but it may sometimes lead to confusion, when we intend to
do boolean operations but instead actually do operations on bit
patterns. It was suggested to call the operators bitand, bitor and
bitand instead, and indeed the bitops module of Nim's standard
library defines operators with these names and provide additional
more useful bit operations, including counting the number of set bits
in a variable or determining the number of leading zero bits. These
operations are not needed that often, but sometimes they can be very
useful, and they are supported by fast CPU instructions on modern PC
hardware. Note that while we have shown these bit operations on
integer numbers only, you can always cast other data types to
integers and then apply these operations as well.

Exceptions

When we execute our code, sometimes something can go wrong: We may
have an unexpected division by zero or an overflow, or we get some
invalid user input. There exists various strategies to handle such
situations. One is to terminate our program, we may do that by a
plain assert() or quit() statements. If we have absolutely no idea
how to recover from an error then that may be our best option. The
user may restart the program, or the program may be restarted by some
sort of supervisor program. For more expectable errors some sort of
error indicator may be a better solution, for example a parseInt()
procedure may return a boolean value for success. As we have to
return the result for success also, the parseInt() proc may return a
tuple, or may get a var parameter in which the result is returned.
Another solutions are exceptions as known from C++.

proc charToInt(c: char): int =
  if c in {'0' .. '9'}:
    return ord(c) - ord('0')
  raise newException(OSError, "parse error")

proc main =
  while true:
    stdout.write("Please enter a single decimal digit: ")
    let s = stdin.readline
    try:
      echo "Fine, the number is: ", charToInt(s[0])
    except:
      if s.len == 0:
        break
      echo "Try again"

main()

This section has to be extended... 

Destructors

Destructors and finalizers are used for automatic resource
management. For example files can be closed automatically when a file
variable goes out of scope, or when we create high level Nim bindings
to C libraries we can use finalizers or destructors to deallocate
entities of the C libs when a corresponding Nim (proxy) object is
freed. Libraries like the gintro GTK bindings make use of this.

Finalizers are procedures that can be passed as a second optional
parameter to the system new() proc. That way the finalizer proc is
attached to the data type of the variable which we pass as first
parameter to new() and that finalizer proc is automatically called
whenever that variable is freed by the Nim memory management system.
As finalizers are passed as a parameter to a new() call, and new() is
only used for references, finalizers work only for ref data types.

Destructors do not have this restriction. We define the destructor
for a value type, but it is also called for reference types by the
compiler.

Starting with version 1.4 Nim got scope based resource management
when the program is compiled with --gc:arc or --gc:orc. In that case
variables are immediately deallocated when they go out of scope, and
if a destructor was defined for the data type of that variable it is
called automatically.

For the programming language C++ it is a common practice that
resources like files are closed and released automatically by
destructors when they go out of scope, and now this is also possible
for Nim. To make use of destructors for our own data types we have to
define a proc called =destroy which gets an instance of our data type
passed as a var value object:

type
  O = object
    i: int

proc `=destroy`(o: var O) =
  echo "destroying O"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: O
      o.i = rand(100)
      echo o.i * o.i

randomize()
test()

In the for loop we enter a new scope when the if condition is
evaluated to true, and at the end of the if block we leaf the scope
and the destructor is called automatically. Inside the destructor 
proc we could do some cleanup tasks, close files and release
resources. Destructors are also called when ref objects go out of
scope:

type
  O = ref object of RootRef
    i: int

proc `=destroy`(o: var typeof(O()[])) =
  echo "destroying O"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: O = O() # new O
      o.i = rand(100)
      echo o.i * o.i

randomize()
test()

To use destructors we have to compile with the option --gc:arc or 
--gc:orc, otherwise the specified destructor procs are just ignored.
In our code we can test for working destructors with a construct like
when defined(gcDestructors):.

Note that destructors do not work for plain pointer types:

type
  O = object
    i: int
  OP = ptr O

proc `=destroy`(o: var O) =
  echo "destroying O"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: OP = create(O) # new O
      o.i = rand(100)
      echo o.i * o.i

randomize()
test()

So using destructors to release data from C libraries directly is not
possible. But at least for Nim >= v1.6 destructors work for distinct
pointer types:

type
  O = object
    i: int
  OP1 = ptr O
  OP = distinct ptr O

proc `=destroy`(o: var OP) =
  echo "destroying OP"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: OP = OP(create(O)) # new O
      OP1(o).i = rand(100)
      echo OP1(o).i * OP1(o).i

randomize()
test()

81
destroying OP
3600
destroying OP
2401
destroying OP
9025
destroying OP

So using destructors to destroy data from C libraries should be
possible now.

Destructors and Inheritance

When we use OOP style programming with subclassing of ref objects,
then it is useful to know that for subclassed ref objects the
destructor of the parent class is automatically invoked when we do
not define our own one for our subclassed type. This works also when
we import the parent type from another module, at least since Nim
v1.6:

# module tt.nim
type
  O1* = ref object of Rootref
    i*: int

when defined(gcDestructors): # check not really needed, as =destroy call is just ignored when condition is false
  proc `=destroy`*(o1: var typeof(O1()[])) =
    echo "destroy O1 ", typeof(o1)

# module t.nim
import tt

type
  O2 = ref object of tt.O1
    j: int

type
  O3 = ref object
    o1: tt.O1

type
  O4 = object
    o1: tt.O1

type
  O5 = ref object of tt.O1
    x: float

when defined(gcDestructors):
  proc `=destroy`(o5: var typeof(O5()[])) =
    echo "destroy O5 ", typeof(o5)
    tt.`=destroy`(o5)

proc main =
  var o1: tt.O1
  new o1
  echo o1.i

  var o2: O2
  new o2
  echo o2.j

  var o3: O3
  new o3
  new o3.o1

  var o4: O4
  new o4.o1

  var o5: O5 = O5(x: 3.1415)
  echo o5.x

main()

When we compile module t.nim with --gc:arc or --gc:orc and run it, we
get this output:

0
0
3.1415
destroy O5 O5:ObjectType
destroy O1 O1:ObjectType
destroy O1 O1:ObjectType
destroy O1 O1:ObjectType
destroy O1 O1:ObjectType
destroy O1 O1:ObjectType

So when our variables o1 to o5 go out of scope, then the destructors
are called. Module tt.nim defines a ref object type, but the
destructor proc takes a var value parameter. The destructor is called
when a value object or a ref object goes out of scope. Our variable
o1 has type tt.O1, so it was indeed expected that its destructor from
module tt.nim is called. Variable o2 is a ref object with parent O1,
as we define no destructor for this type, the destructor of the
parent type is called. The variables o3 and o4 are of ref object and
of value object types, each with a field of type O1, and for that
field the destructor for O1 is called. Finally for type O5 we define
our own destructor, which then additional calls the destructor of
module tt.

Destructors are mostly used for library implementations, e.g. for a
file data type which is automatically closed when a file variable
goes out of scope. As you may never have to use destructors yourself,
it is not necessary to remember all these details. But it is good to
know that destructors behave in a way as we may have expected, and
when you later wants to use a destructor in your own code, you can
consult again this section or maybe better the Nim manual.

References:

  * https://forum.nim-lang.org/t/8013

  * https://forum.nim-lang.org/t/7360

Finalizers

In Nim finalizers are procedures, that we can specify as an optional
second parameter when we call the system new() proc to allocate heap
memory for a reference type variable. That specified finalizer proc
is then later called by the Nim memory management system when the ref
variable is freed:

type
  O = ref object of RootRef
    i: int

proc finO(o: O) =
  echo "finalize O"

proc newO: O =
  new(result, finO)

proc main =
  var o = newO()
  var o2 = new(O)
  var o3 = O(i: 7)

main()

finalize O
finalize O
finalize O

The output of above program is really surprising at first: Only for
variable o we call the proc newO() to initialize it, which then calls
new() by passing a finalizer proc named finO(). For o2 and o3 we
allocate memory as usual, without use of a finalizer proc. But when 
o2 and o3 goes out of scope, even for these two variables the
finalizer proc finO() is called. The reason for this is, that the
system proc new() binds the optional finalizer proc to the data type
of the passed ref variable. This binding process occurs for the first
call with a passed finalizer proc and can not be reverted. We can
later call new() without finalizer or use the similar O() call to
initialize the ref variable, but that can not undo the binding. And
using a different finalizer proc for the same data type would not
work any more. Passing the same finalizer proc multiple times is OK
and may be a common use case, but it has no real effect, as the first
call did the binding already.

This behaviour of finalizers in Nim is indeed a bit confusing and
error prone. Maybe somewhere in a large program we pass a finalizer 
proc to new() and forget about it. Later, we use new () without a
finalizer or use the O() notation to reserve the memory for our ref
variable. So we think that no finalizer is involved, but as a
finalizer was used somewhere at least once, it is now bound to all of
our allocations of that data type. That can easily lead to bugs as
the unintended called finalizers may do things that it should not do
with our data.

Finalizer procs have to be defined always in the same module as the
type for which the finalizer shall be used is defined:

# module tt.nim
type
  O* = ref object of RootRef
    i: int

proc fin*[T](o: T) =
  echo "finalize T"

proc newO*: O =
  new(result, fin)

import tt

type
  OO = ref object of tt.O
    x: float

proc finn[T](o: T) =
  echo "finalize O"

proc main =
  var oo: OO
  new(oo, finn)

main()

We import module tt.nim and subclass the ref object type tt.O. While
module tt.nim defines a generic finalizer proc fin(), we can not use
that one for our subclased type OO, but have to copy it from module 
tt.nim into our main module and we may have even to use a different
proc name. Otherwise we get the compiler message

Error: type bound operation `fin` can be defined only in the same module with its type (OO:ObjectType)

Whenever we should really need a finalizer or a destructor, we should
prefer destructors when we can compile our code with the compiler
options --gc:arc or --gc:orc.

Modules

Modules are Nim's way to divide multiple source codes in clearly
separated units and to hide implementation details. Nim use a module
concept which is very similar to Modula-2 or Oberon. All the Nim
standard libraries are divided into modules which collect and
logically group data types and the related procedures. In some way
modules are Nim's classes.

In Nim each module directly corresponds to one text file. Sub-modules
as known from Ruby, that divide a single text file in multiple
modules are not supported by Nim currently.

Every text file with Nim source code is basically a module, and that
module can be imported and used by other modules. But all symbols
like data types or procedures have to be exported to make them
visible and usable by other modules. That is done like in Oberon by
appending an asterisk character to all names that should be exported.
These restricted exports allows to hide implementation details -- all
symbols not exported are private to that module and can be changed
and improved at any time without noticing the importing module. Note
that when we append the asterisk to the name of an object to export
that type, the objects fields are still hidden and can not be
accessed from within the importing module. You may append an asterisk
to selected field names as well, or you may provide exported getter
and setter procs for the field access. A read-only export es known
from the Oberon language is currently not possible with Nim.

We can import whole modules, that is all symbols that are marked for
export by the asterisk, or we can import only the symbols that we
need by specifying their names. Let us create a module that declares
a single procedure to remove all characters from a string that are
not letters:

# save this textfile with name mystrops.nim
proc remNoneLetters*(s: string): string =
  result = newString(s.len)
  var pos = 0
  for c in s:
    if c in {'a' .. 'z', 'A' .. 'Z'}:
      result[pos] = c
      inc(pos)
  result.setLen(pos)

We save above text file with our Nim source code with name
mystrops.nim. Note the export marker after the proc name. We can
import and use that module like

import mystrops

echo remNoneLetters("3h7.5g:8h")

When we import modules, then we put the import statement generally at
the top of the importing module, that way it is easy to see what
modules are imported. The imported symbols can be used in the code
after the import statement. Module names should be lower case and may
as other Nim symbols only contain letters, decimal digits and the
underscore character. We can import multiple modules with a single
import statement when we separate the module names with commas.
Starting with Nim v1.6 it is recommended to import modules from Nim's
standard library with the std prefix as in import std/math or import
std/[strutils, sequtils]. Importing the same module multiple times is
not a problem, and does not increase the code file of the final
executable. Note that in the import statement the module names have
to be used literally, so this would not work:

const strfuncs = "stringutils"
import strfuncs

Instead of importing whole modules we can import only single symbols
with the from x import y, z syntax like

from mystrops import remNoneLetters

echo remNoneLetters("3h7.5g:8h")

Both forms are an unqualified import, that is we can refer to the
proc by only its name, we do not need the qualified form with module
name prefix like mystrops.remNoneLetters() as long as there are no
name conflicts. But whenever we want we can use the qualified form
also.

Nim programmer generally prefer importing whole modules and use
unqualified names, while that is considered bad style in some other
languages like Python. In untyped languages like Python unqualified
imports may indeed pollute the name space and generate many name
conflicts, but in statically typed languages like Nim unqualified
imports seems to generate name conflicts only in very rare cases.
Procedures with the same name generally have different parameter
lists, so the overload resolution of the compiler can decide what 
proc is to be used. And when really a name conflict occurs then the
compiler will tell us, and we can easily fix it by prefixing the proc
name with its module name.

For data types, constants or enums chances for name conflicts may be
not that tiny, so we may have to use qualified names.

We can also enforce a fully qualified import in Nim by a notation
like

from mystrops import nil

In this case we can use all symbols from that module only in
qualified form. But that does not always work that well in Nim, as
Nim has not classes like Java, so a qualified use of method call
syntax or qualified use of user defined or overloaded operators is
difficult. Imagine strutils.add(s, '\n'), how should that look with
method call syntax?

For imports we have also the except keyword, so we may do something
like

import strutils except toUpper

The except keyword can be used to prevent possible name conflicts
without having to use qualified names.

Note that the system module is imported automatically, we should not
import it directly. Also note that Nim always imports only what is
really needed in the final executable, so importing only a few
symbols from a module has no code size benefit over importing the
whole module. Still it may improve readability of your code when you
import only single symbols when you are sure that you need not more.
Maybe like from math import Pi. Note that you can even in that case
access other symbols of that module by fully qualified names like 
math.sin().

With the growing standard library it may occur that module names of
the standard library interfere with your own module names. So Nim now
allows and recommends qualified import of modules from the standard
library like import std/strutils. And for external packages installed
by the nimble package manager imports in the form import package/
[mod1, mod2, mod3] are permitted.

Finally you can also import modules under another name using the as
keyword like

import tables as maps

With the recent Nim compiler you can also enforce full qualified
import and use of an alternate module name by using an import
statement like

from tables as maps import nil

With this import statement you could access symbols from the tables
module only by use of the maps module prefix like maps.newTable().

Finally with the export keyword one library module can export other
modules, which it imports itself. This may simplify the use of
connected modules. As an example, when using the gintro bindings for
GTK4, we import all the needed modules generally like import gintro/
[glib, gobject, gtk4]. We may decide to simplify that import
statement by creating one more module called gtkplus that consists
only of these two lines:

# module gtkplus
import gintro/[glib, gobject, gtk4]
export glib, gobject, gtk4

Then a user of gintro could just write import gtkplus to have access
to all the modules. Actually for GTK this is not really a good idea,
we will tell you more about the gintro module and maybe about one
more of Nim's GUI libraries in the second half of the book.

Cyclic Imports

Generally we try to arrange our own modules in a tree-like bottom-up
structure. A module x may define basic types and simple functions
working with these types, and a higher level module y may import all
symbols from module x and extend the functionality. But in rare cases
it may be necessary that the modules x and y import each other, as x
has to use types or functions of module y, and vice versa. This case
is called cyclic import and is currently not supported by Nim. Indeed
we should generally try to avoid cyclic imports when possible, as
cyclic imports make the software design difficult. But sometimes we
can not really avoid these cycles. In that case currently the best
solution is, to put all the concerned data types in a separate low
level module, which is then imported from both other modules. The
planned Nim version 2.0 may allow cyclic imports, so this restriction
may vanish in the future.

Include

The include statement should be not confused with the import
statement. Include just insert a text file at the position where the 
include statement occurs. Include can be used to split very large
modules in smaller entities.

Part III: The Nim Standard Library

In this part of the book we will introduce you to some of the most
important modules of the Nim standard library. This includes modules
for common operations like the serialization of Nim data types to
write them to external nonvolatile storage and read them back into
the program later, or handling command line options and parameters
for programs launched from within a terminal window. Further we will
introduce you to important container data types like hash tables
(sometimes called hash maps in other programming languages) and and
various kinds of set data types. We will also introduce modules for
working with regular expressions, and we will show how simple modules
like the times or the random module can be used. Most modules
mentioned in this part will be from the Nim standard library, so that
you will not have to install external packages for that. But there
may be some exceptions, e.g. for some external nimble packages with a
very useful functionality and an easy user interface.

Command Line Arguments

When we launch a program from inside the terminal window, we can pass
it some additional parameters, e.g. the name of a file to process or
option parameters to influence the behaviour of the program. We have
done so already when we launched the Nim compiler or maybe a text
editor from inside our terminal window. Using command line parameters
is convenient when we work from inside a terminal and there are
parameters that we know in advance. A more interactive way to collect
parameters is reading in input while the program is already running,
as we did in part II of the book when processing the list of our
friends. We will learn some more details of this interactive
processing of input in the next section.

Nim allows to process command line arguments in the same basic way as
all C programs do it, but Nim's standard library and some external
packages allow also much more advanced handling of command line
arguments. For simple cases the C-like way is sufficient. For C
programs the command line arguments are even coupled very closely to
the language itself, the number of arguments and the list of
parameters are the two typical parameters of the C main() function
and are used in this way:

// C program expecting one command line argument
// Compile with gcc t.c
#include <stdio.h>
int main( int argc, char *argv[] ) {
  printf("Executing program %s\n", argv[0]);
  if( argc == 2 ) {
     printf("The argument supplied is %s\n", argv[1]);
  }   else if( argc > 2 ) {
     printf("Too many arguments supplied.\n");
  }
  else {
     printf("One argument expected.\n");
  }
}

Here argc is the number of available arguments, and argv is an array
containing the actual arguments in form of strings. These values are
passed to each C program by the OS when the program is launched from
inside a terminal. Actually the value of argc is the number of passed
arguments plus one, that is when we specify no arguments at all, argc
has the value one. And argv[0] is always the name of the executed
program. We have to know that command line arguments passed to a
program are separated by white space, that is at least by one space
or tab character. For this reason we have to enclose single arguments
containing white space in double quotes:

$ gcc t.c -o t
$ ./t Nim two
Executing program ./t
Too many arguments supplied.
$ ./t "Nim two"
Executing program ./a.out
The argument supplied is Nim two

In Nim we have the same functionality available by use of the 
paramCount() and paramStr() procs, which we have to import from the 
os module. But paramCount() gives us the actual number of parameters,
so when we call our program on the command line without any
arguments, paramCount() will return the value zero. The symbol 
paramStr() is not a global array variable, but a procedure. ParamStr
(0) gives us the name of our executable, and with arguments greater
zero we get the passed arguments as strings in ascending order. Using
an index number for an argument that was not provided will cause 
paramStr() to raise an exception.

A argument evaluation similar to our C program from above may look
like

from os import paramCount, paramStr

proc main =
  echo "Executing program ", paramStr(0)
  let argc = paramCount() + 1
  if argc == 2:
    echo "The argument supplied is ", paramStr(1)
    if paramStr(1) in ["-d", "--debug"]:
      echo "Running in debug mode"
  elif argc > 2:
    echo "Too many arguments supplied."
  else:
    echo "One argument expected."

main()

Using this plain API is OK when we expect one or two arguments, maybe
a file name and an option, like the -d or --debug parameter used in
the code above. For more command line arguments, things get
complicated fast, as arguments can be passed in arbitrary orders and
combinations. So you should try one of the available libraries for
that case.

References:

  * https://github.com/c-blake/cligen

Reading data from the terminal

While using command line arguments is convenient for data like file
names or options that we already know when we launch a program from
the terminal window, often we have to provide textual user input
while the program is already running. Functions for this task are
provided by the io module, which is part of the system module, and
which we have not to import explicitly. In one of the introducing
sections of the book we used already the readLine() and the getch() 
procs for reading in a line of text from the terminal and for waiting
on a single key press event.

For input and output operations in a terminal window the io module
defines the three variables stdin, stdout and stderr of File data
type. We will discuss file-based input and output operations in more
detail in part III of the book. Many procs of the io module expects
as first parameter a variable of file type. We can explicitly open a
named file to write data to external media like the SSD, or we can
just use the stdin and stdout variables to read data from the
keyboard and to write text to the terminal window. Unlike other named
files, we do not have to call open() or close() on stdin and stdout
to open or close the files, and some other file operations like 
setFilePos() may not work for these file variables:

var s: string = stdin.readLine()
stdout.write(s)
stdout.flushFile

We mentioned already, that the readline() function let you type
textual user input, including spaces, and that you have to terminate
your input by pressing the return key to pass the input string to the
OS which forwards the input to our program. This form of input is
sometimes called blocking, as for the time that we wait for user
input, our program is really waiting, it can not do other work til
the user has pressed the return key. For single character input,
without the need for pressing actually the return key, e.g. for a
simple yes/no input, you may use the getch() function, which is also
blocking. In a latter section of the book we may show how we can use
threading to actually do some useful work, while we wait for user
input. In the literature stdin, stdout and stderr are often called
streams, where stderr can be used instead of stdout for writing error
message. This can be useful in special cases, when we have an
application where we want to redirect error messages to a file or to
separate regular output and error messages. For more details about
these stream or file variables and the use of the stderr variable you
may consult external literature, if you should really need that info.

The io module does not provide read functions for other basic data
types like numeric or boolean types. So we should use readLine() to
read the user input in string form, which we can convert by functions
like parseInt(), parseFloat() or similar functions to numeric data.
Note that parsing procs like parseInt() are provided by the module 
strutils as well as by the module parseutils -- one function raises an
exception for invalid input, while the other one returns a boolean
value indicating conversion success. Of course we should handle
textual user input always carefully and never just assume that the
input is actually valid data. Some of the modules that can be used
for converting textual input data into other data types like the 
strutils, strscans modules are described in more detail at the end of
this part of the book.

For advanced user input processing, like cursor movement, colored
display or displaying progress bars, you may also consult the 
terminal module. And finally, to create fancy textual user interfaces
(TUIs) we recommend to try external packages like the illwill
library.

References:

  * https://en.wikipedia.org/wiki/Standard_streams

  * https://nim-lang.org/docs/terminal.html

  * https://github.com/johnnovak/illwill

Writing text to the terminal window

In previous sections we have used the echo() function to write
variables of various data types to the terminal window. The echo()
function accepts multiple arguments, writes the string representation
of the passed arguments to the terminal window and terminates the
action by writing the \n character to move the cursor to the
beginning of the next line in the terminal window. We have already
used the write() function from the io module for the case that we
want to write a single string to the terminal without a terminating
newline character. The io module contains some overloaded write()
functions for other basic data types like int, float or bool, and a
variant with a varargs parameter and applied stringify operator, so
that write() can be used like echo, as long as we pass stdout as
first parameter. For the actual output operation the C library
function fprintf() is used. Note that write operations to stdout are
generally buffered, so the result of write() operations may remain
invisible until we write a string containing a newline character or
until we call the flushfile() function to enforce the writing of the
buffer.

Serialization -- storing data permanently on external storage

When you start writing larger programs, these programs may create
data which you may want to store permanently on external nonvolatile
storage like SSDs or traditional hard disks of your computer. For
textual data this is very easy, as you basically only has to write
and read a stream of unstructured bytes. But when your program deals
with object instances, container data types like sequences or
references, things become more complicated. Writing the data is
always easy -- you can just convert all the fields of your object data
type to strings and write them to a stream or a file. But the reading
back part is much more difficult: You would have to read in the data
as strings and then process each string -- maybe converting to a float
number -- and then assign to the matching field of an object instance.

When your data consists only of value objects and no references, then
you may consider just writing that data in plain binary form to a
file and read it back. This strategy seems to be simple and it is
very fast, as no type conversion steps are involved. But at the same
time it has some drawbacks: The stored data can not be checked with
tools like a text editor, it can generally not be used from other
programs, and when you should change the data types used in your
program, you could not read back stored files any more.

So we will explain how you can store Nim data types in a human
readable text format first. Two popular text formats are JSON and 
YAML. JSON is a simple format, which is easy to parse, but not very
good readable for humans. YAML is more complicated, but more flexible
and is very good readable for humans.

For Nim we have already many modules which we can use for storing
data in JSON or YAML format available. The Nim standard library
includes the marshal and the json module. The marshal module uses
like the json module the json data file format, is easy to use and
simple, but is not really designed to generate human readable data
files, as the stored data is not stored as a sequence of individual
lines. So we will describe and use the json module in this section,
which is also easy to use, but has some larger set of functionality
and can generate real human readable text files by use of the pretty
() function.

Other available external packages for data serialization are the
nim-serialization module set from (https://github.com/status-im/
nim-serialization) and the very powerful but complicated NimYaml
implementation (https://nimyaml.org/). We may describe these packages
in part V of the book.

When we have to store and read back Nim data to nonvolatile storage
media, we have some serious points to consider: First we have to
handle various data types like integers, floats, strings,
objects -- and even the container types like sequences. And we may
have to support reference types and maybe also inherited types and
containers filled with heterogeneous, subclassed reference objects.
The json module supports all Nim data types including containers and
references, but not heterogeneous sequences.

For our first json example let us assume that we have written a small
tool that let the user create some geometrical shapes, and we want to
store the shapes to a file and read it back. For that we generally
use an intermediate step, which converts the data to a string and the
string back to the data object. The string is then written to a file
or stream and read back. Let us start with the string
conversion -- storing that string and reading it back from the file
will be explained soon.

import json

type
  Line = object
    x1, y1, x2, y2: float

  Circ = ref object of RootRef
    x0, y0: float
    radius: float

  Data = object
    lines: seq[Line]
    circs: seq[Circ]

var
  l1 = Line(x1: 0, y1: 0, x2: 5, y2: 10)
  c1 = Circ(x0: 7, y0: 3, radius: 20)
  d1, d2: Data

d1.lines.add(l1)
d1.circs.add(c1)
d1.lines.add(Line(x1: 3, y1: 2, x2: 7, y2: 9))
d1.circs.add(Circ(x0: 9, y0: 7, radius: 2))

let str1 = pretty(%* d1) # convert the content of variable d1 to a string
echo str1 # let us see how the strings looks
d2 = to(parseJson(str1), Data) # read the string back into a data instance
let str2 = pretty(%* d2) # and verify that we got back the original content
echo str2

# assert d1 == d2 would fail
assert str1 == str2

When we run the program we would get this output:

{
  "lines": [
    {
      "x1": 0.0,
      "y1": 0.0,
      "x2": 5.0,
      "y2": 10.0
    },
    {
      "x1": 3.0,
      "y1": 2.0,
      "x2": 7.0,
      "y2": 9.0
    }
  ],
  "circs": [
    {
      "x0": 7.0,
      "y0": 3.0,
      "radius": 20.0
    },
    {
      "x0": 9.0,
      "y0": 7.0,
      "radius": 2.0
    }
  ]
}
{
  "lines": [
    {
      "x1": 0.0,
      "y1": 0.0,
      "x2": 5.0,
      "y2": 10.0
    },
    {
      "x1": 3.0,
      "y1": 2.0,
      "x2": 7.0,
      "y2": 9.0
    }
  ],
  "circs": [
    {
      "x0": 7.0,
      "y0": 3.0,
      "radius": 20.0
    },
    {
      "x0": 9.0,
      "y0": 7.0,
      "radius": 2.0
    }
  ]
}

As you can see we converted the instance d1 of type Data to a string
and then we convert that string back to variable d2 with matching
content. We have made intentionally the Circ a ref object, so we can
see that the conversion works for value and reference objects. In the
example program we applied the %* macro to our data instance d1 to
get a JsonNode, and finally use the pretty() function to get a nice
multi-line string. To fill the variable d2 with the content stored in
str1, we first have to apply parseJson() on the string and then use 
to() to unmarshal the json node into the matching object type.

Now let us investigate what happens when we try to use the json
module with a container with heterogeneous ref objects. For that we
subclass the Disc type creating a new Arc type:

import json
from math import PI

type
  Line = object
    x1, y1, x2, y2: float

  Circ = ref object of RootRef
    x0, y0: float
    radius: float

  Arc = ref object of Circ
    startAngle, endAngle: float

  Data = object
    lines: seq[Line]
    circs: seq[Circ]

var
  d1, d2: Data

d1.lines.add(Line(x1: 0, y1: 0, x2: 5, y2: 10))
d1.circs.add(Circ(x0: 7, y0: 3, radius: 20))
d1.lines.add(Line(x1: 3, y1: 2, x2: 7, y2: 9))
d1.circs.add(Arc(x0: 9, y0: 7, radius: 2, startAngle: 0, endAngle: PI))

echo d1.circs[1] of Arc, " ", Arc(d1.circs[1]).endAngle

let str1 = pretty(%* d1)
d2 = to(parseJson(str1), Data)
let str2 = pretty(%* d2)
echo str2
echo d2.circs[1] of Arc

The output of that program looks like this:

true 3.141592653589793
{
  "lines": [
    {
      "x1": 0.0,
      "y1": 0.0,
      "x2": 5.0,
      "y2": 10.0
    },
    {
      "x1": 3.0,
      "y1": 2.0,
      "x2": 7.0,
      "y2": 9.0
    }
  ],
  "circs": [
    {
      "x0": 7.0,
      "y0": 3.0,
      "radius": 20.0
    },
    {
      "x0": 9.0,
      "y0": 7.0,
      "radius": 2.0
    }
  ]
}
false

While our initial instance d1 contains a run-time value of Arc type
and so we can access the endAngle field, we get false as result for
the of Arc test for the d2 instance. So run-time type information is
lost.

When we have to store different data types in one container, then one
solution is to use object variants, which should work with the json
module. Another obvious possibility is to just copy the data into
containers with the appropriate static type before storing to an
external medium and copy them back when we read the data back from
external storage. Will will show an example for that now:

import json
from math import PI

type
  Line = ref object of RootRef
    x1, y1, x2, y2: float

  Circ = ref object of RootRef
    x0, y0: float
    radius: float

  Arc = ref object of Circ
    startAngle, endAngle: float

  Data = object
    elements: seq[RootRef]

  Storage = object
    lines: seq[Line]
    circs: seq[Circ]
    arcs: seq[Arc]

const
  DataFileName = "MyJsonTest.json"

var
  d1, d2: Data
  storage1, storage2: Storage
  outFile, inFile: File

d1.elements.add(Line(x1: 0, y1: 0, x2: 5, y2: 10))
d1.elements.add(Circ(x0: 7, y0: 3, radius: 20))
d1.elements.add(Line(x1: 3, y1: 2, x2: 7, y2: 9))
d1.elements.add(Arc(x0: 9, y0: 7, radius: 2, startAngle: 0, endAngle: PI))

for el in d1.elements:
  if el of Arc:
    storage1.arcs.add(Arc(el))
  elif el of Circ:
    storage1.circs.add(Circ(el))
  elif el of Line:
    storage1.lines.add(Line(el))
  else:
    assert(false)

let str1 = pretty(%* storage1)

if not open(outFile, DataFilename, fmWrite):
  echo "Could not open file for storing data"
  quit()
outFile.write(str1)
outFile.close

if not open(inFile, DataFilename, fmRead):
  echo "Could not open file for recovering data"
  quit()
let str2 = inFile.readAll()
inFile.close

assert str1 == str2

storage2 = to(parseJson(str2), Storage)

for el in storage2.lines:
  d2.elements.add(el)
for el in storage2.circs:
  d2.elements.add(el)
for el in storage2.arcs:
  d2.elements.add(el)

for el in d2.elements:
  if el of Arc:
    echo "found arc with endAngle: ", Arc(el).endAngle

For this example program we use OOP programming style and keep all
the geometric object instances as references in a single sequence.
Note that doing this is not always a good idea, as this OOP style
with the use of references and dynamic run-time dispatch can be
slower due to many small heap allocations for each ref object and due
to the dynamic dispatch (if el of ... ) overhead. Using multiple,
homogeneous sequences with value types for each of our data types can
be a better solution, and in that way you have more control whenever
you process the data, for drawing them on the screen or user
interaction for example. Maybe you want to draw all the lines first?
But there can be situations where we really need to have all the
objects as references in a single container. A typical situation is,
that we use an RTree for fast object location. RTrees are data
structures, that can store two dimensional or multidimensional
geometric object and their rectangular bounding boxes in a tree-like
fashion for fast object location. This may be used in a drawing
program, so that coordinates of a user mouse click can be fast
matched to an object. For such a use case we would really prefer to
have all the object instances available in one single RTree, and not
use one RTree data structure for each object shape.

Our program defines an additional Storage data type, which contains
homogeneous sequences for each possible geometric shape. We then copy
all our ref objects from the elements sequence in the matching
sequences of the storage object using the dynamic of type query to
select the exactly matching sequence.

After that we can use the already known json functions to serialize
the storage object into a string, store the string to a file, read it
back and deserialize the data again into a different variable of 
Storage data type. Finally we use a simple for loop to copy the ref 
objects from the temporary storage object into a Data variable called
d2. For storing the data to a external nonvolatile medium we use the 
File data type and the related functions open(), close, write() and 
read(). Their use should be obvious: We pass a uninitialized variable
of File data type, a file name and a file mode to open(), use write()
to write the whole string, and use readAll() to read the data back.
When done with each file we use close() to close the file. The File
data type is part of the io module, which is again part of module 
system, so we don't have to import these modules. We could have used
as an alternative also the streams module. We will learn some more
details about the File data type and the streams module in later
sections of the book.

 We should mention that unfortunately live is not always that easy,
 as sometimes we can not freely select the textual output format.
 Imagine that you create a CAD (computer aided design) tool that
 should be compatible with another existing tool. In this case the
 textual storage format is already defined by the existing tool, and
 generally that format does not match the json or yaml file format.
 Even when the format should be one of these, matching it exactly
 would be difficult. While writing out own data in that foreign
 format is still not really difficult, as we can just write single
 matching strings, reading in the textual data is more complicated:
 Generally we would read the input file line by line and we would
 have to inspect and interpret each input string, maybe by use of
 regular expressions or a custom parser. That generally includes
 handling of missing or invalid data.

References:

  * https://nim-lang.org/docs/json.html

  * https://nim-lang.org/docs/io.html

  * https://nim-lang.org/docs/marshal.html

  * https://github.com/status-im/nim-serialization

  * https://nimyaml.org/

Streams and Files

In the previous section we learned how we can store structured data
like a sequence of objects in human readable form to nonvolatile
media by use of the json module.

Text in form of a single string or in form of a container holding
multiple strings is some kind of unstructured data which we can write
directly to nonvolatile storage, and later read it back. We can do
the same with containers of basic, unstructured data types like
integer or floating point numbers, and with some restrictions we can
even write tuples or objects directly as raw bits and bytes to
external storage and read it back later. Of course this way the
stored data is a binary blob, which can not be read or modified by
other tools like a text editor. But that may be not intended or
advantageous at all, maybe we do scientific data processing with a
single tool, and we just want to temporary store the data and
continue with the processing later.

Files

For storing unstructured data Nim provides the io module with the 
File data type and related procs, and the streams module with the 
Stream data type and related procs. While a File in Nim is currently
only a pointer to a C file, the streams module has a higher
abstraction level. Although the Nim language does not directly
support interfaces, the Stream data type of the streams module is
some form of an interface, which is implemented by a StringStream and
a FileStream data type. Internally this interface concept is realized
by storing a set of function pointers in the Stream instance.

When we have to store unstructured data like text it is not always
clear if we better should use Files or Streams. Streams may be the
better choice when we (also) want to use a string as data source like
a file or when we need the peek() functions of the streams module to
access data without advancing the position in the stream.

We will use the File data type of the io module first. As the io
module is part of the system module, we do not have to import it
before we can use it. The principle usage of files is, that we call
the function open() to open a file with given name, call some procs
to write or read data, and finally close() the file. While Nim
support destructors when we compile with --gc:arc or --gc:orc, the io
module does not yet use them, so we should actually call close() to
close the file.

Historically a file is a one dimensional data type which is accessed
in sequential order. Up to the end of the twenty century it was not
uncommon that large files where stored on magnetic tapes, which could
be read or written only slowly in sequential order. Read or write
operations could take place only at the actual position, and
available functions like f.setFilePos() where very slow as it
involves moving the tape. The introduction of hard disks and solid
state disks removed this restriction, and modern operating system
often buffers files in RAM for longer time periods, so that files may
have actually similar performance as arrays or sequences. The funny
fact is, that with the modern CPU caches the ordinary RAM storage can
look similar slow and sequential compared to the extreme fast cache
as magnetic tapes in the past.

from os import fileExists
proc main =
  const FN = "NoImportantData"
  if os.fileExists(FN):
    echo "File exists, we may overwrite important data"
    quit()
  var f: File = open(FN, fmWrite)
  f.write("Hello ")
  f.writeLine("World!")
  f.writeLine(3.1415)
  f.close
main()

Running that program will create a text file with this content in the
current working directory:

Hello World!
3.1415

At the start of our function we check if a file with that name
already exists in the current working directory by using the function
os.fileExists() to ensure that we do not overwrite important data.

Module io provides multiple overloaded open() procs. We use here a
variant which returns a file, and raises an exception for the
unlikely case of an error. We provide a file name and a file mode as
parameters. We use mode fmWrite as we want to create a new file. Note
that fmWrite would clear the content of an existing file, so we can
not use fmWrite to append data to an existing file. We would have to
use fmReadWriteExisting or fmAppend to append data to an already
existing file. As this open() proc can raise an exception, it may
make sense to enclose it in a try/except block, or we could use a 
open() variant which returns a boolean value to indicate success
instead. When the file is successfully opened, we can use procs like 
write() or writeLine() to write text strings to the file. Both procs
accept multiple arguments and apply the stringify operator $ on them
before writing the content. WriteLine() writes a '\n' after the last
argument to start a new line. When done we call close() to close the
file. The operating system would close the file for us when our
program terminates, so calling close is not that important, but when
we open many files without closing them we may get errors from the
operating system finally about too many open files and our program
may fail or terminate.

The close() proc gets passed the file not as a var parameter, so it
can not set the file to value nil. When the file has the value nil,
then the close() call is ignored, but when we would call close()
multiple times with a non nil argument we get a program crash. We may
use the try/finally or the defer construct to ensure that we really
close the file when done.

The io module provides some procs like writeBuffer(), writeBytes() or
writeChars() which gives us as return value the actual number of
bytes written. This return value should generally match the requested
number of bytes to write, but can be smaller when the write operation
fully or partially failed, e.g. because the storage medium had no
capacity left.

When performance really matters, we should note that passing
non-string arguments to write() or writLine() procs using their
optional auto-stringify for us, involves allocation of new strings
and cost some performance. When we have in our program already a 
string variable available, it can be faster to convert our data into
that variable first and then pass that variable to the write() or 
writeLine() procs.

Reading strings from a file works very similar:

proc main =
  var f: File
  try:
    f = open("NoImportantData", fmRead)
    echo f.readLine
    echo f.readLine
  finally:
    if f != nil: # test for nil not really necessary, close() would ignore the call for f == nil
      f.close
main()

The readLine() proc reads in a line of text. The LF, CR or CRLF line
end markers are not part of the returned text string. Of course we
may get an empty string with length zero back when we read a line
which immediately starts with LF, CR or CRLF, or we may get back a 
string with no visible characters but only a few spaces or tabulator
characters '\t' when a line contains only white space. When our read
() operations have moved the actual file io position to the end of
the file, and we try to read more content, then an exception is
raised.

The io module provides a readLine() proc that returns a newly
allocated string, and one that takes an existing string as var
parameter. The later may be a bit faster, as it can avoid the
allocation of a new buffer when the passed string has already enough
capacity.

The io module provides a function called endOfFile() with a boolean
result which we can use to check if the end of file position is
already reached. The provided functions readBuffer(), readBytes() or 
readChars() return the actual number of bytes read, which can be
smaller than the requested value when the end of the file is reached
earlier. Currently readChars() checks if the passed openArray[char]
has enough capacity for the request, but readBytes() does no check!

We can use also the lines() iterator to iterate over the lines of a
text file, or use the readLines() proc to read the content line by
line.

proc main =
  var f: File
  f = open("NoImportantData", fmRead)
  for str in f.lines: # iterator
    echo str
  f.setFilePos(0) # read again from start index 0
  var s: string
  while f.readLine(s): # proc
    echo s
  f.close
  var sq = readLines("NoImportantData", 2) # read lines to seq of strings
  echo sq
main()

As iterating over the whole file line by line moves the actual file
position to the end of the file we called setFilePos() to move again
to the start position. The readLines() proc takes a filename and the
number of lines to read as parameters and returns a seq of strings.
When the file does not contain at least the number of requested lines
an EOF exception is raised. Another provided proc is readAll() which
reads the whole file content into a returned string variable. For 
readAll() to work the actual file position has to be the the start of
the file. In case of an error an exception is raised.

We can also write and read binary data directly to a file, without
converting it to (human readable) strings first:

proc main =
  var f: File
  f = open("NoImportantData", fmWrite)
  var i: int = 123
  var x: float = 3.1415
  assert f.writeBuffer(addr(x), sizeof(x)) == sizeof(x)
  assert f.writeBuffer(addr(i), sizeof(i)) == sizeof(i)
  f.close
  f = open("NoImportantData", fmRead)
  assert f.readBuffer(addr(x), sizeof(x)) == sizeof(x)
  assert f.readBuffer(addr(i), sizeof(i)) == sizeof(i)
  f.close
  echo i, " ", x
main()

Of course these are low level, dangerous operations. While 
writeBuffer() should never crash our program, readBuffer() can do
that easily when we specify wrong sizes or destination addresses, as
that may overwrite other data unintentionally. So we would generally
not use these procs directly but write more safe helper procs, when
we really need or want this form of binary file access. Fast storing
big data sets with restricted hardware may be an use case, e.g.
storing a float32 takes only 4 bytes on the storage medium and file
io is fast, while that number as human readable digits may need more
than 8 bytes (1.234567E3) and converting to string and and parsing
back costs some time.

In the same way we can use writeBuffer() and readBuffer() to store 
tuples, objects and arrays or sequences of these directly in binary
form:

type
  O = object
    x: float
    i: int
    b: bool

proc main =
  var s: seq[O]
  s.add(O(x: 3.1415, i: 12, b: true))
  var f: File
  f = open("NoImportantData", fmWrite)
  assert f.writeBuffer(addr(s[0]), sizeof(O) * s.len) == sizeof(O) * s.len
  f.close
  f = open("NoImportantData", fmRead)
  var s2 = newSeq[O](1)
  assert f.readBuffer(addr(s2[0]), sizeof(O) * s2.len) == sizeof(O) * s2.len
  f.close
  echo s2[0]
main()

Output should look like

(x: 3.1415, i: 12, b: true)

But of course this is dangerous and fragile. We just show that
example as beginner generally ask about it, and may want to try it at
least once. Obviously this can only work when the tuples or objects
contain only plain data types, that is no string, no references and
of course no other nested container types like sequences or tables.
And reading back data may fail when we use a different OS or a
different compiler version.

The io module provides the File variables stdin, stdout and stderr,
which are the standard input, output and error streams. Sometimes we
use stdout.write() instead of the common echo() proc when we want to
write something to the terminal window without moving the cursor to
the next line already.

An important function of the io module is flushFile(), which is used
to ensure that all buffer content of buffered files is actually
written to the file. This is important when we use the stdout File
variable, maybe to ask the user a question in the terminal window. We
would call sdtout.flushFile() to ensure that the user really sees the
text on the screen immediately. The echo() proc calls flushFile()
automatically after each output operation. When we close a file 
flushFile() should be called automatically, but when our program is
terminated without calling close() it may depend on the actual
implementation and operating system

The io module provides some more useful procedures, but we will stop
this introducing section here and continue with the streams module in
the next section.

References:

  * https://nim-lang.org/docs/io.html

Streams

A stream is an abstract interface for performing certain I/O
operations, which was introduced by languages like C or Modula-2
decades ago. The streams module of the Nim standard library provides
a FileStream and a StringStream implementation, which behaves very
similar. Nim's streams module provides similar functions as the io
module with its File data type, but it can operate on strings instead
of on Files, and it provides a set of peek() functions to access data
at the current read position without moving forward. And some
functions are more robust, for example closing a stream multiple
times does not crash the program, as the first close() call sets the
file variable of file streams to nil, so that following close() calls
are ignored. Currently the streams module does not support
automatically closing of streams when they go out of scope.

We can create a new FileStream by calling the overloaded procs 
newFileStream() with an already opened file or a filename as
parameter, or we can use openFileStream(). The later raises an
exception when the stream can not be opened, while the former procs
just return nil. We can write and read textual data with the streams
module in a very similar way as we did it with the io module and the 
File data type:

from os import fileExists
import streams

proc main =
  const SN = "NoImportantData" # stream name
  if os.fileExists(SN):
    echo "File exists, we may overwrite important data"
    quit()
  var fstream = newFileStream(SN, fmReadWrite)
  if fstream != nil:
    fstream.write(123, ' ')
    fstream.writeLine(3.1415)
    fstream.setPosition(0)
    let l = fstream.readLine()
    fstream.close()
    assert l == "123 3.1415"
main()

We test again if a file with that name already exists. Then we try to
create a new FileStream by using file mode fmReadWrite, so that we
can write and read from that file. Finally we write two numbers,
which are automatically converted to strings, set the file position
back to the beginning and verify what we wrote by reading it in
again, before we close the stream.

In a very similar way we can write to and read from string streams

import streams
proc main =
  var stream = newStringStream()
  stream.write(123, ' ')
  stream.writeLine(3.1415)
  stream.setPosition(0)
  let l = stream.readLine()
  stream.close()
  assert l == "123 3.1415"
main()

In the example above we do not test if the stream variable is not nil
, as newStringStream() should never fail.

For buffered streams we can call flush() to ensure that the buffer
content (of file streams) is written, similar as we can do it for
plain Files of the io module. Instead of io.endofFile() we use the 
proc atEnd() to test if the current stream position is already at the
end of the stream. Functions getPosition() and setPosition() are
available to query or set the actual position in the stream. While
the io module with its File data type supports for io.setFilePos()
although position modes relative to the actual position or relative
to the file end, streams.setPosition() use always absolute values,
that is positions measured from the beginning of the stream. The 
streams module provides also the low level procs readData() and 
readDataStr() which reads data to a memory region or into a string
and returns the actual number of bytes read to indicate success. And
as for the io module a proc readAll() is available to read all data
of a stream into the returned string variable.

The procs writeLine() writes the passed arguments always as strings.
The overloaded write() procs with varargs arguments write the passed
values as strings and apply the stringify operator $ if necessary.
The same does the writeLine() proc, but it writes a newline character
when all passed variables have been written. One more overloaded
write proc for single string parameters exist.

But for single non string arguments a generic write() proc is used,
which writes numbers (and other data types like boolean types or
single characters) directly in binary form without converting them to
strings.

To read the binary numbers back we can use functions like readFloat64
() which have a well defined return type and read a fixed number of
bytes. Or we can use the generic read() proc which accepts a var
parameter which defines the data type that we intend to read in
binary form. Additional to the various read() procs the streams
module provides a set of peek() procs which reads data in without
moving the actual position in the stream forward. This may be useful
for parsing of files, as we can read the same information multiple
times easily. Internally the peek() functions uses a call of 
setPosition() to save the current position and one more call of 
setPosition() to set back the old position to the initial value, so 
peek() has some overhead.

import streams
from os import getFileSize
proc main =
  const SN = "NoImportantData" # stream name
  var fstream = newFileStream(SN, fmReadWrite)
  if fstream != nil:
    fstream.write("012") # write a 3 byte string
    var pi: float64 = 3.1415
    fstream.write(pi) # write as 8 byte binary blob
    fstream.setPosition(0) # prepare for reading from start
    var i16: int16
    i16 = fstream.peekint16 # read first 2 bytes as int16, do not change actual position
    assert i16 == '0'.ord + '1'.ord * 256
    var i8: int8
    i8 = fstream.readInt8 # read back one byte
    # fstream.read(i8) # does work also
    assert i8 == '0'.ord # char(0) has position 48 in ASCII table
    assert i8 == 48
    var buffer: array[2, char]
    fstream.read(buffer)
    let x = fstream.readFloat64 # read back in binary form
    assert x == 3.1415
    fstream.close()
    assert buffer == ['1', '2']
    assert os.getFileSize(SN) == 3 + 8 # 3 byte string and a float64
main()

In the example above we write a three byte long string and a float64
to the file stream. We call setPosition(0) to read the stream from
the beginning again, and then read in an int16 with the function 
peekint16() without moving the actual position forward, followed from
readInt8(), which moves the actual position one byte forward.
(Instead of readInt8() we could also call read() with variable i8 as
passed var parameter.) Then we read in two bytes and finally the 
float64 value at the end of the stream. Finally we check by use of
the function getFileSize() from the os module if the file has really
the expected size.

The streams module provides many functions, and the possible writing
data as strings or in binary form can make using that module a bit
daunting at first. But most procs have examples in the API docs which
helps you using it.

For reading strings and whole lines the streams module provides
functions like readLine(), peekLine(), readStr() and peekStr() each
in a variant which returns a newly allocated string and one that uses
a passed var parameter to return the string. The variants with var
parameters may be a bit faster, as they can avoid allocating a new 
string when the passed in var parameter has already enough capacity.

References:

  * https://nim-lang.org/docs/streams.html

String Processing

string processing is a wide area. Nim's standard library provides
various modules like strutils, parseutils and strscans for supporting
this task, and external packages supports more advanced operation
like string pattern matching with regular expressions (regex). We
will start with the strutils module, which is one of the mostly used
modules of the Nim standard library. Then we will introduce some more
specialized modules like strscans, parsecvs, parseutils, strformat,
and finally we will give an introduction to use of regular
expressions (regex) and to the use of the parsing expression grammar
(PEG). While we describe generally only modules of Nim's standard
library in this part III of the book, we will make an exception for
regex and PEG. The reason for this is, that regex and PEG use is
closely related to basic string processing and that very powerful
external packages exists, which are not too difficult to use and so
needs not too much introductory explanations.

Whenever we do string processing in Nim we should care a bit for
performance, as some string operations can be slow by design. For
simple tasks we should prefer to use functions from the simple
modules like strutils when possible, and use regex or PEG only when
really necessary or when performance is uncritical. And even when we
use elementary simple functions like a string split, it is generally
good to have a feeling how the requested operations may work.
Whenever string functions return a string as a result, this implies
an allocation, which takes some time and consumes some memory. An
example is the split() operation which return a sequence of multiple 
strings. The split() function is easy to use, so it is often the
first choice when we read in lines of text from files and want to
process it. But as for each section of the split line a string is
allocated, it may be not as fast as desired. In some cases the
compiler may be able to optimize the slitting process, but it may be
also a good idea to think about other ways to extract the data, maybe
by applying procs from the strscans module which can parse lines
directly into passed var parameters avoiding unnecessary allocations.

Remember that Nim strings are value types and have value semantic.
String assignment copies the string content and does not create just
a reference as in some other programming languages. Nim defines also
a string variant called TaintedString, which is generally just an
alias for an ordinary string as long as the taint mode is not turned
on. Functions like io.readLine() return tainted strings which
generally can be used like ordinary strings.

Basic string operations

We discussed Nim's string data type already in part II of the book.
Remember that a string in Nim is a variable size container for ASCII
characters. strings can be plain ASCII strings, or the bytes of the 
string can be interpreted as unicode glyphs. Nim has also a cstring
data type, which was initially introduced to be compatible with the
character arrays used in C libraries as strings, but is now called
compatible string as cstrings do also support the JavaScript backend.
Nim strings can be passed directly to C libs, as a Nim string
contains a Null terminated buffer for the actual data, which is
identical to a C language string. So whenever we convert a Nim string
to a C language string or pass a Nim string to a C library, this is
free of costs, while converting a C string to a Nim string always
means allocating a new Nim string and coping the data content.
Technically for a Nim string s addr s[0] is the C string pointer,
called * char in C language. Whenever we pass strings to C libraries
we have to care for the fact that Nim's garbage collector may
deallocate the string automatically. Most C libs create copies of
passed strings when the libs use the string for a longer time span.
GTK for example does this with text for its widgets. But when the C
lib does not copy the string but use it directly for a longer time,
then it can occur that the Nim code frees the string, as the only one
Nim variable referring to the string goes out of scope, but the C
library still uses the string. For that rare case we may call GC_ref
() on the string to prevent garbage collection, but that may generate
memory leaks then. For the case that C libs create strings, they
provide generally also a function to deallocate the string. When we
use such a C function, it is generally the best solution that we copy
the string from the C lib to a Nim string and immediately deallocate
the C string by a call of the provided free()/dealloc() function. For
most C libs there exist good high level bindings which do not have
this issues, so we generally can use the C libs like pure Nim libs.

Nim' s system module provides already some basic string operation
like accessing single characters by the subscript operator [],
accessing slices of multiple adjacent characters, or joining multiple
strings with the & operator. The overloaded add() functions to append
single characters or other strings to existing string variables are
also provided by the system module.

var s: string = "I like"
s &= " Nim."
s[^1] = '!'
s[0 .. 4] = "We low" # result is: "We love Nim!"

We start above example by assigning to the string variable s a string
literal, then we append one more string literal, and finally replace
the last character and the first five characters by another character
and by another string. Note that by using the slice operator we can
not only replace character ranges, but we can also replace slices of
different length. This way we can also delete ranges in the string by
replacing it with the empty string "".

The system module also defines the stringify operator $, which
converts expressions to the string presentation when we put it in
from of it. procs like echo() apply the stringify operator
automatically on all of its arguments when necessary. And the system
module provides the contains() function which we can use to test if a
string contains a character. Instead of contains we can also use the
in operator.

echo $7
echo "Nim".contains('i')
echo 'i' in "Nim"

The system module also provides the procs newString() and 
newStringOfCap() which are mostly used for optimizing purposes. The
function newString(n) creates a string of length n, but with
uninitialized content. We would have to assign characters to the
positions 0 .. n-1 to create a valid string. Function newStringOfCap
() creates a string with length zero, but with a buffer capacity of n
characters. When we know the needed buffer capacity, or at least a
lower bound of it, it makes sense to create the string with 
newStringOfCap() with optimal buffer size to avoid reallocations. Of
course we could still append more data, Nim would allocate a larger
buffer and copy content.

var s1 = newString('z'.ord - 'a'.ord + 1)
for c in items('a' .. 'z'):
  s1[c.ord - 'a'.ord] = c

var s2 = newStringOfCap(32) # we intend to append not more than 32 characters, but we could do.
for c in 'a' .. 'z':
  s2.add(c)
# s1, s2 is abcdefghijklmnopqrstuvwxyz

Filling in characters into an existing string by use of the subscript
operator [] is faster than appending single characters with the add()
function, because the add() function has to check if the string has
still enough capacity and because add() has to increase the actual 
string length by one for each call.

Note that a single character like 'a' is very different from a string
with only one character like "a". A character in Nim is nothing more
than a single byte, while a string -- even one with only one character
or an empty one -- is an opaque entity with length, capacity and a
pointer to a data buffer. When a single character is sufficient, we
should use that and not a string containing a single character. A
function call like s.add("a") may produce less optimized code than 
s.add('a'), but maybe the compiler optimized the former for us. When
we consider optimization we may wonder if in

var s = "Hello!"
echo s
s = "Bye."
echo s

line 3 allocates a new string or just copies the string literal 
"Bye." in the existing data area. Well we would hope for the later of
course.

Another interesting question is if in

var s = "Result is: "
var x = 12.34
echo s, x
echo s & $x

we should better use line 3 or line 4. Line 3 looks more clear and we
assume that it also would produce better code, as the actual append
operation is avoided.

Often used functions are len() and setLen() to query and set the
length of a string. While len() looks like a function call, it is a
compiler intern function, so calls are fully optimized. So it is OK
to write

for i in 0 .. s.len():
  echo '*'

It is not necessary to introduce a temporary variable like let l =
s.len() to avoid many function calls. In C this is different, as in C
a call of function strLen() would not only imply a function call, but
that function would really have to count all characters up to the
terminating '\x0' as C strings have no length field. The function 
setLen() is generally used to truncate strings. A call like s.setLen
(0) makes s look like a newly allocated string, but its data buffer
can be reused. Reusing strings is generally better for performance
than allocating many new strings. The setLen() function is rarely
used to increase the string length -- in that case an allocation of a
larger data buffer can occur, and the string would still look the
same as initially all the new string positions would still contain
the default binary zero content. We would have to fill in actual
characters by use of the [] subscript operator. To get the lowest and
highest character index of a string we can use s.low and s.high. 
s.low should be always return zero, and high() is identical to len()
- 1, so high() is -1 for an empty string. Note that while calling 
high() and len() on a Nim string has no costs, this may be different
to C strings as these have no length field.

The system module provides the overloaded & operator which we can use
to concatenate chars and strings, the &= operator to append a new 
string to an existing string, and the add() functions to append
characters or strings to an existing string. For best performance we
should try to use always the most simple, "native" operations, at
least as that does not make the code ugly or we know for sure that
the compiler optimizes it for us.

var s = "Ni"
s = s & "m" # maybe not a good idea for optimal performance
s.add('m') # better

The function add() can be also used to add a cstring to a string, and
the JS backend allows even to append one cstring to another one by 
add() calls.

Module system also exports a substr() proc which copies and return a 
slice of a string. Overloads with optional first index with default 0
and optional last index with default s.high exists.

var s = "Hello!"
assert s.substr(2, 3) == s[2 .. 3]

s = "ABC"
echo s[0 .. 5] # fail
echo s.substr(0, 5) # index is clipped to s.high
s[0 .. 5] = "" # fail

And of course == and != operators for string comparison are provided.
To test if a string is empty we can compare with an empty string
literal or test if len() is zero. The later is guaranteed to have
best performance, but the former is a bit shorter and should be not
bad performance wise. Some other languages provide an empty() test
function for this, we may define our own when we really want.

if s != "":
if s.len != 0:

When we pass a string to a proc that does no operations with it,
maybe it only calls echo() or stdout.write() to print it, then it may
have a tiny performance advantage to pass it as cstring. This is
similar as we may pass sequences as openArray to functions, which
also avoids one level of indirection. Also note, that while a Nim 
string is a value type, so we can not test it for nil or return nil
from a proc that shall return a string, this restriction does not
hold for cstrings.

Module stringutils

Module stringutils provides a set of functions (120 currently) and a
few iterators for simple string operations. Using that functions and
iterators is simple in most cases and generally well explained in the
API docs. Remembering which functions exists, their exact name and
the function arguments can be a bit difficult at first. We will
introduce in this section some of the most used or more difficult
functions, and give some warnings when the actual performance may be
not as good as expected.

Performance critical operations are generally that one which has to
allocate new strings or that has to shift many characters, like text
inserting operations. Note that some functions of this module like 
toUpperAscii() work only with the lower and upper ASCII letters. For
unicode operations we may need the unicode module.

In this section we use the term whitespace, which refers to the
invisible characters {' ', '\t', '\v', '\c', '\n', '\f'} (space, tab,
vertical tab, carriage return, new line, form feed). Note that we
have two possible newline characters {'\c', '\n'} that starts a new
line, and that older windows text files may still use the two
character string "\c\n" to start a new line. The character set
{'A'..'Z', 'a'..'z'} is called (ASCII) letters, the set {'0'..'9'}
(decimal) digits and the set {'0'..'9', 'A'..'F', 'a'..'f'} hexdigits
used to represent hexadecimal numbers.

The stringutils module support string interpolation by use of the %
operator.

echo "The $1 programming language is $2." % ["best", "Nim"]
echo "The $# programming languages are $#." % ["most difficult", "C++, Rust and Haskell"]
echo "A $adj programming language is $lang." % ["adj", "low level", "lang", "assembly"]
echo "Let's learn $#" % "Nim!"
echo format("I know $# programming languages.", 2)

We can use $1 up to $9 to mark positions where string n from the
array should be inserted, or just $ to insert the strings in the
order as they appear in the array. We can also use named insert
markers and specify name-value string pairs in the array. For a
single string we can omit the array and pass just as string, and
finally we can use the format() proc to enable stringify for the
parameters.

We mentioned already the useful but performance critical set of split
() functions:

import strutils
let str = "Zero, first, second, third"
var s: seq[string] = str.split(", ")
echo s
echo str.split(", ", 0)
echo str.split(", ", 1)
echo str.split(", ", 3)
echo str.split(", ", 4)

@["Zero", "first", "second", "third"]
@["Zero, first, second, third"]
@["Zero", "first, second, third"]
@["Zero", "first", "second", "third"]
@["Zero", "first", "second", "third"]

We used the split() variant which accepts a string as split marker.
This function accepts one optional parameter for the number of splits
to execute and returns a sequence containing the single strings. The
split marker string, also called separator, is removed from the 
strings. The default value for the number of splits is -1 indicating
that we want a split at each separator position. If we specify a
positive number n, then only n splits are executed and the last
element of the returned sequence will contain the remainder of the 
string. When we specify for the intended splits a value which is
larger than the number of contained split markers, then we get a full
split.

The reason why this function is not very fast is that it has to
allocate a sequence for the return value and a new string for each
split and one more for the last string or the remainder. For the case
that we do need only the first few strings of the split, it is a good
idea to specify the number of actual splits to increase performance.

The strutils module provides overloaded functions which use single
characters as separators or which accepts a set of characters as
separators, so we may split at space, tabulator, comma or semicolon
with {' ', '\t', ',', ';'}. And a function splitWhitespace() is
available to split at whitespace like spaces and tabulators, which
removes all the whitespace between the strings. Notice that the split
() function for strings or single characters does one split for each
separator, so we can get empty strings as result as in

import strutils
let str = "Zero___first_second__third"
let str2 = "Zero first\tsecond           third"
echo str.split('_')
echo str2.splitWhiteSpace()
# @["Zero", "", "", "first", "second", "", "third"]
# @["Zero", "first", "second", "third"]

An interesting behavior of splitWhiteSpace() is that whitespace at
the start or end of a string is just ignored, while the split()
function returns additional empty strings when the string to split
starts or ends with the separator:

import strutils
let str = "_Zero_first_second_third_"
let str2 = "   Zero first\tsecond           third   "
echo str.split('_')
echo str2.splitWhiteSpace()
# @["", "Zero", "first", "second", "third", ""]
# @["Zero", "first", "second", "third"]

Another function is splitLines() which splits a string at CR, LF or
CR-LF characters.

For these splitting functions also iterator variants exists, which
behave like the functions with same name. When we limit for the
iterators the number of splits to perform, we may get as last
returned value the remained string. You may consult the API docs for
details if necessary.

Functions with names rsplit() are also available which behave like 
split() but start the splitting process from the end of the string,
so we can get the file extension of a file name with something like 
filename.rsplit(1)[^1].

The functions removePrefix() and removeSuffix() can sometimes help to
avoid expensive split operations. There is an overloaded function
that removes all single characters, all characters from a set or a
single string:

import strutils
var s = "NNimmm Language"
s.removePrefix('N'); echo s # immm Language
s.removePrefix({'N', 'i', 'm', ' '}); echo s # Language
s.removePrefix("Langu"); echo s # age
s. removeSuffix("ge"); echo s # a

Other useful functions are startsWith() and endsWith() which accept a
single character or a string and return a boolean value:

import strutils
var s = "Nim Language"
echo s.startsWith('N') # true
echo s.startsWith("Nim") # true
echo s.endsWith('e') # true
echo s.endsWith("Programming") # false

An efficient function similar to find() is continuesWith() used like 
"Nim Language".continuesWith("Lang", 4) which tests is a string
contains a substring at position n.

We can use the join() proc to join strings in arrays or sequences to
single strings with an optional glue string. Join() works also when
the elements are not already strings -- in that case the stringify
operator is applied first:

import strutils
var a = ["Nim", "Rust", "Julia"]
var s = a.join(", ")
echo s
echo s.split(", ").join("; ")
echo [1, 2, 3].join("; ")
# Nim, Rust, Julia
# Nim; Rust; Julia
# 1; 2; 3

The overloaded find() functions accepts optional start and end
positions and returns the index position of the first match or -1
when the search gave no result:

import strutils
var s = "Nim Language"
echo s.find('L') # 4
echo s.find({' ', 'a'}) # 3
echo s.find("age") # 9
echo s.find("Rust") # -1

The contains() function can be used to test if a character, a set of
characters or a substring in contained in a string. Instead of 
contains(s, sub) we can write sub in s. Note that a function variant
for single characters is defined in the system module.

The replace() function can be used to replace all occurrences of a
character or a substring in a string and to return the new string. We
can use replace() with an empty replacement to delete a substring.
The also available delete() function is used to delete a range
specified by two indices in place.

Additional a replaceWord() function exists, which does only replaces
whole words, i.e. words that are surrounded by word boundary
characters like spaces, tabulators or newlines.

The function multiReplace() is a variant that can replace multiple
substring/replacement pairs passed as tuples in one pass.

The function strip() can be used to remove multiple characters from a
set at the start and the end of a string. The default character set
is whitespace, and per default strip removes characters at both ends
of the string. The function stripLineEnd() removes a single line end
marker as \r, \n, \r\n, \f, \v from the end of the string, but only
once.

Sometimes useful is the boolean function isEmptyOrWhitespace() which
checks if a string is empty or contains only whitespace. Also useful
can be the function repeat() which returns a string that contains the
passed character or the passed string n times, and the function 
spaces() which returns a string containing only n spaces.

For single character tests we have functions like isDigit(), 
isUpperAscii(), isLowerAscii(), isSpaceAscii, isAlphaNumeric().
Function isDigit() test for characters '0'..'9', isUpperAscii() for
'A'..'Z', islowerAscii() for 'a'..'z', isSpaceAsccii() for ASCII
whitespace (' ', '\t') and isAlphaNumeric() test for lower or upper
case ASCII letter or a decimal digit.

Function toLowerAscii() convert all the characters 'A'..'Z' to lower
case, and toUpperAscii() converts all the characters 'a'..'z' to
upper case. The function argument can be a string or just a single
character. With capitalizeAscii() we can convert the first ASCII
character of a string to upper case.

The overloaded functions count() can be used to count the characters
or substrings in a string, and countLines() is available to count the
number of lines, where lines are separated by CR, LF or CR-LF.

Sometimes we may also need functions like formatFloat(), 
formatBiggestFloat() or formatEng() to format float numbers for
output purposes. You would have to consult the strutils API docs for
all the format details. An intToStr() function with an argument to
specify the minimal string length is also available. The string may
get leading zeros for alignment.

Finally some important functions are the parsing functions like 
parseFloat() or parseInt() which converts strings to float or integer
numbers. Both raises an exception when the string does not contain a
valid number.

The strutils module contains some more not that often used function,
like functions to convert data to hexadecimal, octal or binary
representation, or to parse numbers back from that string
representation into numbers. Other functions like align(), center()
and indent() are available for string positioning. We will not try to
describe these seldom used functions here, as it is hard to remember
the detailed behavior. You should skim the API docs and consult them
when you need one of the exotic functions or when you have forgotten
how to use a concrete function.^[37]

Module parseutils

The module parseutils provides a set of functions for efficient and
fast parsing of strings. The functions avoids the allocation of new 
strings by passing back results in var string parameters and by
returning the number of processed characters. The module parseutils
is a good choice when we need efficient parsing of strings and the
input strings have a simple structure. For more complicated input
data we may have to use RegEx or PEGs. Let us assume that we have a
set of library names which includes the version numbers, but we need
the plain names. The function parseWhile() is a good candidate for
this task:

import parseutils
var libs = ["libdconf.so.1.0.0", "libnice.so.10.11.0", "libwebkit2gtk-5.0.so.0.0.0"]
var l = newStringOfCap(128)
for s in libs:
  echo s.parseUntil(l, {'.', '-'})
  echo l
  echo s.parseWhile(l, {'a'..'z'})
  echo l

First we allocate a string with enough capacity, so that the parse()
functions can use it without having to do allocations. As we want to
receive the plain names, using parseWhile() with a charset as last
parameter may be a possible solution. But as we see this will not
really work for webkit2gtk which contains a digit in its name:

8
libdconf
8
libdconf
7
libnice
7
libnice
13
libwebkit2gtk
9
libwebkit

We can fix this by passing the extended char set {'a'..'z', '0'..'9'}
to parseWhile() or by use of parseUntil() with a character set that
does not belong to a name. Both functions return the number of
processed characters and provide the captured string in the passed
var parameter. Not the we can use the slice operator .. to specify
character ranges for the charset parameter when the characters build
a continues sequence in the ASCII table.

A related function is skipUntil(), which we may use when we are more
interested in the version numbers after the name:

  let p = s.skipUntil({'.', '-'})
  echo s[p .. ^1]

All these functions accept an optional start parameter as last
argument. A common use case is to use an integer position variable
initialized with zero, which we increase by the returned value so
that the parsing can continue at the current position in the string.
The next example will use this strategy. For parseUntil() overloaded
functions are available which gets not a char set but a single
character or a substring as parameter. These functions stops parsing
when the character or the substring is found and return that
position.

Functions like parseInt() and parseFloat() can be used to extract
numbers from strings:

import parseutils
var s = "In the year 2020 I gain 2.5 kg more fat."
var year: int
var value: float
var p = s.skipUntil({'0'..'9'})
p += parseUtils.parseInt(s, year, p)
p += s.skipUntil({'0'..'9'}, p)
p += parseUtils.parseFloat(s, value, p)
echo year, ": ", value # 2020: 2.5

In above example we used the module prefix as strutils contains also
a parseInt() and a parseFloat() function. The functions parseBin(), 
parseOct() and parseHex() behave in a similar way. Returned is the
number of processed characters. We add the returned value to the
start position so that parsing can continue at the new position.

There are some more functions available in this module, which we will
not discuss further. It is enough that you know that this module
exists and provides some efficient parsing functions. Whenever you
should really need one of these procs you would have to consult the
API documentation for details.

Module strscans

The strscans module provides a scanf() macro which can be used to
extract substrings from textual user input. The content of the
substrings is automatically converted to Nim variables of matching
data types.

Processing of well defined strings is easy, and with user defined
matcher functions even text input with less strict shape can be
processed.

Let us start with a simple example: We may have to create a program
where the user should be able to create rectangles by entering the
coordinates of two opposite corners in the form

Rect x1,y1,x2,y2

import std/strscans

var x1, y1, x2, y2: float
var name: string

let input = "Rect 10.0,20.0,100,200"

if scanf(input, "$w $f,$f,$f,$f", name, x1, y1, x2, y2):
  echo name, ' ', x1, ' ', y1, ' ', x2, ' ', y2 # Rect 10.0 20.0 100.0 200.0

The first parameter for the scanf() macro is the user input string,
and the second parameter is a pattern string that specifies how the
input string should be processed. The following parameters are
variables that gets the results of the input evaluation. The second
parameter has some similarity with a regular expression. The letters
after the dollar sign specifies the data type of the substrings, $i
or $f ask to process an integer or a floating point number, and $w
requests to process an ASCII identifier. Other characters are
captured verbatim, that is the space character after $w has to match
a space in the input string, and the comma characters that separates
the $f has to match commas in the input string. The scanf() macro
supports capturing of some more data types, i.e. $c for an arbitrary
character or $s for optional white space. The optional white space is
not captured, just ignored. With the use of $s our program allows
already a more flexible input string:

let input = "Rect10.0,    20.0,100,200"

if scanf(input, "$w$s$f,$s$f,$s$f,$s$f", name, x1, y1, x2, y2):
  echo name, ' ', x1, ' ', y1, ' ', x2, ' ', y2 # Rect 10.0 20.0 100.0 200.0

To allow to process even more flexible input strings it is possible
to use user definable matchers in form of Nim procs with a well
defined parameter signature. There are two different types of matcher
procs supported -- matchers to just skip a part of the input string,
and capturing matchers. For the next example we will use a proc which
can skip various separators like comma, semicolon or white space. And
we will use a capturing matcher proc for the object name.

import std/strscans

proc sep(input: string; start: int; seps: set[char] = {' ',',',';'}): int =
  while start + result < input.len and input[start + result] in {' ','\t'}:
    inc(result)
  if start + result < input.len and input[start + result] in {';',','}:
    inc(result)
  while start + result < input.len and input[start + result] in {' ','\t'}:
    inc(result)

proc stt(input: string; strVal: var string; start: int; n: int): int =
  if input[start .. start + "Rect".high] == "Rect":
    strVal = "Rect"
    result = "Rect".len

var x1, y1, x2, y2: float
var name: string

let input = "Rect 10.0    ;20.0,100  ,  200"

if scanf(input, "${stt(0)}$s$f$[sep]$f$[sep]$f$[sep]$f", name, x1, y1, x2, y2):
  echo name, ' ', x1, ' ', y1, ' ', x2, ' ', y2 # Rect 10.0 20.0 100.0 200.0

The use of our user definable matcher sep() allows to separate the
four numbers with a colon or a semicolon with arbitrary leading or
trailing white space, or with only white space. Multiple colons or
semicolons between two numbers resulting from a typo would be not
permitted.

The signature for this matcher proc has this shape:

proc sep(input: string; start: int; seps: set[char] = {' ',',',';'}): int =

The first parameter is the string to process, and the second
parameter is the position at which the processing should start. (Or
in other words, the second parameter of integer type is the actual
position in the input string, that position moves during the whole
capering process from the start of the input string to its end. The
end may not be reached if the capering fails at some point.) The last
parameter has always the type set[char] with a default value
indication which characters that proc can process. Actually a default
value seems to be necessary, but the actual value seems not to
matter. The proc returns the number of characters that should be
skipped. Zero is a valid return value, so we can support optional
separators. In most cases separators are necessary to process the
input string, but we can imagine input formats where separators are
optional, e.g. when an integer number is followed by a name. A name
never starts with a digit, so the boundary between the two values is
well defined. This none capturing matcher proc is called by use of $
[sep] in the pattern string.

The signature of the capturing matcher proc has this shape:

proc stt(input: string; strVal: var string; start: int; n: int): int =

That proc also gets as parameters the input string and the start
position, and has to return the number of processed characters. But
additional a var parameter of arbitrary data type is used to return
the result of the capture, and the last parameter with arbitrary data
type can influence the capturing process. One possible use of the
last parameter is to use an integer value to limit the maximum number
of characters to process. This proc is called in the pattern string
by using curly braces like ${stt(0)}.

Scanf() returns true when all the parameters match, for that case all
the passed in variables get assigned a value. Currently scanf() does
not support the capturing of optionally data, as the whole processing
stops when one capture fails, i.e. when $i is used to request the
capture of an integer value but the input string does not contain
decimal digits at the current capture position. In the same way the
whole capturing process stops when a user defined capturing matcher
returns zero as no capturing is possible. So intermediate optional
arguments are currently not supported. When the processing stops due
to missing arguments, scanf() returns false, but the already
processed captures still have a valid value assigned. In this way we
can use at least optional trailing arguments.

As next example for the use of the scanf() macro we will give a real
world example: A simple CAD (Computer Aided Design) program has a PCB
(Printed Circuit Board) mode, in which the user can create new PCB
pads by entering the pad data in a text entry widget. A PCB pad is a
rectangular shaped copper field which has an associated number and a
name and maybe rounded corners. The user should be able to enter two
2D coordinates, the corner radius, an optional x/y translation for
the next pad of same size, followed by the number of pads to create
and the pad number and name. That is

pad x1 y1 x2 y2 r dx dy n num name

The first five arguments are mandatory, the rest is optional with
default values. The user should be able to separate the arguments
with white space or with a colon or a semicolon. Additional the
values x2 and y2 can be preceded with a + character to indicate that
the x2, y2 tuple is not an absolute coordinate value but the width
and high of the pad.

A program fragment to process this form of user input may look like

import std/strscans

proc jecho(x: varargs[string, `$`]) =
  for el in x:
    stdout.write(el & " ")
  stdout.write('\n')
  stdout.flushfile

proc stt(input: string; strVal: var string; start: int; n: int): int =
  if input[start .. start + "pad".high] == "pad":
    strVal = "pad"
    result = "pad".len

proc pls(input: string; plusVal: var int; start: int; n: int): int =
  if input[start] == '+':
    plusVal = 1 # bool
    result = 1

proc sep(input: string; start: int; seps: set[char] = {' ',',',';'}): int =
  while start + result < input.len and input[start + result] in {' ','\t'}:
    inc(result)
  if start + result < input.len and input[start + result] in {';',','}:
    inc(result)
  while start + result < input.len and input[start + result] in {' ','\t'}:
    inc(result)

proc plus(input: string; plusVal: var int; start: int; n: int): int =
  result = sep(input, start)
  if input[start + result] == '+':
    plusVal = 1 # bool
    result += 1

var st: string
var x1, y1, x2, y2, dx, dy: float
var px2, py2: int # bool
var n: int
var number, name: string

(st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name) = ("pad", NaN, NaN, 0, NaN, 0, NaN, NaN, NaN, 0, "", "") # defaults

var res: bool
var input = "pad 10.0, 10   12 +12.0 ;20 0 8 Num Name"

# unfortunately the input start with "pad" is needed for unpatched strscan!

# using the pls matcher, this fails when there is no '+'
res = scanf(input, "${stt(0)}$[sep]$f$[sep]$f$[sep]${pls(0)}$f$[sep]${pls(0)}$f$[sep]$f$[sep]$f$[sep]$i$[sep]$w$[sep]$w", st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)
jecho(res, st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)

# using the plus matcher, so the '+' is optional
res = scanf(input, "${stt(0)}$[sep]$f$[sep]$f${plus(0)}$f${plus(0)}$f$[sep]$f$[sep]$f$[sep]$i$[sep]$w$[sep]$w", st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)
jecho(res, st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)

input = "pad 10.0, 10   12 +12.0" # test with missing optional values
(st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name) = ("pad", NaN, NaN, 0, NaN, 0, NaN, NaN, NaN, 0, "", "") # defaults
# using the plus matcher, so the '+' is optional
res = scanf(input, "${stt(0)}$[sep]$f$[sep]$f${plus(0)}$f${plus(0)}$f$[sep]$f$[sep]$f$[sep]$i$[sep]$w$[sep]$w", st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)
jecho(res, st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)

When we compile and run this program we get this output:

false pad 10.0 10.0 0 nan 0 nan nan nan 0
true pad 10.0 10.0 0 12.0 1 12.0 20.0 0.0 8 Num Name
false pad 10.0 10.0 0 12.0 1 12.0 nan nan 0

The first scanf() call uses the sequence $[sep]${pls(0)} which fails
when the float value has no leading + sign, so this call is of no
real use. The second and third plus() proc processes the separators
as well as the optional + character, so that proc has never to return
zero, and the capturing process continues. For the last scanf() call
we give as input only five values, so scanf() returns false, but the
first five values gets assigned values, and the rest has default
values. One restriction of above code is, that we have always to
start the input string with pad, otherwise the processing stops
immediately. As scanf() does not support the capture of boolean
values, we uses the integer data type for the variables px2 and py2.
The value zero means that there is no + prefix, and 1 indicates that
there is a plus prefix.^[38]

The next tiny example shows how we can use the last parameter of the
user defined matcher proc to control the matching process:

import strscans

proc ndigits(input: string; intVal: var int; start: int; n: int): int =
  var i, x: int
  while i < n and i + start < input.len and input[i + start] in {'0'..'9'}:
    x = x * 10 + input[i + start].ord - '0'.ord
    inc(i)
  # only overwrite if we had a match
  if i == n:
    result = n
    intVal = x

var input = "1234"
var a, b: int

if scanf(input, "${ndigits(2)}$s${ndigits(2)}$.", a, b):
  echo "Input is OK:", a, " ", b

We want to capture two integer values, each with one or two decimal
digits. By passing the upper limit of digits to the ndigits() proc,
we get the intended result even when the user does not separate the
two numbers with white space. Additional we have used $. at the end
of the pattern string. The $. matches only when the end of the input
string is reached, so that scanf() would return false if there are
more characters in the input string left.

The latest version of the strscans module provides also a variant of
the scanf() macro called scanTuple() which returns a tuple. We could
use it in this way in our example above:

let (res, a, b) = scanTuple(input, "${ndigits(2)}$s${ndigits(2)}$.", int, int)
echo res, " ", a, " ", b

So we have not to declare the capering variables in advance. The
final result of the scan process is returned in an additional boolean
variable. When we use user defined matchers as above, we have to
specify the data types of the returned values as additional
parameters after the pattern string.

Additional the strscans module provides a scanp() macro which works
somewhat similar as PEG or RegEx libraries. We will not try to
explain the scanp() macro, as its use may be too difficult for a
beginner book. And when we really have to process text strings with
regular expression grammars, then we can use the available RegEx or
PEG modules which have no restrictions and work for Nim in a similar
way as for other programming languages. We will introduce the Nim
RegEx and PEG modules later in the book -- maybe we will compare the
scanp() macro there.

Module strformat

With the fmt() macro from the strformat module we can format and
interpolate strings similar as Python3 with its f-strings.

import strformat
from strutils import `%`
var lang = "C"
var year = 1972
stdout.write "The programming language ", $lang, " was created in ", $year, ".\n"
echo "The programming language " & $lang & " was created in " & $year, "."
echo "The programming language $# was created in $#." % [$lang, $year]
echo fmt"The programming language {lang} was created in {year}."
# The programming language C was created in 1972.

From the four ways to print some text the last one with fmt() is the
shortest and maybe the cleanest. As fmt() is a macro which is
processed at compile time there is no unnecessary run-time overhead
involved. A small restriction of fmt() is, that it's argument is
regarded as a generalized raw string literal. So we can not use
escape sequences like "\n" in the string literal. But the strformat
API docs mention various solutions for this: We can use the unary &
operator instead of the fmt() call, or we can use the notations {'\
n'}, fmt() or "".fmt.

import strformat
var lang = "Fortran"
var year = 1957
stdout.write &"The programming language {lang} was created in {year}.\n"
stdout.write fmt"The programming language {lang} was created in {year}.{'\n'}"
stdout.write fmt("The programming language {lang} was created in {year}.\n")
stdout.write "The programming language {lang} was created in {year}.\n".fmt
# The programming language Fortran was created in 1957.

The fmt() macro works also with multi-line raw strings:

import strformat
echo fmt"""This is a {1 + 1 + 1} lines
multiline
string."""

The fmt() macro accept a few directives for the formatting of integer
and floating point numbers like

import strformat
var pi = 3.1415
var people = 123
echo fmt"The number{pi:>8.2f} is called PI by {people:>8} people."

The basic pattern is, that the numeric variable is followed by a
colon and the total number of desired characters. We can precede the
number with a < or > to indicate left or right alignment, and for
floating point numbers we can use a notation similar as used for
printf() in the C language: n.mf stands for a float formatted with n
characters total and m decimal places. As in C we can use e instead
of f to indicate scientific notation. If the value which specifies
the total number of characters starts with a zero digit, then the
formatted number uses zeros instead spaces for leading digits. And X
after the colon generates hexadecimal value for integer numbers.

A useful property of the fmt() macro is, that we can put an equal
sign into the curly braces to get the initial expression both as 
string and as interpolated value, as in

import strformat
var pi = 3.1415
echo fmt"{2 * pi = }" # 2 * pi = 6.283

This is similar as the dump() macro from the sugar module and is
mostly used for debugging purposes.

To use curly braces as literals in a fmt() argument we can use
character literals with a backslash as we did to include a newline
character, or we can use an extended fmt() macro with two additional
arguments which specifies the two characters that should be used
instead of {} to mark the expression which should be interpolated:

import strformat
echo fmt"{2} curly braces {'\{'} {'\}'}." # 2 curly braces { }.
echo "three time three is <3 * 3>".fmt('<', '>') # three time three is 9

We will not try to explain all these various formatting options in
detail, as it is really hard to remember. It is enough that you know
that these options exists, so you can consult the API docs for
details when you would need it.

References:

  * https://nim-lang.org/docs/strutils.html

  * https://nim-lang.org/docs/parseutils.html

  * https://nim-lang.org/docs/strscans.html

  * https://nim-lang.org/docs/strformat.html

  * https://en.wikipedia.org/wiki/Regular_expression

  * https://en.wikipedia.org/wiki/Parsing_expression_grammar

  * https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions

Arrays and Sequences

Together with strings arrays and sequences are the most important
built-in containers for the Nim language. While arrays have a fixed
size defined already at compile time, sequences are like strings of
dynamic size and can grow when we append more elements. As arrays
have fixed size they can be allocated on the stack, while due to the
dynamic size of sequences the actual data buffer has to be allocated
on the heap. We explained some details about sequences already in
part II of the book. One important aspect was that sequences use a
continues data buffer with a fixed capacity to store the actual
elements. When that data buffer is fully occupied, and we try to add
more elements, then a new larger buffer is allocated on the heap and
the contained elements have to been copied from the old to the new
larger buffer before the old buffer can be deallocated.

When we pass arrays or sequences to procs, then we can use the
special data type openArray when we define the proc to allow passing
both arrays and sequences. Note that this is very different from
generic procs: When we define a generic proc, then the compiler
creates a new proc instance for each of the generic data type that we
use, so when we call a generic proc which accepts floats and signed
and unsigned integers, and we call it a few times with float and with
signed integer arguments, then the compiler has to create two
distinct instances of the proc. For openArray parameters all the time
only one proc instance is necessary as sequences behave like arrays
in many ways. Both use a continues block of memory where the elements
are stored, and the position of an entry is given by the start
address of this memory block and an offset given by the index
multiplied with the size of an array element. So when passing the
actual parameter to the proc, the compiler passes the array and the
data section of the sequence in the same manner. Both can be passed
by copy or by address. The compiler passes also the actual size, the
lower index is always zero for openArrays. Of course when we pass a 
seq as openArray, there are some restrictions, e.g. we could not add
elements in the proc as the passed variable behaves like an array.

Memory layout of sequences and strings is very similar, both have
length, capacity and a data buffer on the heap, and some procs that
work on the data structure has the same names like add(), len() and 
setlen() and both support operators like [], & and .. for access to
single elements, and for concatenation and slicing.

Some often used functions and operators for sequences and arrays are
defined in the system module, like creating new sequences, converting
arrays to sequences, joining sequences or adding elements to it.
Other important functions, operators and iterators are defined in the
sequtils module which we describe in the next section.

var s0: seq[int]
var s1: seq[int] = newSeqOfCap[int](2)
var s2: seq[int] = newSeq[int](2)

s0.add(3)
s0.add(5)
s1.add(3)
s1.add(5)
s2[0] = 3
s2[1] = 5
echo s0; echo s1; echo s2 # @[3, 5] for each

We can initialize sequences by a call of newSeq(), newSeqOfCap() or
not at all. When we use newSeq(n) we get a seq with n elements
initialized to binary zero each, and we then can just overwrite the
elements by use of the subscript operator [] which is faster than
appending elements with add() to an empty seq. With newSeqOfCap() we
can allocate a seq of size zero but with a buffer size of that
specified capacity. We can append elements by calling the add()
function, and as long we append not more elements as specified in the
newSeqOfCap() call we can avoid reallocations of the internal seq
buffer. When performance in not that critical we can just use a
uninitialized seq and add() elements -- when the default capacity is
exhausted a reallocation occurs, generally with doubled data buffer
size.

We can use the overloaded add() proc to append single elements or to
append an whole array to a seq, and the & operator is available to
join two sequences:

s0.add([7, 9])
s0 &= s1
s1 = s0 & s2
echo s1 # @[3, 5, 7, 9, 3, 5, 3, 5]

We can use len() to query the length of a seq and setLen() to set a
new length. In most cases setLen(n) is used to shorten a seq, that is
to keep the first n elements, but we can also use setLen() to
increase the length of a seq. In that case the new entries get the
value binary zero as default and we can use the subscript operator []
to fill in actual content. Increasing the length with setLen() may
cause a reallocation if the current capacity is not sufficient.
Functions low() and high() are available to get the lowest and the
highest index position of an array or a seq. As arrays can have
negative indices, low() can be less than zero for arrays, but for
sequences and openArray proc parameters low() is always zero.

Module system provides also the @ array to seq operator:

var i = 7
var j = 9
var s0 = @[1, 2]
s0 = s0 & @[i, j] # don't use this variant!
s0.add([i, j]) # faster
echo s0 # @[1, 2, 7, 9, 7, 9]

Note that the code in line 4 would be really slow as that would have
to allocate a temporary seq. Using add() to add a temporary array to
the seq should be faster.

Slicing is also supported by the system module:

var s0 = @[1, 3, 5]
var s1 = s0[1 .. 2] # @[3, 5]
for el in s0[1 .. 2]: # may create a copy of the seq
  echo el

Note that line three may create a temporary copy of the sequence,
which may be not that nice for optimal performance. We discussed that
topic already in part II of the book, Nim 2.0 may improve the
situation further by introducing views which create no copies. Beside
the s[a .. b] slice operator which includes the elements at position 
a and b, there is s[a ..< b] which does not include position b and
and s[a .. ^b] where position b is taken from the end of the seq or 
array, e.g. ^1 is the last position, ^2 the second last.

For deleting elements from a sequence we have del() which replaces
the element at the specified position with the last element of the 
seq and reduces the seq length by one, and the delete() function
which shifts all elements after the specified position one step
forward. Obviously the later is slower, but it preserves the order of
elements. The function pop() deletes and returns the last item of a 
seq. For using pop() on an empty seq we may expect a raised
exception. With insert we can insert an item at the specified
position by moving all the elements after this position upwards. Note
that del(), delete(), pop() and insert() are not available for arrays
.

Comparison of two arrays or sequences by the == operator returns true
when the length as well as all contained items matches.

With the function contains() we can test if an item is contained in a
seq or an array. We can also use the operators a in b and a notin b
instead. The elements are tested from the start of the container
until a match is found or the last position in the container is
reached, so this is a O(n) operation.

Module sequtils

This module defines some useful procs, iterators and templates for
working with arrays and sequences. Some functions of module sequtils
use an generic openArray parameter and so can be used for strings as
well. While the max() and min() procs are available from the system
module, the minIndex() and maxIndex() procs are provided by sequtils:

import sequtils
var s = @[7, 3, 5]
echo s.min, " ", s.max # 3 7
echo s.minIndex, " ", s.maxIndex # 1 0

Sometimes we may need a minmax() proc which gives us both values, but
that one is currently not available. We have to create it our self if
needed, when performnce is not that critical we can call min() and 
max() separately.^[39]

The functions minIndex() and maxIndex() as well as count() or 
deduplicate() can work with strings also:

import sequtils
var s = @[3, 5, 1, 7, 3, 3, 5]
echo s.count(5) # 2
echo s.deduplicate # @[3, 5, 1, 7]
echo "abc".maxIndex() #2

The function name deduplicate() may be irritating, as the function
does not work in place, we may expect the name deduplicated() as for 
sort() and sorted(). Other programming languages use the name uniq()
instead. Deduplication is easy when the elements are sorted, as for
that case we can just iterate over the seq and ignore equal adjacent
items. That is why deduplicate() accepts an optional boolean
parameter indicating if the seq is sorted. If the seq is not sorted,
it may be necessary to create a temporary set to store the already
seen items so that they can be ignored the next time when they occur
again in the seq.

The function concat() can join multiple sequences (yes only
sequences, it does not work with arrays currently), maybe that is
more efficient than using the & operator. And insert() can insert a
new value at a position by shifting following items, and delete()
allows to remove a range of items from the seq when we specify two
index positions:

import sequtils
var s: seq[int]
s = concat(@[1, 2], @[3, 4], @[5, 6])
echo s # @[1, 2, 3, 4, 5, 6]
echo @[1, 2] & @[3, 4] & @[5, 6] # @[1, 2, 3, 4, 5, 6]
s.insert(7, 2)
s.delete(3, s.high)
echo s # @[1, 2, 7]

Proc repeat() is used to create a seq which contains a single value
multiple times, and cycle() repeats the items of an existing seq
multiple times:

import sequtils
echo 2.repeat(4) # @[2, 2, 2, 2]
echo @[1, 2].cycle(3) # @[1, 2, 1, 2, 1, 2]

A bit more complicated but really useful are functions like map(), 
filter(), keep() and the corresponding ... It() templates:

import sequtils, sugar
var s = (0 .. 9).toSeq
echo s.map(proc(x: int): int = x * x) # always @[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
echo s.map(x => x * x) # from sugar module
echo s.mapIt(it * it)

echo s.mapIt("*" & $it & "*") # @["*0*", "*1*", "*2*", "*3*", "*4*", "*5*", "*6*", "*7*", "*8*", "*9*"]

echo s.filter(proc(x: int): bool = (x and 1) == 0) # both @[0, 2, 4, 6, 8]
echo s.filterIt((it and 1) == 0)

The map variants return a new seq, with an operation performed on all
items. The returned seq can have a different base type. In line 4 we
used the = operator from the sugar module for a simpler notation

The filter() variants apply a proc with boolean result type on the 
seq items and return the items for which the result is true.
Remembering if the elements for which the procs gives a true result
are returned or removed from the initial seq may be not easy. It
helps to remember that filter() behaves like the keepIf() procs
 -- items with positive result survive.

import sequtils, sugar
var s = (0 .. 9).toSeq
var s1 = s
s1.apply(x => x * x) # @[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
s1 = s
s1.keepIf(proc(x: int): bool = (x and 1) == 0) # @[0, 2, 4, 6, 8]

In above code we used toSeq() with a slice argument to create an
initial sequence with continues integers from the slice. The apply()
function performs a transformation operation on all the items, and 
keepIf() preserve only the items for which a boolean predicate
evaluates to true.

Two useful predicate functions are any() and all() to check if at
least one item fulfills a condition or if all items fulfill a
condition:

import sequtils
var s = (0 .. 9).toSeq
echo s.all(proc(x: int): bool = x < 10) # true
echo s.any(proc(x: int): bool = x * x == 25) # true

With zip() we can join the items of two sequences to tuples, and with
unzip() we can separate the tuple items again in two separate
sequences:

import sequtils
var s = (0 .. 9).toSeq
var s1 = s.mapIt(it * it)
var z = zip(s, s1)
echo z
echo z.unzip
# @[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25), (6, 36), (7, 49), (8, 64), (9, 81)]
# (@[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], @[0, 1, 4, 9, 16, 25, 36, 49, 64, 81])

Finally sometimes the templates foldl() and foldr() can be useful for
folding a sequence, that is to generate one final value from all the
items. The fold templates uses the variables a and b to generate the
result, a + b would sum all the items. Foldl() performs the operation
from left to right, returning the accumulation and accepts an
optional start value, while foldr() starts from right, i.e. with the
item at the end of the seq.

import sequtils
var s = (0 .. 9).toSeq
echo s.foldl(a + b, 100) # 145
echo s.foldr(a + b) #45

The sequtils module contains some more procs, templates and macros
that are not needed that often. It would not make much sense to
mention all of them here, as it is already not easy to remember the
ones that we have introduced above. You should skim the sequtils API
docs from time to time to remember what is available.

Maybe you have missed the difference of two sequences which some
other programming languages provide:

[1, 2, 3, 4, 1, 5] - [2, 4] # [1, 3, 5]

Well that is an expensive operation, O(n^2) if implemented in a naive
way, as it may iterate for all items in the second seq over the whole
first seq to remove the unwanted items. A better approach is to
convert the items in the second seq to a temporary (hash) set to
allow faster query:

A fast on the fly solution is this, as suggested by someone in the
Nim forum:

import sequtils, sets, sugar
let a = [1, 2, 5, 2, 9, 7, 0]
let b = [7, 4, 1, 10, 7]
let bSet = b.toHashSet()
echo a.filter((x) => x notin bSet) # @[2, 5, 2, 9, 0]

When we should need this operation often, we may define our own proc
like

# https://ruby-doc.org/core-2.6/Array.html#method-i-2D
# Array Difference
import sets

proc `-`*[T](a, b: openArray[T]): seq[T] =
  let s = b.toHashSet
  result = newSeq[T](a.len)
  var i = 0
  for el in a:
    if el notin s:
      result[i] = el
      inc(i)
  result.setLen(i)

proc `-=`*[T](a: var seq[T]; b: openArray[T]) =
  let s = b.toHashSet
  var i = 0
  var j = 0
  while i < a.len:
    if a[i] notin s:
      a[j] = a[i]
      inc(j)
    inc(i)
  a.setLen(a.len - (i - j))

proc main =
  let a = [1, 2, 5, 2, 9, 7, 0]
  let b = [7, 4, 1, 10, 7]
  echo a - b
  echo b - a

  var x = @a
  x -= b
  echo x

main()

@[2, 5, 2, 9, 0]
@[4, 10]
@[2, 5, 2, 9, 0]

Note that this preserves the order in the first seq, which is often
requested. If order is not important, then we could convert both
sequences to a set and build the set difference -- but when order does
not matter we may use sets instead of sequences from the beginning.

Maybe you missed also a shift() function which other container types
or programming languages may provide, that is used similar to pop(),
but deletes and returns the the first item of a seq? Well it should
be obvious why a shift() is not provided by default and why such a
function would generally be avoided -- the function name gives you
already a good hint. And if you should need such a function it should
be no problem to implement one if efficiency is really not important.
But maybe in that case it would be better to use a different
container type, maybe the double-ended queue provided by the deques
module.

References:

  * https://nim-lang.org/docs/sequtils.html

  * https://forum.nim-lang.org/t/7753#49189

Random Numbers

Most computer programs work fully deterministic, that is one input
data set generates exactly one well defined output data set. This
behavior is not desired when we create games or simulations: The
actions of computer controlled characters should not do exactly the
same again and again when we restart the game, and maybe the computer
generated landscape in the game should look different when we restart
the game too.

To generate such an unpredictable behavior we use random number
generators, which can generate sequences of random number of integer
or float type. The most important property of true random numbers is
that we can not predict the next one from the sequence of all values
seen before. Random number sources have no memory! Children often
think they have. If a child got value six in a dice game tree times
in sequence, its often assumes that it is extremely unlikely that
next roll will give again the value 6. But as the dice has no memory,
chance to get a specific number is always 1/6, as we have 6 possible
values all with same probability. At least when the dice is not
manipulated. Another important property of random numbers is the
distribution of the possible values. For most random number sources
we would expect a uniform distribution of all possible values: For a
dice with numbers 1 .. 6 we would expect that we get all these
numbers with nearly the same total quantity when we roll the dice for
a long time again and again. But of course not all random quantities
are distributes uniformly. The distribution can have different
shapes. An important non uniform distribution is the Gauss
distribution, where the final result depends on many random
decisions. So an average value is more likely than extreme values.
You may know the marble nail board with multiple slots as an example:
Marbles are thrown in at the top, and whenever they hit a nail, they
get distracted to the left or right.

To built a perfect random generator we would have to use some
physical noise sources like photons emitted by a thermal light source
falling on a light detector with single photon resolution
(Photo-multiplier), radioactive decay, thermal noise or similar
physical entropy sources. But using real physically sources for
random numbers is difficult and the random number generation is slow.
So in computer programming we generally use so called
pseudo-random-numbers, which are sequences of numbers calculated
based on a given starting number. A mathematical function gets the
last n numbers seen before and generates the next one from that. If
that function uses a smart mathematical expression, its results looks
really like random numbers. For games the generated sequences are
generally good enough, for cryptographic applications they may be not
good enough. So what we need for a random number generator is a
sequence of start numbers, and a mathematical function with internal
state. For each call of that function a new random number is returned
and the internal stage is changed so that next call will result in a
different number. When we use always the same sequence of starting
numbers, then our generator would generate always the same sequence
of random numbers. Sometimes this is desired, e.g. when we want a
behavior that looks random, but is reproducible, maybe for debugging
tasks. But in most cases we would use starting numbers that are
different for each program start. To get well suited start numbers we
can just use the current time with nanosecond resolution which most
computer hardware do provide.

Most simple and fast random number generators use for its internal
state two integer numbers. From these two numbers the next random
value is calculated and then the numbers representing the internal
state are modified also, to ensure that the next generated number is
again different.

Nim uses in its random module an implementation of the xoroshiro128+
(xor/rotate/shift/rotate) library. A Rand object with two integer
fields is used to store the actual state, and some simple and fast
logic operations as bit shift, logical xor and addition is used to
update state and to generate the next number:

type
when defined(js):
  type Ui = uint32
  const randMax = 4_294_967_295u32
else:
  type Ui = uint64

  Rand* = object # State of a random number generator.
    a0, a1: Ui

proc rotl(x, k: Ui): Ui =
  result = (x shl k) or (x shr (Ui(64) - k))

proc next*(r: var Rand): uint64 =
  let s0 = r.a0
  var s1 = r.a1
  result = s0 + s1
  s1 = s1 xor s0
  r.a0 = rotl(s0, 55) xor s1 xor (s1 shl 14) # a, b
  r.a1 = rotl(s1, 36) # c

The Rand object stores the internal state, rotl() is a helper
function which updates the state for each call, and next() is the
actual generator procedure returning an uint64 value. Note that the
addition used in next() does wraps around instead of giving an
overflow error as unsigned integers are used. The numbers returned by
the rand() proc are the foundation for all the other random number
types provided by the random module. To get integers with reduced
numeric range, we can just use the modulo operation, and to get float
results we may convert the integer value to float and apply some
basic mathematical operations like division for range reduction.

The most basic functions provided by the random module are the
overloaded rand() functions. Rand() called with an integer parameter 
n gives us integer random numbers in the range from 0 up to n, and 
rand() called with a float parameter x will give us random float
numbers in the range 0 .. x. When we just use the rand() functions in
this way we would get the same sequence of numbers for each run of
our program, as the generator always starts with the same well
defined initial state. We can call the proc randomize() before
calling rand() to initialize the generator to a different state based
on the current time. Then rand() will provide us with different
number sequences for each start of our program.

Generally it is a good idea to not use the one internal global state
of the random module for generation our random number, but to use our
own state variable. That way we prevent conflicts with other modules
which may use the random module as well. Imagine that we want to get
the same sequence of random numbers for each run of our program as we
are debugging our game, but another module initialize the internal
state of module random with a value based on the current time.

So the module random provides overloaded rand() functions that gets a
state variable:

from times import getTime, toUnix, nanosecond
import random

let now = getTime()
var rstate = initRand(now.toUnix * 1_000_000_000 + now.nanosecond)

for i in 0 .. 5:
  echo rstate.rand(5) + 1 # dice roll

for i in 0 .. 2:
  echo rstate.rand(100.0) # float random number in range 0.0 .. 100.0

Unfortunately the initRand() call to initialize it with a current
time value is a bit complicated, as we have to provide the current
time value directly. Note that you generally should call initRand()
only once in your program. A common mistake of beginners is to call 
initRand() each time directly in from of the rand() call. That is not
only not needed and slows down the generation process, it also can
lead to strange number sequences.

At the end of this section we will discuss the problem of filling a
container with random but unique numbers. For example assume that we
want to generate a sequence of 100 random numbers in the range 1 ..
100, with the restriction that each number should occur exactly once
in the sequence. Of course a code segment like s[i] = (rand(99) + 1)
would not work, as the same numbers could be generated multiple times
or not at all. The obvious solution for this task is to fill an array
first with consecutive numbers 1 to 100 and then exchange the initial
positions with destination positions determined by rand(99) The 
random module provides the [.func]#shuffle() function for this
shaking of a container. A related function is sample(), which is used
to randomly select an element from an openArray or a set.

References:

  * https://en.wikipedia.org/wiki/Random_number_generation

  * https://en.wikipedia.org/wiki/Applications_of_randomness

  * https://en.wikipedia.org/wiki/Pseudorandom_number_generator

  * https://nim-lang.org/docs/random.html

  * https://prng.di.unimi.it/

Timers

Sometimes we may want to measure the execution time of a code segment
of our program. For this the Nim standard library provides various
modules including the larger times module and the monotimes module.
The times module provides many functions and data types for handling
dates and times, while the small monotimes module is more specialized
for measuring of time intervals. For our first test we will use the 
times.cpuTime() function and the monotimes.getMonoTime() function.
The former gives us time values as seconds in float format, while the
later returns an int64 nanosecond value. To measure execution times
of code segments we ask for the current time at the start and at the
end of that segment and build the difference. Actually we will try to
measure the time needed for a float square root calculation. In the
past calculating square roots was considered a relatively slow
operation, slow compared to a plain floating point math operation
like multiplication or division. But on most modern hardware square
root calculation is really fast actually as we will see.

import std/[random, times, monotimes]
from std/math import sqrt

proc warmup =
  var x: float
  for i in 1 .. 1e10.int:
    x += 1.0 / i.float
  echo x

proc main1 =
  let x = rand(3.0)
  let y = rand(7.0)
  let start = cpuTime()
  let res = sqrt(x) + y
  let stop = cpuTime()
  echo stop - start
  echo res

proc main2 =
  let x = rand(3.0)
  let y = rand(7.0)
  let start = getMonoTime()
  let res = sqrt(x) + y
  let stop = getMonoTime()
  echo stop - start
  echo res

randomize()
warmup()
main1()
main2()

To get meaningful results we have to take some care: Most important
is, that the code segment that we want to measure is really executed.
This may sound odd, but assume that the code produces no noticeable
result at all. In that case the compiler may just remove that code
fragment from the generated executable, as it is not needed for
correct program execution. Or assume that the code fragment uses only
data which is already known at compile time. Then the compiler may do
all of the calculations already at compile time, and the whole code
fragment is removed again and replaced by the pre-calculated values.
And finally we have to remember that our computer may execute other
task at the same time or may be in various power saving states with
reduced CPU clock frequency, from which it takes some time to wake
up. To take care of this we try to execute some warmup code before
our actual timing task, and we do use the rand() function from the 
random module to provide input values for our code that are not known
during compile time. Finally we output the result of the calculation
by use of an echo() statement to make clear to the compiler that the
result of the calculation is really needed. Now let us compile and
run this program. We compile with option -d:release or -d:danger to
enable optimizations and avoid generation of debugging code that may
distort our timing. The result is still a bit surprising:

$ ./t1
23.6030665949975
4.98999998654881e-07
4.676567857718999
42 nanoseconds
8.527354349689125

Lines tree and five are the results of our timing attempt. Both
values are obvious too large and do not match. The reason for the
wrong results is the overhead by the function calls itself. That
overhead seems to be much larger for the cpuTime() call, as we get
result of about 500 nanoseconds. Maybe the reason for this is that 
cpuTime() works with floats internally. At least we see that 
getMonoTime() can measure time intervals in the range of a few
hundred nanoseconds. When we run the program a few times the printed
time intervals may vary. The reason for that are internal processes
in the CPU like clock rate and state changes. Generally the smallest
time value of multiple program executions is the most important for
us, as that is the minimal time which is actually needed for the
program execution. With this example we have learned how we can
measure time differences in our program, and that measuring really
small time intervals is difficult.

Fortunately measuring such tiny time intervals is supported by the
criterion package, which we may describe in later sections of the
book.

For now we will present another example program, where we measure
program code with longer running times. For that we create a loop
that is executed many times. So the offset of the timer functions
calls can be neglected compared to the actual running time of the
loop, and due to the longer running time the printed time values get
more reliable with low variations:

import std/[random, times, monotimes]
from std/math import sqrt

proc main3 =
  var s: array[64 * 1024, float]
  var res: float
  for i in 0 .. s.high:
    s[i] = rand(100.0)
  let start = getMonoTime()
  for i in 0 .. s.high:
    res += sqrt(s[i])
  let stop = getMonoTime()
  echo stop - start
  echo res

proc main4 =
  var s: array[64 * 1024, float]
  var res: float
  for i in 0 .. s.high:
    s[i] = rand(100.0)
  let start = cpuTime()
  for i in 0 .. s.high:
    res += sqrt(s[i])
  let stop = cpuTime()
  echo stop - start
  echo res

randomize()
main3()
main4()

For this example program we first fill an array with 64k random float
numbers and than sum the square root of these numbers. As the total
running time of our loops is not that tiny, we do not need a special
warmup function which is executed in front of the timed code. The
output of our program show that both timing functions match well for
longer time periods:

$ ./t2
112 microseconds and 701 nanoseconds
436909.89897942
0.0001135580000000001 # 114 microseconds
437222.3001038401

The float result in line 4 is 114 microseconds which matches well
with the 112 microseconds from line two. When you run this program
multiple times you may notice that you may get sometimes much lager
results for both or for only one of the two values. That is not
surprising, as the computer is processing not only our program but
many more, and due to task switching our program may be suspended for
some time. When we divide that 114 microseconds by the number of loop
iterations (64 * 1024) we get 1.7 nanoseconds, which is really
surprising fast for a square root calculation. The concrete value is
for a modern Intel I7 CPU. Of course these 1.7 nanoseconds is not
only the time needed for the square root calculation, but it includes
the operations with the loop counter and the time needed to fetch and
to access the actual array elements.

As timing code segments is not an uncommon use case, there exists
some external package which improve or simplify these operations,
like the criterion or benchy package. A related task is profiling our
program to find the part which takes the most CPU time, so that we
can concentrate on these parts to improve the total performance of
our program. For profiling various tools like the Linux perf tool are
available, which we will discuss in more detail later in this book.

References:

  * https://github.com/LemonBoy/criterion.nim

  * https://github.com/disruptek/criterion

  * https://github.com/treeform/benchy

Hash Tables

A common task in computer programming is the storing and retrieving
of data records. The situation is very easy whenever each data record
is directly mapped to continuous integer number n0, n0+1, n0+2, ... ,
n0+M. In that case we can use that numbers as index keys to access
the data records and store the data as objects or references to
objects in a sequence, or maybe when the data set in small and the
maximal number of entries is known at compile time in an array.

In the past it was a common practice to give hardware parts in a shop
and even customers an unique id number from a continues range, so
that storing and fast access in indexed containers is possible. This
works well when we really use the numbers as keys. But actually we
generally work with data which is already labeled by names expressed
as sequences of ASCII character: Customers in a hardware store, food
in a super marked. Assigning id numbers to people is possible, but
generally people do not like to have to remember the assigned id
number when they want to buy something in a online shop.

So let us investigate how we can store and retrieve data objects
without the use of continues numbers as key. Assume we have a
customer data base

type
  Customer = object
    lastName: string
    firstName: string
    age: int
    postalAdress: string
    phone: string
    credit: float

Of course we can store the customers just in a seq, and do a linear
search when we want to access a person by name:

customers: seq[Customer]

# ...

var found = false
for c in customers:
  if c.lastName == queryName:
    found = true
    echo "Person ", queryName, " has a credit limit of ", c.credit
if not found:
  echo queryName,  "not found"

Such a plain linear search is not very fast of course. An obvious
improvement would be to sort the customers by names, as we can do a
so called binary search in that case as we did long time ago in
printed telephone registers: Open the telephone book somewhere in the
middle, and when the names on that page are all greater than our
friends name, then continue the search in the first half of the book,
otherwise in the last half. We continue the halving strategy until we
find the name. As we half the data set in each step this way, we say
that the algorithm has log2(N) cost, where log2 is the logarithm with
base two and N is the size of the data set.

A similar solution would be to use some form of an ordered binary
tree, which has also log2(N) costs for retrieving operations. We will
learn more about sorting sequences, doing a binary search in a sorted
seq and about tree structures later in the book.

A hash table, called only table in Nim, is a homogeneous resizable
container that behaves similar as the Nim sequences, but releases the
restriction that for direct accessing an element its position in the
sequence has to be known.

The idea of a hash table is to use an arbitrary data type to directly
access objects stored in a container in a similar way as we can do it
for arrays and sequences with integer keys. The first step is to use
a so called hash function to map key objects, which are not already
of integer type, to the integer type. We would try to use a hash
function which can be evaluated fast and which maps our data to
integer values distributed to the whole integer value space without
clustering. The integers generated by a hash function look in some
way like random numbers, they are distributed over the full integer
value range without an obvious order or systematic. Mapping arbitrary
objects to integers is generally not difficult. For as string a first
attempt would be to use the characters of a string in the same way as
we calculate the value of a number literal by summing up the digits
each multiplied with powers of ten given by the position. For a
string that may look like

intVal = uint(s[0]) * 256^0 + uint(s[1]) * 256^1 + uint(s[2]) * 256^2 + ...

We multiply with powers of 256 as we have 256 different ASCII
characters.

That would be not really a good hash function, as all short strings
would be mapped to low integer values, not distributed over the full
value range. But similar, smarter has functions are available.

So a hash function can map arbitrary data types to integers. But what
we really want is a sequence of continues integer values, which the
hash functions does not provide by design. But that is no real
problem: In the same way as we can do a range reduction for a random
number generator function generating random numbers using the full
integer range by just applying a modulo operation, we can apply the
modulo operation on the value returned by a hash function.

Range reduction by modulo gives us smaller integer numbers, which may
already be some form of index values for an array or a seq. With two
restrictions: Index collisions can occur, as applying the hash
function and the modulo range reduction on different strings may give
us the same index value. And some index values may be never
generated. The later is not that serious, some positions in the
container would be remain unpopulated. Collisions are much more
serious of course, we have to handle them somehow. One solution is,
that we make each storage location in our container again a sequence,
which can store all the colliding data sets. That way our hash table
would be a sequence, where each element is again a short sequence
containing all the colliding data record. For a data retrieving
operation applying the hash function with modulo data reduction would
give us the position in the larger seq, and we then would have to
check all the elements in the short seq to find the actual record. In
best case the short seq contains only one entry, when no collision
has occurred. For a customer data base this strategy may be indeed
the best solution, as in rare cases multiple different customers may
have exactly the same name. So it would be nice if for query
operations in that case a list of all customers with exactly that
name is returned.

In practice often a modified strategy is applied preventing the seq
in seq container type: We use only one loosely populated seq, and
whenever a collision occur, we just put the colliding data record at
some free index position after the position determined by the hash
index value. That way data retrieving starts by the position given by
the hash key and then checks the data record at that position and the
following positions, until a matching record or an empty position is
found, the later case indicates that the queried data record is not
contained in the data base. Storing data records works similar: When
the position given by the hash key is void, then the new entry is
stored at that position. If the index position is already occupied,
then following positions are examined until a void one is found and
the data is stored there.

Hash tables work well generally when they are not too dense
populated. Generally we make the number of available index position
double the size as the number of expected entries. Then the change of
collisions is not too large, and when a collisions occurs, then
chances are high that one of the next positions are still
unpopulated.

When by inserting more and more data records the population density
becomes too high, then generally a new, larger table is allocated and
the data records are moved from the old to the new table, in a
similar way as it is done for plain sequences when all capacity is
occupied.

Now let us see how we can us the tables module of the Nim standard
library to store the customer record we introduced above:

import std/tables

type
  Customer = object
    lastName: string
    firstName: string
    yearOfBirth: int
    postalAdress: string
    phone: string
    credit: float

var customers: Table[string, Customer]

proc addNewCustomers =
  var c: Customer
  c = Customer(lastName: "Turing", firstName: "Alan")
  c.postalAdress = "England"
  c.yearOfBirth = 1912
  customers["Turing, Alan"] = c

  c = Customer(lastName: "Zuse", firstName: "Konrad")
  c.postalAdress = "Germany"
  c.yearOfBirth = 1910
  customers["Zuse, Konrad"] = c

proc queryCustomer(key: string) =
  if customers.hasKey(key):
    echo "known customer:"
    echo customers[key]
  else:
    echo "customer key not found in data base"

addNewCustomers()

queryCustomer("Zuse, Konrad")
queryCustomer("Gates, Bill")

The basic usage of Nim tables is very similar to the use of
sequences. While we have to specify only the base type for a Nim seq,
we have to specify the type of the key and the type of the stored
entities for a hash Table. The line var customers: Table[string,
Customer] defines a variable with a generic Table type. The table
uses strings as keys and stores Customer objects. We can then use the
[] subscript operator to store Customer objects in the Table. As we
created a Table with string key type, we have to specify strings when
we use the subscript operator or other functions to access entries of
our table. For the query operation we first call the function hasKey
() to check if the customer with that name is contained in the data
base and then use again the subscript operator to access the data
record.

The tables module of the Nim standard library provides many more
functions for interacting with Tables. Most are easy to understand
and use. When you inspect the API docs of the tables module you will
discover that beside the Table data type also a TableRef exists. The 
Table type has value semantics, that is if you copy a whole table
instance, then the whole content is copied. TableRef instances have
reference semantics, the content is not copied when you assign one
instance of a TableRef to another variable.

In the example above we called hasKey() to check if a data record is
available before we accessed that record. Access with the subscript
operator [] would raise an exception when an entity is not available
in the table. HasKey() and [] both would have to locate the data
record. A faster way to access data record when we are not sure if
they exists in the table is the getOrDefault() proc:

let dummy = Customer()
let query = customers.getOrDefault(name, dummy)
if query.lastName.len == 0: # we know all entries in the database have a lastName, so we got the dummy default value
  echo name, "not found"
else:
  process(query)

A useful variant of the Table data type is the CountTable, which we
can use to count data object, maybe words in a text:

import tables
var ct: CountTable[string]
ct.inc("Nim")
ct.inc("Rust")
ct.inc("Nim")

echo ct["Nim"]
for k, v in ct:
  echo k, ": ", v

A Table instance stores entries not in the order of insertion. When
we iterate over the table, we get the results not back in the order
of insertion. If we really should need to preserve the insertion
order we may use the OrderedTable variant. Note that an OrderedTable
does not sort its entries, it remembers insertion order. Ordered 
Tables have some internal overhead, so we should use them only when
necessary.

For all the various table variants we can use procs like clear(), len
() or del() to remove all entries from a table, to check for the
number of entries or to delete entries. Note that some functions may
throw exceptions when we try to access entries that are not
available. And note that the subscript operator =[] overwrites
already existing entries.

For our initial customer data base the current table implementation
may still be not optimal, as it is not clear how to handle different
customers with the same name. But customer data bases are really
special cases, in most cases different things have different names.

User defined hash values

The tables module uses the hashes module to calculate the hash value
for the keys that we use to access table content. For many data types
the hashes module already defines a hash function. When we would like
to use tuples or object data types as keys for table access, then we
would have to define a hash function for that key objects first. The
API documentation of the tables module contains an example for this,
where as key an object data type with firstName and lastName fields
is used to store salary entries in the table. While firstName and
lastName are strings, and for single strings a predefined hash
function is available, we have to declare another hash function for
objects with two strings:

import tables, hashes

type
  Person = object
    firstName, lastName: string

proc hash(x: Person): Hash =
  ## Piggyback on the already available string hash proc.
  ##
  ## Without this proc nothing works!
  result = x.firstName.hash !& x.lastName.hash
  result = !$result

var
  salaries = initTable[Person, int]()
  p1, p2: Person

p1.firstName = "Jon"
p1.lastName = "Ross"
salaries[p1] = 30_000

The hash generation is a bit cryptic: First we mix various existing
hash values using the !& operator, and finally we use the !$ operator
to generate the final hash value. For details please see the API
documentation of the hashes module.

Hash tables can be seen as a way to attach arbitrary data to other
data. The above example attaches a "salary" to a person object. In
most cases we would just create one more object field when we have to
store more data, but sometimes that is not easily possible. One
example is when we use a low level C library, which gives us some C
objects back. Maybe we use an advanced C or C++ math library like
CGAL, and we get some abstract low level objects from it, maybe
circles with center coordinate and diameter. As that objects are not
Nim object, but C or C++ entities, we can not easily subclass them to
attach more properties like a color attribute. But as each entity has
a unique address, we can just use a table with key type address and
all the needed values like color as data. That would be some overhead
of course, as each color lookup would mean a table access, but it is
a simple solution.

We can even attach properties to plain data types this way:

import tables

var t: Table[float, string]

let PI = 3.1415
t[PI] = "Pi"
t[2.0] = "two"

echo t[PI]
echo t[2.0]
echo t[5.0 - 3.0]

For floats a predefined hash function is available, so the code above
should work. But floats as keys are a bit fragile due to the fact
that float math is not really exact. So the last line in above code
may raise an exception due to access of a non existent entry, as the
difference 5.0 - 3.0 may not exactly be identical to the value 2.0.

Hash tables can be even useful containers, when we already have
numeric data as possible keys for indices in a sequence: In
mathematics we could have a two dimensional array, that is an array
of array in Nim, to store matrices. This is OK when the matrix is
really populated, that is most entries have meaningful non trivial
values. But for very large, loosely populated matrices, with mostly
just zero or one entries, storing the whole matrix as a table using a
row, column tuple as key, may save a lot of RAM.

Equality and Identity

When we use objects or references to objects as keys for tables, we
have to remember how Nim compares value and reference types:

import tables, hashes

type

  O = object
    i: int

  R = ref object
    j: int

proc hash(o: O): Hash = hash(o.i)

proc hash(r: R): Hash = hash(cast[int](addr(r[])))

var o1 = O(i: 7)
var o2 = O(i: 7)
var r1 = R(j: 13)
var r2 = R(j: 13)

echo o1 == o2
echo r1 == r2

var t1: Table[O, float]
t1[o1] = 3.1415
echo t1.hasKey(o2)

var t2: Table[R, float]
t2[r1] = 2.7
echo t2.hasKey(r2)

The output of above program is

true
false
true
false

By default the == operator compares content for value objects, but
the instance addresses for references. Because of this it makes sense
to define hash functions for object types and ref object types in a
compatible way: We use the hash value of the single integer field of
our value object as hash result for the whole object, and we use the
address of the instance for the hash value of the reference object.
As different instances of ref objects have always different
addresses, the hasKey() does return false when we use as argument a
different instance variable, independent on the content of its
fields.

For special use cases we may redefine the == operator, but we have to
ensure that the defined hash function matches the == operator: When a
== b is true, then hash(a) has to be identical to hash(b)! The reason
is that tables first compare the hash value of query key with key of
entities in the table, and only for matching hash value do comparison
of the actual data content.

Performance

Hash table lookup is fast. We say that hash table lookup is a O(1)
operation, which shall indicate that the time needed for doing a
table lookup does not depend on the total number of entries stored in
the table. The reason for that is that for a lookup it is necessary
to calculate the hash value, do the modulo operation and access the
table content and maybe a few of the following table entries in the
case that the first entry is not a match. Storing data is also an O
(1) operation, as it works very similar, as long as the table is not
already too dense populated so that a recreation is necessary. In
that case that single storing operation is obviously very slow, but
that occurs very rarely, and maybe not at all when we use a large
enough table from the beginning. Still small tables are much faster
than larger tables due to cache effects. For small tables all data
may fit into the caches of the CPU, while for large tables most data
is located outside of caches in RAM, and RAM access is magnitudes
slower than cache access.

And hash table lookup is slower than array or seq access. To access
an element from an array or seq we have only to multiply the index
value with the byte size of the stored elements type and maybe to add
an offset when the array does not start at index 0. For tables we
have to calculate the hash value, do the modulo operation, access
some elements at the calculated position, and most importantly, to
compare the content at that positions with the actual key data. If
the key is a string, a few string comparisons (at least one) are
necessary to determine if the query element is available in the
table. So while array access may take less than a nanosecond on
modern hardware, table lookup may take a few dozens of nanoseconds.
Lookup with string keys is generally slower than for other key types
likes integer, as for string comparison it may be necessary to
compare many characters to get a result and because strings generates
some memory indirection by the fact that string content is stored
somewhere in the heap outside of any cache.

Tuples or other containers as Keys

At the end of our introduction to hash tables we will present a very
useful, but maybe not that obvious property of hash tables: The keys
used for table access don't have to be simple data types, but can be
container types like tuples or arrays. Imagine you have a map in 2d
with a set of points on that map each presented by an x, y coordinate
pair. That points could be cities, and some cities may have a direct
connection by a road. So how can we test if two cities are directly
connected and get the distance between the two cities? With a hash
table using a tuple of two city coordinates as key it is easy:

import tables, math

const
  InvalidFloat = 1e30 # arbitrary marker that in not a valid value
  InvalidCoord = (InvalidFloat, InvalidFloat)

type
  Coord = tuple
    x: float
    y: float

  Cities = Table[string, Coord]
  Distances = Table[(Coord, Coord), float]

var cities: Cities
var distances: Distances

proc insertCity(name: string; coord: Coord) =
  cities[name] = coord

proc insertDist(a, b: string) =
  var (a, b) = (a, b)
  if a > b: swap(a, b)
  let ca = cities[a] # caution, will raise an exception if name is not a know city
  let cb = cities[b]
  distances[(ca, cb)] = math.hypot(ca.x - cb.x, ca.y - cb.y)

proc checkDirectConnection(a, b: string): float =
  var (a, b) = (a, b)
  if a > b: swap(a, b)
  let ca = cities.getOrDefault(a, InvalidCoord)
  let cb = cities.getOrDefault(b, InvalidCoord)
  if ca == InvalidCoord or cb == InvalidCoord:
    return -1 # marker when cities are unknown
  result = distances.getOrDefault((ca, cb), InvalidFloat)

insertCity("aTown", (2.0, 7.0))
insertCity("bTown", (2.0, 11.0))
insertCity("cTown", (17.0, 23.0))

insertDist("aTown", "bTown")

var d = checkDirectConnection("aTown", "bTown")
if d == -1:
  echo "query for unknown town"
elif d == InvalidFloat:
  echo "Cities have no direct connection"
else:
  echo "Distance: ", d

echo checkDirectConnection("bTown", "aTown") # 4, same as above
echo checkDirectConnection("aTown", "cTown") # 1e30
echo checkDirectConnection("aTown", "xTown") # -1

For the procs insertDist() and checkDirectConnection() we use a trick
to get the same results when we exchange the names: We sort the names
alphabetically when we insert the distances and also when we query
the distances. So we get the same result. Of course we could insert
the tuple also twice instead, but as distances is the same in both
direction sorting and inserting only ones makes some sense. Note that
we used tuples for the coordinate pairs in the distances tables.
Maybe the more obvious data type would be an array with two entries,
as the array type is a container for homogeneous data, while a tuple
can also contain different data types. But currently Nim supports
tuples better in some situations, e.g. for automatic tuple unpacking.
So often we use tuples when array would be the first choice maybe. 
array type would have the benefit of iteration over elements at
runtime, while tuples have the benefit that we can access elements by
names and by an integer constant. For performance array or tuple
should make no difference here.

References:

  * https://nim-lang.org/docs/tables.html

  * https://nim-lang.org/docs/hashes.html

Regular Expressions

A regular expression, shortened as regex or regexp, is a sequence of
characters that specifies a search pattern, which is used to find or
replace parts of a string or of a whole text document, or just to
validate it. It is a technique developed in theoretical computer
science and formal language theory, introduced in the 1950s, when the
American mathematician Stephen Cole Kleene formalized the description
of a regular language. The use of regular expressions became popular
with Unix text-processing utilities like sed, grep and awk, were used
in early text editors like vi and emacs for pattern matching, and are
commonly used in modern text editors and word processing programs in 
find and find and replace dialogs. Different syntaxes for writing
regular expressions have existed since the 1980s, one being the POSIX
standard and another, widely used, being the Perl syntax.

To demonstrate a first simple example for the usefulness of regular
expressions, we will start with a sed call that can be used to
replace all snake case symbols in a text file with camel case, e.g
convert the symbolic name line_width into lineWidth:

sed -i -E 's/_([a-z])/\U\1/g' myfile.txt

Here the option -E tells the sed program to use the extended regular
expressions rather than basic regular expressions, and option -i
specifies to work in place, instead to just print the modified text
in the terminal window. The pattern s/a/b tells it to substitute
pattern a by expression b, and the final /g stands for global and
tells sed to do substitutions in the whole file. The actual
interesting part is the search pattern _[a-z], which specifies the
actual underscore character followed by a single lower case letter.
Whenever such a pattern is found, it is replaced with a capitalized
version of the found letter. The /U tells sed to convert to upper
case, and \1 refers to the captured text segment. You may still
wonder why [a-z] is enclosed in braces -- well matches enclosed in
braces are actually captured, so we can refer to the captured letter
later, in this case we refer to the first captured match with /1 and
apply /U on it to convert it to upper case.

As you see, regular expressions are useful, but difficult to
understand and to remember.

Some programming languages like Perl or Ruby have built in support
for regular expressions, and others use external libraries. For
interpreted languages like Perl, Python or Ruby it makes a lot sense
to use regular expressions for parsing strings, as the regex engines
of these languages are generally written in C language, which leads
to the fact that even for very basic string operations like spitting
strings into single tokens or doing simple character replacements,
the use of regexes can be faster than doing it with multiple
statements in the interpreted program code. For compiled languages
like Nim the situation is very different -- using regexes is fast, but
doing simple things directly in the compiled languages is still much
faster. And Nim provides many other libraries like strscans or 
parseutils, which can do even advanced string operations much faster
than by use of regular expressions.

So actually the use of regular expressions in Nim is very limited, in
most cases there exists other, simpler and faster solutions. As
learning the use of regexes is not that easy, and it is hard to
remember all the details, we may hesitate to try it at all. But
actually for text processing tools like sed and grep, and for the use
in text editors and word processors regexes are very useful, so it
makes some sense to learn at least the basic use of regular
expressions. And when we learn to use regexes at all, then we can use
them in Nim as well.

Each character in a regular expression (that is, each character in
the string describing its pattern) is either a metacharacter,^[40]
having a special meaning, or a regular character that has a literal
meaning. For example, in the regex b., b is a literal character that
matches just 'b', while . is a metacharacter that matches every
character except a newline. Therefore, this regex matches, for
example, b%, or bx, or b5. Together, metacharacters and literal
characters can be used to identify text of a given pattern or process
a number of instances of it. Pattern matches may vary from a precise
equality to a very general similarity, as controlled by the
metacharacters. For example, . is a very general pattern, [a-z]
(match all lower case letters from 'a' to 'z') is less general and b
is a precise pattern (matches just 'b'). The metacharacter syntax is
designed specifically to represent prescribed targets in a concise
and flexible way to direct the automation of text processing of a
variety of input data, in a form easy to type using a standard ASCII
keyboard.^[41]

In this section we will not try to explain all the details of the
syntax and semantic of regular expressions, but only show you how the
regex module is used in principle, and give a few examples for its
use. For details you should consult the API documentation of the 
regex module, and for concrete use cases you may additional consult
the Wikipedia article and the various internet resources.

The Nim standard library provides two modules for the use of regular
expressions, called re and nre, which both are wrappers for the PCRE 
(Perl Compatible Regular Expressions) C library. Additional, a module
called regex is available as an external package, which is fully
written in Nim language. These three modules are similar, but their
API is different. When you intent to use re and nre you have to
ensure that the PCRE C library is also installed on your computer. As
the external regex module is written in pure Nim and is of high
quality, we will actually use that one for our examples -- actually if
using one of the two others, it would be not easy to decide which to
use. You may wonder why we present the regex module already here, as
it is not part of the Nim standard library? Well, a regex library is
am important part of each programming languages, and re and nre are
actually included in Nim's standard library. Due to Nim's package
managers like nimble, using external packages is very easy, we just
have to execute

nimble install regex

We will start to demonstrate the use of the regex module with a very
simple example:

import regex

let r: Regex = re"\w\d"
let t1: string = "a1"
let t2 = "nim"
var m: RegexMatch
if match(t1, r, m):
  echo "match t1"

if match(t2, r, m):
  echo "match t2"

We use the re() function with the search pattern as argument to
generate an instance of a Regex variable. Then we can use the match()
function to match a textual string against this regex. The last
argument of the match function is a variable of RegexMatch type,
which captures the matched terms, so that we can use them later.

In our pattern "\w\d" the \w stand for a word character which
includes upper and lower case ASCII letters, and the \d stands for a
decimal digit. So the string t1 matches that pattern, but the string 
t2 does not, as there is no decimal digit following the first letter.
In the example from above we actually check only if a string matches
the pattern, but we do not capture the matches. So we do not need the
RegexMatch variable m at all and could call the match() function
without that parameter. To actually capture a match, we would have to
enclose the subpattern in braces like "\w(\d)" to capture the digit
in case of a successful match.

As next simple example let us match a string starting with the
capital letter A, followed by an arbitrary number of letters,
followed by an integer number. We want to capture the integer in case
of a match:

import regex

let r: Regex = re"A[a-z, A-Z]*(\d+)"
let t1: string = "Alex77"
let t2 = "nim"
var m: RegexMatch
if match(t1, r, m):
  echo "captured: ", m.group(0, t1)

if match(t2, r, m):
  echo "captured: ", m.group(0, t2)

To understand the regex pattern, we have to know that we can use * to
specify an arbitrary number of repetitions, and + to specify one or
more repetitions. The initial A is not a metacharacter and stand for
the literal A. The content of the square brackets specifies a
character class, a-z specifies the range of lower case letters, A-Z
the range of upper case letters, and the following * indicates an
arbitrary number of repetitions. Finally, the \d stands for a decimal
digit, and + specifies one or more repetitions. As we enclosed the
last subpattern in braces, that group is captured. For a successful
match we can access the capture with the group() function(, where we
have to specify the index number of the capture, and the actual text
string that was used for the match. The fact that we have to specify
the initial text may look a bit strange indeed. For string t1 we get
a successful capture with the result @["7"]. So our actual captured
string is contained in a seq, which is useful when multiple (nested)
strings are captured. In the code from above we could have used 
groupFirstCapture() instead to get directly the first captured
string.

Greedy matching

Whenever we create regex patterns, we have to care for the fact if
sub-matches should be greedy or not. In most cases greedy is the
default, and we have to take some care when we need none-greedy
behaviour. Greedy means just, that the regex engine captures as many
characters as possible, while none-greedy capturing stops the
capturing process early. Indeed these greedy/none-greedy capturing
can be one of the most demanding tasks when we create larger and
complicated patterns. Imagine that for our above example, we would
have used the pattern re"A\w*(\d+)". For the same string "Alex77" we
would then get the output @["7"]. The reason for that is, that \w*
does a greedy processing, eating all but the last decimal digit,
which it left to satisfy /d+. From the API documentation of the regex
module we learn that we can specify \w*? instead to get a none-greedy
processing, so both digits are left for \d+ and we get again @["77"]
as output.

Escape sequences

The use of escape sequences in regex patterns is another difficulty
for beginners. The first problem can be that the Nim compiler may
process the escape sequences already itself, while we intent to left
them for the regex engine. We can avoid that when we use Nim's raw
strings, e.g. we can use triple quotes when we construct the pattern
from individual strings as done in our next example. In an regex, we
can use escape sequences to specify special literal characters, we
may use \t for a literal tabulator for example. And finally, we may
have to escape some punctuation characters like *, + or ? that have a
special meaning for the regex engine when we intent to use that
character as an ordinary literal. For example, to match a letter
followed by a question mark, we have to use a pattern like "[a-z]\?"
or "[a-z][?]". Inside of a square bracket we can use the punctuation
characters without the need to escape them.

As next example, let us assume that we have to process a text file in
which each lines starts with a name consisting of lower case letters,
and three decimal numbers. The name and the three numbers can be
separated by spaces, or by commas or semicolons:

import regex

let h = """\s*[,;]?\s*(\d+)"""
let r: Regex = re("[a-z]+" & h & h & h)
let t1: string = "nim 12;8  ,   17"

var m: RegexMatch
if match(t1, r, m):
  echo "captured: ", m.group(0, t1), " ", m.group(1, t1), m.group(2, t1)

To understand the pattern that we use in the above code, we have to
know that we can use \s for a white-space character, so \s matches a
single space or a tabular character. We could have used just a space
literal instead, or the [ ] character class containing just a single
space. And we have to know that we can use ? to specify an optional
entity. We have split the total pattern into two parts, where the
variable called h stands for the sequence of any number of
white-space, followed optionally by a single comma or a single
semicolon, followed again by any amount of white-space, that is
finally followed by at least one decimal digit. As we want to capture
the decimal numbers, the sequence of decimal digits is enclosed in
round brackets. The total regex pattern is constructed by the
subexpression [a-z] for at least one letter, followed three times by
the integer pattern with the allowed separators. Note that we allow
any amount of spaces or tabulators, but only a single comma or
semicolon between the different entities. Note that the match()
function of the regex module does always a full match, so a single
space at the beginning or end of the text string would make the match
fail. We could compensate for that by starting and ending the regex
pattern with "\s*". Or we could use instead of match() the find()
function, which search through the string looking for the first
location where there is a match. When we use find(), we may use the
special characters ^ and $ to match the start or end of the string,
that is with find() and re"\s+$" we could find all strings which have
trailing white-space. Note that find(text, re"^regex$", m) is the
equivalent to the match() function.

The regex module provides us also with two replace() functions, which
we can use to replace matched patterns with literal strings or
captured and modified strings. The first replace() function uses as
third argument a string, which is used for replacements and in which
we can refer to captured groups with the symbols $N, where N is the
index of the captured group starting at one. The second replace()
function uses a function as third argument, that function gets an
instance of the RegexMatch type as first parameter and returns the
string replacement. We will use both variants of the replace()
function to create a tiny app that we can use to fix typos in program
and text files: Text files can contain typing errors, which includes
two or more spaces between adjacent words, unneeded trailing
white-space at the end of lines, and the use of a instead of an in
front of words starting with a vocal. And program source code may use
snake case for names instead of camelCase, e.g. line_counter instead
of lineCounter. We will create a tool that can fix these four
issues -- ignoring the fact that an actual a/an replacement may
corrupt program source code. To demonstrate the four issues, we have
created this small test file -- line three contains two unneeded
spaces, and the last line has some unwanted trailing white space:

# this is a example
var   line_width:  int

echo        line_width

We will fix these four issues independent of each other, so we will
try to find a regex that matches for each issue, and then use the 
replace() function to fix it.

import regex, strutils
let fileName = "test.nim"
let trail = re"\s+$"
let aan = re"a(\s+[AEIOUaeiou])"
let space = re"(\S\s)\s+(\S)"
let snake = re"_([a-z])"

proc toUpper(m: RegexMatch, s: string): string =
  when defined(debugThis):
    echo "a: ", s
    echo "b: ", m.group(0)
    echo "c: ", m.group(0)[0]
    echo "d: ", s[m.group(0)[0]]
    echo "e: ", strutils.toUpperAscii(s[m.group(0)[0]])
  return strutils.toUpperAscii(s[m.group(0)[0]])

for l in filename.lines:
  var h = l.replace(trail, "")
  h = h.replace(aan, "an$1") # caution, this is for text files!
  h = h.replace(space, "$1$2")
  h = h.replace(snake, toUpper)
  echo h

We process our file with the issues line by line, using the lines()
iterator to which we pass a file name and which gives us the
individual lines of the file. We will start with the simplest task,
that is removing trailing white-space. The search pattern for this
issue is obviously "\s+$", that is at least one white-space at the
line end, which we have to replace with an empty string. So we pass
this regex pattern called trail and an empty string literal to the 
replace() function. Replacing a by an is also easy -- we search for an
a followed by white-space and a vocal, for which the regex pattern is
the aan variable in above code. In this case we have to preserve the
actual white-space and the vocal, so we enclose these in brackets to
capture it. The replacing string is "an$1", where $1 stands for the
captured white-space and the captured vocal. Replacing too much
inter-word space is a bit more difficult. The actual issue is one
white-space followed by one or more white-space, for which a possible
match pattern is "\s\s+". But actually we do not want to remove all
white-space consisting of more than one character, but only
white-space between words. So multiple white-space at the beginning
of a line should be preserved. One solution is, that we use the
metacharacter \S, which matches all none-white-space characters, and
then use this search pattern: "(\S\s)\s+(\S)". The pattern starts
with a none white-space character, followed by a white-space, then at
least one more whites-space character, and finally a none white-space
character. We capture the two first characters, and the last one.
This way we can replace the whole match with the two captures and we
are done. Finally, we have to replace underscore characters followed
by a lowercase letter with the capitalized letter. Some tools like 
sed provides the \U to capitalize a capture, but this is not
available for the regex module. So we use the replace() variant which
uses a proc as last parameter -- to that proc the capture and the
original string is passed, and that function should return the
replacement string. The capture which we have to use to catch a snake
element is obvious just "_([a-z])". We call the converter proc 
toUpper(), its parameters and its return type is specified by the 
regex API docs. But unfortunately the actual structure of the passed 
RegexMatch instance is not that detailed described. So we created
some conditional echo() statements inside the body of our toUpper() 
proc to show us the structure of the parameters. When we compile our
program with the -d:debugThis option, and run it, we get this output:

nim c -d:debugThis t.nim

$ ./t test.nim
# this is an example
a: var line_width: int
b: @[9 .. 9]
c: 9 .. 9
d: w
e: W
var lineWidth: int

a: echo line_width
b: @[10 .. 10]
c: 10 .. 10
d: w
e: W
echo lineWidth

So the last string parameter is always the whole string that was
passed as first argument to replace(), and m.group(0) is a sequence
of slices for the first capture. We need only the first element of
this seq, as we have only one capture, and we use that slice to
extract the captured sub-string by use of s[m.group(0)[0]]. Finally
we apply strutils.toUpperAscii() on this sub-string to capitalize it,
and return that result.

When you run above program, you should get a text file with all
issues fixed. You may redirect the output to a file with "\.t
test.nim > newtest.nim" and load newtest.nim into an editor to proof
that the trailing white-space is removed as well.

Final remarks

The use of regular expressions is not that easy, and makes in most
cases not much sense in Nim. Maybe the largest problem of regular
expressions is, that it is hard to understand patterns that we
created some years ago, or that have been created by other people.
And it is difficult to modify that patterns later. Maybe you should
play a bit with regexes yourself now, and come back to this topic
when you think that you need them. In this book we were only able to
give a tiny introduction into the stuff -- you will have to carefully
study the API docs of the regex module and a lot of other resources
in the Internet when you should intent to seriously use them. We
should also mention, that while regexes are very powerful, for some
tasks they work not that well, e.g. parsing math expressions with
nested braces, or just skipping nested comments in some source code
can be very difficult or even impossible.

References:

  * https://en.wikipedia.org/wiki/Regular_expression

  * https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions

  * https://www.regular-expressions.info/pcre.html

  * https://nitely.github.io/nim-regex/regex.html

Part IV: Some Programming Tasks

In this section we will present a few simple programming exercises.

Sorting

Sorting a sequence or an array of numbers is generally a component of
each computer programming course. While we would not really code
sorting algorithm for the actual software that we write, but use the
generic sorting algorithm from the standard library, sorting
algorithm can teach us some basic programming skills. When we sort a
small number of items manually, we would generally use selection or
insertion sort intuitively: For selection sort we pick the smallest
element and move it at position one, then pick next smallest item and
move it at position two. This strategy is easy to implement and works
not bad for small quantities. For larger containers algorithm like
quicksort or mergesort gives better performance.

Selection Sort

#[ <--s.len
5
7 <-- i
4 <-- k, x == 4
6 <-- j
3
2 first tree entries are already sorted
1
]#

proc selectionSort(s: var seq[int]) =
  var i: int # used to step through the still unsorted range
  var j = 0 # lower bound for still unsorted range
  var k: int # position of currently smallest candidate
  var x: int # and its value

  while j < s.len: # while there is an unsorted section left
    i = j # start with i one above the already sorted range
    x = s[i]; k = i # assume first element is the smallest
    inc(i) # continue with next one in the still unsoprted range
    while i < s.len: # while there are unchecked candidates
      if s[i] < x: # that one is smaller than current candidate
        x = s[i] # remember its value
        k = i # and remember its position
      inc(i) # examine next candidate
    swap(s[j], s[k]) # exchange smallest value with the one currently at position k
    inc(j) # sorted range increased by one

import random
proc main =
  var s: seq[int]
  for i in 0 .. 9:
    s.add(rand(100))
  s.selectionSort
  echo s

main()

The comment on the top of the above example shows a partly sorted
list of 7 integer numbers. The lowest three positions already
contains the sorted numbers 1 to 3. The next four positions are still
unsorted. For the sorting process we need the three indices i, j, k
and the variable x to store an actual value for the comparison. The
index j is the lower bound for the still unsorted range, j starts at
zero obviously. The variable x stores the currently smallest value of
the still unsorted range, and k is the index position of that value.
Finally i is a counter that is used to step through all the values of
the unsorted range. The outer loop is executed as long as j is
smaller than the length of the sequence that we want to sort. We set 
i to the value of j, k to the same value and assume initially that s
[i] is the smallest value from the still unsorted range. That value
is stored in x. Then we execute the inner while loop until we have
processed all elements of the still unsorted range. Whenever we find
an element that is smaller than x, we store that position in k and
the value in x. When the inner loop has finished, we exchange the
smallest value with the first element of the still unsorted range.
This way the sorted range increases, and the unsorted range decreases
by one.

To test our sorting procedure we generate some random numbers, sort
them and print the result. Selection sort is said to be of order O(n^
2), with n being the number of values to sort. So the effort
increases quadratic with the number of values. This is because we
have to test all the still unsorted values just to increase the
number of sorted values by one. Selection sort has a natural
behavior, that is for an already sorted array the test s[i] < x would
be always false and we would have to do no movement of values in that
case. So performance is best for an already sorted or partly sorted
list, and the sorting is stable in the sense that we do not move
elements when it is not really necessary. In one of the following
sections we will discuss the quicksort algorithm, which is not a
stable sorting method: Elements with equal value may be move with
quicksort. For plain numbers that does not really matter for the
result, as numbers are indistinguishable. But when we sort objects,
maybe persons by age, persons of same age would be exchanged by
quicksort, which may not be desired.

Note that the code above is not really optimized for performance yet.
One possible improvement may be to iterate the two loops not from
zero to s.len, but in the opposite direction. In that way comparison
of loop indices with a constant value, zero in this case, could be
used to terminate the loop. Comparison with constants can be faster
than comparison with actual variables, and comparison with zero is
generally fastest. Note that we compared indices with s.len() in
above code, which is not that bad, as len() is a field in the seq
data structure, so the compiler should be smart and replace s.len
with just a field access without proc call overhead.

We started this section with thinking about how we would sort a small
quantity of items lying on the table manually. This strategy is
generally a a good one. Sometimes it can even help to ask how a child
would solve a problem to find a way how to do it with the computer.

Insertion Sort

Insertion sort is another simple sorting method, which some card
players like to use: They hold two sets of cards in their hand, one
unsorted set, and one sorted which is initially empty. They pick one
card from the unsorted set and insert it at the right position in the
already sorted set. That action is repeated until the unsorted set is
empty. For our next example we sort our data not in place as we did
it in the previous example, but we generate a new sorted copy:

#[
unsorted     sorted
<-- k
4
7                     3 <-- result.high
5                     2
6                     1
]#

proc insertionSort(s: seq[int]): seq[int] =
  var j: int # current position in the new, sorted range
  var k = s.len # index one above the still unprocessed range
  var x: int # the value we have to insert next
  while k > 0: # as long as we have still unprocessed entries
    dec(k)
    x = s[k]
    j = result.high # top of sorted range
    result.setLen(result.len + 1) # reserve space for one more entry
    while j >= 0 and x < result[j]: # move the already sorted entries up
      result[j + 1] = result[j]
      dec(j)
    result[j + 1] = x # insert x

import random
proc main =
  var s: seq[int]
  for i in 0 .. 9:
    s.add(rand(100))
  echo s.insertionSort

main()

The commented code in front of the above program code shows at the
left the still unsorted numbers and at the right 3 already sorted
numbers. For the sorting process we need two index variables j and k,
and one variable to store the actual value that we have to insert
called x. The variable k is used as the index of the top entry of the
still unsorted range. To insert the value x in the already sorted
result, we first reserve space for one more entry by calling setLen()
and then iterate over the sorted values and move them one place to
the top. We do that moving to the top as long as we have not already
reached the bottom of the sorted range and as long as the current
entry is larger than the value x which we want to insert. We take the
values from the top of the unsorted range, as that is convenient, but
of course we could pick an arbitrary element from the unsorted range.

Insertion sort has O(n^2) cost, as for each element that we take from
the unsorted range we have to iterate over the sorted range to insert
it. As we have to move elements before we can insert an element,
insertion sort is slow for larger containers. Selection sort, which
is also of O(n^2) should be faster, as it does not use an expensive
shift of many elements.

When you look at the example code you may immediately find two
possible improvements: We do not really need the variable x, as we
can just use s[k] instead. The compiler should optimize the code so
that the subscript operator is not executed multiple time for the
same index k, so use of s[k] or x should make no difference for
performance. And we call setLen() in the outer loop to increase the
capacity of the result sequence by one each. Of course setting
capacity only one time to the value of s would suffer, as obviously
result has the same length as out input data. Another possible
optimization would be to take advantage of the fact that the
destination sequence is sorted, so that we would not have to do
linear search to find the insertion position, but we could use a
binary search. But that would be more complicated and the benefit
would be not large.

Quick Sort

As the name implies this sorting method is one of the fastest. We
will explain it in some detail with various variants as it can teach
us two important concepts: Recursion and avoiding recursion by use of
a stack container.

The idea of the QuickSort algorithm is simple, and the code is also
simple and short, but we have to care for some details like exact
index stop positions. Generally sorting seems to be a O(n^2)
operation, at least from the two traditional sorting methods,
Insertion- and Selection-Sort it seems to be the case. So doubling
the container size seems to increase the needed sorting time by a
factor of four, which is really bad for large arrays or sequences.
The trick of QuickSort is, that instead of sorting a container with n
elements, we just sort the first half and the second half separately,
each with approx n / 2 entries. This would be faster, as 2 * (n/2)^2
is only half of (n)^2. And we do apply this trick in a recursive
manner on each halved range, until the range is reduced to only one
or two entries. But to make this work we have to partition the full
range in the first half r1 so that all entries of range r1 are
smaller or equal to a median value x, and so that all entries in the
second half r2 are all greater or equal to the median x. Let us
consider an example with six numbers:

5 3 2 8 1 7
1 3 2 8 5 7

To partition that set of numbers we have only to exchange the numbers
5 and 1, with the value 5 or 4 being a possible median. Exchanging
numbers in an array or a seq is a fast O(n) operation, we have to
iterate the container only once. The problem is finding the median.
For picking a perfect median we would need a sorted container so that
we could pick the center entry. But of course our container is
unsorted, if it would be sorted our work would be done already. Note
that even summing up all entries and dividing by n would give only
the average value, not the median. Average and median can be very
different, e.g. for many small numbers and a few very large ones. But
in practice picking an estimation for the median, maybe picking one
by random from the whole range or picking the center entry is good
enough. That choice will not really half the whole range in each
step, but on average it splits the range in two parts with not too
different size. This works really well when the input data looks like
random numbers, but it may work bad in some unlikely cases when all
the input numbers are equal or are already sorted. Already sorted can
indeed occur -- for that case picking the center elements of the range
gives the perfect median, so we will choose that strategy.

Partitioning the full range is basically very simple: We move from
the left to right with index i and stop when we find a value s[i]
that is not smaller than our median x. And we do the same from the
right to the left with index j until we find a value that is not
greater than median x. After we have done that, we can just exchange
the values at s[i] and s[j] and continue. We continue until i is
close to j. The difficult part is to handle this terminating
condition exactly, that is to stop exactly at the right position so
that the first half really contains all entries with value less or
equal to median x, and that the second half contains only entries
with value equal or greater than median x. To make it more clear:
Such a partition decouples the two ranges. Sorting the whole range
would result in the same state as sorting first and second range on
its own.

For writing our actual sorting function we use the fact that Nim like
most modern programming languages support recursion, that is a
function can call itself again. We saw at the beginning of the book a
few examples for that. So we can pass our function the initial full
container, then the function can partition the container in first and
second half and call itself again on the two parts. This way the
actual size of the ranges to sort decreases, and finally recursion
stops when the range contains only one or two elements. For size one
we have to do nothing at all, but for size two we may have to
exchange the two elements if order is wrong.

A naive implementation may create for each function call two new
sequences for the two parts, partition the initial sequence by
inserting the values in one of the new shorter sequences, call itself
on both parts and finally join the parts and return it -- as result or
as var parameter. But that would be really slow. A much simpler and
faster solution is, when we work all the time on the same container,
and just tell the function which range the function has to work on.
So we pass to the function the whole sequence and two integers a and 
b which tell it the range to process: s[a], s[a +1] .. s[b].

from algorithm import isSorted, sort
import std/monotimes

proc qsort(s: var seq[int]; a, b: int) =
  assert a >= 0 # a .. b is the range that we have to sort
  assert b < s.len
  assert b - a > 1 # it may work for smaller intervals, but this is the intended use case
  let x = s[(a + b) div 2] # use element from center of range
  # var x = s[a] div 2 + s[b] div 2 # bad, x may be smaller than smallest entry in range
  # x = s[a .. b].min # worst case test!
  var i = a
  var j = b
  while true:
    while s[i] < x:
      inc(i)
    while s[j] > x:
      dec(j)
    if i < j:
      swap(s[i], s[j])
      inc(i)
      dec(j)
    else:
      break
  dec(i)
  inc(j)
  assert i >= a
  assert j <= b
  if i - a > 1: # still more than 2 entries
    qsort(s, a, i)
  elif i - a > 0: # two entries
    if s[i] < s[a]: # wrong order
      swap(s[i], s[a])
  if b - j > 1: # and the same for the other half
    qsort(s, j, b)
  elif b - j > 0:
    if s[b] < s[j]:
      swap(s[b], s[j])

proc quickSort(s: var seq[int]) =
  if s.len == 2 and s[0] > s[1]: swap(s[0], s[1])
  if s.len > 2:
    qsort(s, 0, s.high)

import random
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize() # give us different random numbers for each program run
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.quickSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  start = getMonotime()
  s.quickSort
  echo getMonotime() - start
  assert(isSorted(s))
  s.shuffle
  start = getMonotime()
  s.sort()
  echo getMonotime() - start
main()

The function qsort() does the whole work. It is called from function 
quicksort() passing it the whole sequence and the interval to sort.
For the first call the interval is the whole content of the seq, from
s.low to s.high. Function qsort() first asserts that the range is
valid, that is that b > a and that both indices are valid positions
in the sequence s. That check makes it easier to find stupid errors,
the assert is automatically removed when we compile finally with
-d:release. We set the iterating indices i and j to the interval
boundaries a and b and enter an outer loop. In that outer loop we let
run i and j to the center of the interval as long as the actual entry
at the position i or j belongs in the range. If both inner loops have
stopped, we swap the entries at position i and j. As positions i and 
j contain now again valid entries, we can move both indices one step
further to the center. If i and j becomes the same we are done.
Unfortunately both may stop too late, so we move both one position
back after the outer loop has terminated. You may create a small
example with pen and paper to recognize how the indices behave in
detail and why fixing by one is necessary. The remainder of the qsort
() proc is really easy: For both partitions we check if the interval
size is still larger than two entries, in that case we call qsort()
again to continue with partition and sorting. But if the size is two
entries, then we just swap() them if the order is wrong. If the size
of the range is just one, we have nothing to do at all.

We test our quicksort() proc by calling it from another proc called 
main(). In that main() we fill a seq with random integer values, and 
shuffle() and sort() it a few times. Shuffle() reorders the entries
by random. After our call of quicksort() we call isSorted() from
Nim's standard library to check the success of our sorting. After
these tests, which does some warmup of the CPU for us, we add more
random entries and again sort and test it, while we record the needed
time with module monotimes as we did before in the Timers section. To
get a feeling about the performance of our sorting proc we shuffle()
again and sort this time with the sort() proc from Nim's algorithm
module. Sort() from algorithm module uses currently another sorting
method called merge sort, which has the advantage that it is a stable
sorting algorithm, but it may be a bit slower than quicksort. And 
sort() from algorithm may pass a cmp() proc around, which may cost
some performance, while our plain, non generic proc compares entries
directly with < and > operators. So it is not surprising that our 
proc is a bit faster.

You may wonder if it is really necessary to pass the sequence s for
each call of qsort(), as for all time the same seq is used. Indeed
Nim support nested procs, so we could just make qsort() proc local to
quicksort() proc and let qsort() work (as a closure) on the s
variable of proc quicksort(). But this does not compile currently,
sequences can not be used by closure procs. But actually passing the
sequence to proc qsort() should be only a minimal overhead.

One general problem of QuickSort is that the sort is not stable. When
we sort an already sorted sequence again, entries with same value may
move. For plain numbers that is not really a problem, we do not
really notice it, as we can't mark a number in some way, plain
numbers are indistinguishable just as elementary particles like
electrons and protons are. But when we sort a container with objects
by some field, then we notice that objects with same value for
sorting may move. The other problem of QuickSort is a generally
problem of recursive algorithm: Each new call of a proc generates
some stack usage, as proc parameters may be passed on the stack and
because the proc may allocate its local data variables on the stack.
So many nested calls may need a very large stack, the program may
fail with a stack overflow error. Generally, we have no real problem
with stack overflow, as for each partition the size of the two new
partition is nearly halved, so that process stops soon. But imagine
someone prepares a special data set for our sort proc. That data may
be prepared in such a fashion that at the center of each range, where
we pick the estimated median value from, always an extreme values is
stored. So our partition would work very badly, in each step we would
get a new range with only one element, and one with n - 1 elements.
So the recursion dept would go very deep, and the performance would
be very bad also. Preparing such a data set would be difficult, but
possible in theory. One way to protect us from that attack would be
to select the median by random. But unfortunately all strategies
different from picking the leftmost, the center, or the rightmost
entry as median are not very fast and make the whole sorting
significantly slower. Note that the strategy of not picking a single
element as median, but calculating a median value, works generally,
but has some shortcomings: s[a] div 2 + s[b] div 2 would not work
when both values are odd, as we then would get a value that can be
smaller than all of our entries and our function would fail. We would
have to add one to the average value when both summands are odd, and
that fix cost performance again. And calculating the average by (s[a]
+ s[b]) div 2 could generate an overflow when both summands are
large.

Because of the stack size restrictions we have a good motivation to
show how we can replace recursion with plain iteration, when we
provide a "buffer" variable that acts as a data stack. For each new
partition of our data we have to put only the two bounds a and b on
that data stack, which is not that much as a recursive proc would put
on the real computer stack. The modifications to our code from above
are tiny:^[42]

from algorithm import isSorted, sort
import std/monotimes

proc qsort(s: var seq[int]) =
  var stack: seq[(int, int)]
  var maxStackLen: int
  stack.add((s.low, s.high))
  while stack.len > 0:
    if stack.len > maxStackLen:
      maxStackLen = stack.len
    var (a, b) = stack.pop
    assert(a >= 0 and b < s.len and b - a > 1)
    let x = s[(a + b) div 2]
    var (i, j) = (a, b)
    while true:
      while s[i] < x:
        inc(i)
      while s[j] > x:
        dec(j)
      if i < j:
        swap(s[i], s[j])
        inc(i) ; dec(j)
      else:
        break
    dec(i); inc(j)
    # assert(i >= a and j <= b) caution, this is not always true!
    if i - a > 1:
      stack.add((a, i))
    elif i - a > 0:
      if s[i] < s[a]:
        swap(s[i], s[a])
    if b - j > 1:
      stack.add((j, b))
    elif b - j > 0:
      if s[b] < s[j]:
        swap(s[b], s[j])
  echo "Max Stack Length: ", maxStackLen

proc quickSort(s: var seq[int]) =
  if s.len == 2 and s[0] > s[1]: swap(s[0], s[1])
  if s.len > 2:
    qsort(s)

import random
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.quickSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  start = getMonotime()
  s.quickSort
  echo getMonotime() - start
  assert(isSorted(s))
  s.shuffle
  start = getMonotime()
  s.sort()
  echo getMonotime() - start
main()

We added a variable called stack which is a seq which stores integer 
tuples. The qsort() proc first stores the borders of the whole seq on
the stack, and then executes a loop which takes in each iteration a
set of two borders from the stack and processes that range. It may
sound a bit strange that we start by putting the whole range onto the
stack and then took it from stack immediately at the start of the
loop. But that makes more sense when we look at the bottom of the 
qsort() proc. Instead of recursively calling qsort() again, we just
put the borders of the two new partitions on the stack and continue.
The whole process terminates when the stack becomes empty, as then
all partitions are processed. Note that the actual partition code and
the main() proc are still unchanged. We added a maxStackLen variable
to get a feeling how large our stack has to be. Actually not that
large, as the partition size shrinks in a logarithmic way. So we
could replace the seq that we use now as stack with a plain array, as
sequences have same overhead and the add() is slower than plain index
access. But how can we prepare for worst case attacks? Indeed there
exists a simple solution: Worst case occurs, when first we put a tiny
one element range on the stack and then the large one, as we would
continue with the large one in the same way in the next loop
iteration. The other way round would be fine. If we put the tiny
range on the stack last, next iteration would pick that one and
iteration would stop immediately or at least very soon, as ranges
drops to two or one entries. When an iteration for a range stops, all
ranges pushed to the stack are removed already again, so total stack
size will never become large. So the trick is to just sort the
partitions in a way that we put the larger partition first on the
stack, and the smaller partition second. So next iteration picks the
smaller one and the whole process stops soon. This way a stack array
of 64 entries should be enough, as max needed stack size should be 
log2(2^64) to sort a seq with 2^64 entries.

from algorithm import isSorted, sort
import std/monotimes

proc qsort(s: var seq[int]) =
  var stack: array[64, (int, int)]
  var stackPtr: int
  var maxStackLen: int
  stack[0] = (s.low, s.high)
  while stackPtr >= 0:
    if stackPtr > maxStackLen:
      maxStackLen = stackPtr
    let (a, b) = stack[stackPtr]; dec(stackPtr)
    assert(a >= 0 and b < s.len and b - a > 1)
    let x = s[(a + b) div 2]
    var (i, j) = (a, b)
    while true:
      while s[i] < x:
        inc(i)
      while s[j] > x:
        dec(j)
      if i < j:
        swap(s[i], s[j])
        inc(i); dec(j)
      else:
        break
    dec(i); inc(j)
    # assert(i >= a and j <= b) caution, this is not always true!
    var c, d: int
    for u in 0 .. 1:
      if (i - a > b - j) == (u == 0):
        (c, d) = (a, i)
      else:
        (c, d) = (j, b)
      if d - c > 1:
        inc(stackPtr) # inc before push!
        stack[stackPtr] = (c, d)
      elif d - c > 0:
        if s[c] > s[d]:
          swap(s[c], s[d])
  echo "Max Stack Length: ", maxStackLen

proc quickSort(s: var seq[int]) =
  if s.len == 2 and s[0] > s[1]: swap(s[0], s[1])
  if s.len > 2:
    qsort(s)

import random
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.quickSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  start = getMonotime()
  s.quickSort
  echo getMonotime() - start
  assert(isSorted(s))
  s.shuffle
  start = getMonotime()
  s.sort()
  echo getMonotime() - start
main()

Instead of processing the two new partitions at the end of the qsort
() proc each, we apply only one processing code block now, which we
execute in a loop that is executed two times. At the start of that
loop we assign the actual interval boundaries to the variables c and 
d. That assignment depends on the actual loop index u, so that we
push the larger range always first on the stack. You may modify the
condition u == 0 to u != 0 and observe what happens to the maximum
used stack dept. We could write that condition also with a boolean
loop variable and a xor operator like

    for u in [false, true]:
      if (i - a > b - j) xor u:

We should not believe all what we think

And what seems to be correct. Our nonrecursive function seems to be
fine, and indeed inverting the == (u == 0) condition makes a
difference for random data, so it is correct? Well when we think
about it again the next day we may get some doubts. The outer loop
pops one entry from the stack, but in the loop we may push two new
entries. Pushing the smaller interval helps, as we continue with the
smaller interval in the next iteration and remove it so from the
stack. But the net effect is still that we push one interval onto the
stack for each iteration, and for the worst case that interval
shrinks only by one in each iteration. So it should still not work.

But well, there are rumors that solutions exists. When we think about
it, we may ask our self if we can just continue with one interval in
a loop and push only the other one on the stack. And indeed, that is
possible, and this time we did testing for worst case scenario:

from algorithm import isSorted, sort
import std/monotimes

proc qsort(s: var seq[int]) =
  var stack: array[64, (int, int)]
  var stackPtr: int = -1 # empty
  var maxStackLen: int
  var a = s.low
  var b = s.high
  while true:
    if b - a == 1: # done with actual interval, but we may have to swap()
      if s[a] > s[b]:
        swap(s[a], s[b])
    if b - a > 1: # interval has still more than two entries, so continue
      discard
    elif stackPtr >= 0: # get next interval from stack
      (a, b) = stack[stackPtr]; dec(stackPtr)
    else:
      break # all done
    if stackPtr > maxStackLen:
      maxStackLen = stackPtr
    assert(a >= 0 and b < s.len and b - a > 1)
    let x = s[(a + b) div 2]
    # let x = s[a .. b].max # worst case test! Slow, test with smaller container size.
    var (i, j) = (a, b)
    while true:
      while s[i] < x:
        inc(i)
      while s[j] > x:
        dec(j)
      if i < j:
        swap(s[i], s[j])
        inc(i); dec(j)
      else:
        break
    dec(i); inc(j)
    # assert(i >= a and j <= b) caution, this is not always true!
    if (i - a < b - j): # put large interval on stack and cont. directly with the small
      swap(i, b)
      swap(a, j)
    if i - a > 1: # interval has more than two entries, needs further processing
      inc(stackPtr) # inc before push!
      stack[stackPtr] = (a, i)
    elif i - a > 0: # two entries, we may have to swap()
      if s[a] > s[i]:
        swap(s[a], s[i])
    (a, b) = (j, b) # the smaller interval, we continue with that one
  echo "Max Stack Length: ", maxStackLen

proc quickSort(s: var seq[int]) =
  if s.len == 2 and s[0] > s[1]: swap(s[0], s[1])
  if s.len > 2:
    qsort(s)

import random
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.quickSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  start = getMonotime()
  s.quickSort
  echo getMonotime() - start
  assert(isSorted(s))
  s.shuffle
  start = getMonotime()
  s.sort()
  echo getMonotime() - start
main()

The modifications to the code are again tiny. We use an outer while
true: loop, which continues with interval a .. b until its size is
less than three entries. Then its pops() a new interval from the
stack. At the end of that outer loop, we put only one interval on the
stack and directly continue with the other interval. But which
interval should we push on the stack and which one should we process
directly further in the outer loop? The solution is to process the
smaller interval in the outer loop further, as we are soon done with
it. For processing that smaller interval, we may push some more
ranges onto the stack, but we come to interval sizes of less than
three soon and then we start popping intervals from the stack. And
when we are done with that, we pop the larger interval again from the
stack. This way the worst case, where we pick each time a min or max
value as median, has the smallest stack consumption, that is one
entry. We push the large interval with size n - 1 on the stack,
continue with the tiny one entry range, which signals that we are
done with that interval in the next loop iteration and so the just
pushed n - 1 interval is popped from the stack again. This continues
in this way. Slow, but minimal stack consumption.

From the above code it becomes clear that for our initial recursive 
qsort() function changing the order in which we process the
partitions would not really help, as we continue the recursion until
all is processed. There is no intermediate pop() involved.

Maybe you still wonder why the tiny inner loops use the conditions 
while s[i] < x: and not while s[i] <= x: as we said that both
partitions are allowed to contain the median element. Well with <=
there would be no guaranteed stop condition for the interval, so
indices could run out of the interval. Using an additional condition
like and i <= b would make it slower. Another possible modification
would be to not use inner while loops at all. Tiny while loops with
only one simple termination condition are fast, but the inner while
loops would always terminate fast for random data. So we may try
instead something like

    while true:
      if s[i] < x:
        inc(i)
      elif s[j] > x:
        dec(j)
      else:
        if i < j:
          swap(s[i], s[j])
          inc(i); dec(j)
        else:
          break
    dec(i); inc(j)

You may try that variant yourself, or maybe look for other variants
in internet sources or textbooks. Our intention in this section was
not to present a perfect sorting function, but to teach you some
basic coding strategies and related traps.

Merge Sort

Initially we did not intend to discuss the actual MergeSort algorithm
at all, as it is a bit more complicated and whenever we may have seen
a sketch of it somewhere, it is generally not easy to remember
details.^[43] But MergeSort is indeed an important algorithm, it is
used by default in Nim's standard library and as we have discussed
QuickSort already in some detail, we should be prepared for MergeSort
now. When we regard the name Merge, which is some form of joining
multiple sources to one destination, we may begin to remember the
idea of MergeSort: The trick of QuickSort was, that we tried to split
in a recursive manner the set of all container elements into two
subsets, which we can process separately. That improves performance,
as sorting is basically an O(n^2) process, and 2 * (n/2)^2 is only
half of n^2. For QuickSort we partitioned the initial range into two
ranges a and b, where all elements of range a are less or equal to a
median element x, and all elements of range b are greater or equal to
the median x. That way we decoupled the two ranges, we can sort a and
b independently and get a fully sorted range. MergeSort starts also
with splitting the full range into two parts, but it really only
splits, without any form of rearrangement. Then it continues with
sorting each part independently. That sounds strange at first, as we
get two sorted parts a and b, but of course we can not simple append
one to the other. The idea of the whole algorithm become clear
immediately when we think about how we can find the smallest elements
of the joined content from a and b. That one is obviously the
smallest value of a, or the smallest value of b, min(a, b) = min(min
(a), min(b)). But when a and b are already sorted, then the minimal
value of each is the first element, and so one of these elements at
index position zero is the smallest one for a and b joined. And this
condition holds even when we pick and remove the smallest element
from a or b.

So the basic algorithm is this: Split the whole container in parts a
and b, sort them separately. Then create a new, sorted container by
iterative picking the first elements from a or b, which ever is
actually the smaller one.

Unfortunately it seems to be impossible to sort the initially
container in place in this way, as we would take always elements from
the front from both and put them at the front of the destination, so
that newly inserted elements could overwrite still unprocessed
elements. So we will try to give a sketch of an very slow unoptimized
algorithm which creates and returns a new sorted container first to
lean the fundamental idea of the algorithm:

proc msort(s: seq[int]; a, b: int): seq[int] =
  assert(b - a >= 0)
  if b - a == 0:
    result.add(s[a])
    return
  elif b - a == 1:
    var (a, b) = (s[a], s[b])
    if a > b:
      swap(a, b)
    result.add(a)
    result.add(b)
    return
  result = newSeq[int](b - a + 1)
  var sl = result.len
  assert b - a > 1
  var m = (a + b) div 2
  assert m >= a
  assert m < b
  var s1, s2: seq[int]
  s1 = msort(s, a, m)
  var (i, j) = (a, m)
  s2 = msort(s, m + 1, b)
  assert s1.len + s2.len == result.len
  var (k, l) = (m + 1, b)
  var l1 = s1.high
  var l2 = s2.high
  while sl > 0:
    dec(sl)
    if l1 >= 0 and l2 >= 0: # merge
      if s1[l1] > s2[l2]:
        result[sl] = s1[l1]
        dec(l1)
      else:
        result[sl] = s2[l2]
        dec(l2)
    else: # plain copy
      while l1 >= 0:
        result[sl] = s1[l1]
        dec(sl); dec(l1)
      while l2 >= 0:
        result[sl] = s2[l2]
        dec(sl); dec(l2)

proc mergeSort(s: seq[int]): seq[int] =
  if s.len < 2:
    return s
  msort(s, s.low, s.high)

import random, algorithm
proc main =
  var s, st: seq[int]
  randomize()
  for i in 0 .. 9:
    s.add(rand(100))
  st = s
  echo s.mergeSort
  assert(s.mergeSort == st.sorted)

main()

The example above is indeed not very complicated, the most daunting
task is to get all the indices right. Due to the involved recursion
and the fact that we have to do the sorting of the two parts first,
before we can join them, debugging would be not easy. So we added
many asserts to early find stupid errors.

The function mSort() starts by checking if the range to sort has only
one or two entries and handles this simple case directly. Then we
allocate the result sequence, find the center position m of the
interval a and b and sort the intervals a .. m and m + 1 .. b each
into a new sequence s1 and s2. We have decided that we will do the
merging of s1 and s2 into the result sequence from the back, starting
with the largest elements. This way we can count down to zero, which
is a bit faster and simpler. We do an actual merging as long as s1
and s2 have still elements left. An we do the merge from end to
start, we have to always pick the largest element from s1 or s2. If
we have used all the elements from at least one of the sequences s1
or s2, then there is nothing more to merge, we can just copy the
remaining elements from the other seq that has elements left. Of
course the above example is very slow, as we allocate the result and
additional the sequences s1 and s2, and as we have to copy many
elements.

When we now think again about the problem, we get the feeling that
allocating three sequences is really too much. When we regard both,
in place sorting and sorting with a return value, we may discover
that in place sorting allows us to partly reuse the passed container,
and we have only to allocate one additional seq with half the size of
the passed container. We copy the second half of the passed container
into a newly allocated seq, and then can merge values from the new 
seq and from the first half of the passed seq to positions starting
at the end of the passed seq, without overwriting values that we
still have to process.

This way our code becomes even shorter and basically simpler -- we
have only to care very exactly to use the right index positions.

proc msort(s: var seq[int]; a, b: int) =
  assert(b - a >= 0)
  if b - a == 0:
    return
  elif b - a == 1:
    if s[a] > s[b]:
      swap(s[a], s[b])
    return
  var m = (a + b) div 2
  assert(m >= a and m < b and b - a > 1)
  var sh: seq[int] = s[(m + 1) .. b]
  var ls = b + 1
  msort(s, a, m)
  msort(sh, sh.low, sh.high)
  var lh = sh.high
  var lm = m
  while ls > a:
    dec(ls)
    if lh < 0:
      assert ls == lm
      break
    elif  lm < a:
      while lh >= 0:
        s[ls] = sh[lh]
        dec(ls); dec(lh)
    else:
      if sh[lh] > s[lm]:
        s[ls] = sh[lh]
        dec(lh)
      else:
        s[ls] = s[lm]
        dec(lm)

proc mergeSort(s: var seq[int]) =
  msort(s, s.low, s.high)

import random, algorithm, std/monotimes
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.mergeSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  var st = s
  start = getMonotime()
  s.mergeSort
  echo getMonotime() - start
  start = getMonotime()
  st.sort()
  echo getMonotime() - start
  assert s == st
main()

For the merging process, we first test if one of the source areas is
already exhausted. If all entries from the newly allocated seq sh are
consumed, then we can just stop, as continuing would only copy
elements of the passed container with equal index positions. And if
all elements from the first half of the passed container are
consumed, we can just copy the elements from sh into the passed var
container. Only if both sources have elements to process left, we
have to do the actual merging.

When we run the program above, we may find that it is about 50%
slower than our QuickSort functions. The reason for that may be that
we have to allocate the temporary seq sh, fill it with values, and
merge it back. And the total memory consumption is high, our
recursive function calls consumes for the buffers sh totally the same
amount as the initial container. The advantages of MergeSort is that
there is no worst case as for QuickSort, as we have not to select a
median but can split the range just at the center, and that the sort
is stable, that is merging does not exchange the position of elements
with same value.footnote[ Well to ensure this, we may have to check
the actual merge condition, do we have to test for < or <=. Currently
we do not really care for that, but is is clear and well known that
merge sort can be stable.]

When we think a bit more about the algorithm above and maybe try to
sketch the recursive steps for a short sequence with pencil and paper
we get the strong feeling that the additional buffer sh is only
needed for the merging step, and as the merging process occurs from
bottom to top (containers with only one or two entries are returned
immediately, which are merged to a larger section, and these larger
section is again merged ... ) one single buffer could be used. So we
have modified our example again. Now the quicksort() proc allocates
the buffer seq with half the size of the actual data container, and
we pass that buffer recursively to the qsort() proc and use it for
the merging only. We call recursively qsort() on the first and second
half of the full range that we have to sort, then copy the sorted
second half into the buffer and merge the fist half and the buffer
into the final location.

proc msort(s, sh: var openArray[int]; a, b: int) =
  assert(b - a >= 0)
  if b - a == 0:
    return
  elif b - a == 1:
    if s[a] > s[b]:
      swap(s[a], s[b])
    return
  var m = (a + b) div 2
  assert (m >= a and m < b and b - a > 1)
  msort(s, sh, a, m)
  msort(s, sh, m + 1, b)

  var ls = b + 1
  var lh = b - m - 1
  var lm = m
  #sh[sh.low .. lh] = s[m + 1 .. b]
  for i in 0 .. lh: # a bit faster
    sh[i] = s[m + 1 + i]
  while ls > a:
    dec(ls)
    if lh < 0:
      assert ls == lm
      break
    elif  lm < a:
      while lh >= 0:
        s[ls] = sh[lh]
        dec(ls); dec(lh)
    else:
      if sh[lh] > s[lm]:
        s[ls] = sh[lh]
        dec(lh)
      else:
        s[ls] = s[lm]
        dec(lm)

proc mergeSort(s: var seq[int]) =
  if s.len == 0: return
  var sh = newSeq[int](s.len div 2)
  msort(s, sh, s.low, s.high)

import random, algorithm, std/monotimes
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.mergeSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  var st = s
  start = getMonotime()
  s.mergeSort
  echo getMonotime() - start
  start = getMonotime()
  st.sort()
  echo getMonotime() - start
  assert s == st
main()

An additional tiny performance improvement results from the fact that
we now pass the seq s and the buffer sh as openArrays. This is
generally a good idea, as we can so sort arrays also with the same
sorting proc, and it improves performance, as this way the actual
data buffer is directly passed to the qsort() proc, while passing a 
seq means that we pass the opaque seq structure which contains a
pointer to the actual data. In the example above we do not only call 
isSorted() to prove our result, but we really sort a copy of our data
with a sorting routine from Nim's standard lib to ensure that our
result is not only sorted data, but that it is indeed based on the
actual values. That is a good idea, because although the algorithm is
simple, getting some indices wrong may give us wrong results.

Our recursive merge sort routine is really not that bad. It does a
fast stable sort and needs only a single buffer of half the size of
our actual data. As the interval size is halved in each recursion
step, the max recursion depth should be only 64 for a gigantic
container with 2^64 elements. As the recursion occurs in a dept first
fashion, that is msort() calls itself until range size is only one or
two elements, then recursion continues in other branches of the whole
sorting three, there should be never more than log2(n) actual
recursion steps stored on the CPU stack. Non recursive, iterative
merge sort algorithm exists, but converting the recursive algorithm
in an iterative one is not that simple as for the QuickSort case. The
reason is that qsort() has first to partition the input range, then
call msort() on both subranges and finally do the merging. We will
not try to present in this book an iterative msort(), which you may
find in textbooks or somewhere in the internet, as that would be a
bit too much for an introducing course.

We did the QuickSort and the MergeSort in a top down fashion, that is
we split the initial container in two subpartitions and continue in
this way until we have only ranges with one or two entries, and than
do the merging from bottom to top. For MergeSort we could just start
from the bottom joining single adjacent elements to sorted tuples of
two entries, when done with that merging the tuples of two to sorted
tuples of 4 and so on. This would work really well when the initial
number of elements in our container is a power of two, and it would
work well iterative without recursion. Unfortunately in most cases
the container size is not a power of two, so such a bottom up merge
sort needs some math to get all the ranges sizes right. But the
bottom up process has a big disadvantage on modern hardware, as it
has no locality for element access operations: We would iterate
repeatedly over all the container entries in sequential order, so the
CPU cache can not support our element access operation that well.

Other known sorting algorithms are the easy, funny and slow
BubbleSort, or Shell- and Shaker-Sort. But these are not used in
practice. As an exercise you can try to make our QuickSort or
MergeSort generic and pass a cmp() proc, and make it work for sorting
in ascending and descending order. Or, you may try to fall back to
selection sort when the partitions become small. In theory
SelectionSort is faster for ranges of only a few dozen elements, but
when we have to do a decision which one to use inside of the qsort()
or msort() proc, then this decision compensates generally the
advantages again, so that the net benefit is tiny. Of course we would
have to test all of our sorting procs for special cases, that is for 
seqs of length 0, 1 or two, and for sequences with all entries equal,
all inversely sorted or presorted. And we would have to check how
performance is when we sort not containers containing plain data like
numbers, but containers which elements are objects, strings or again 
arrays or sequences. string sorting is special for various reasons: 
strings in Nim are an opaque object with a pointer to the actual
data. This is some indirection, and the actual data can be located
somewhere in the RAM in a cache unfriendly manner, so the actual
comparison process can be slow. Swapping of strings is also special,
as swap() generally just does a pointer exchange for the data areas,
and does not have to copy the actual data. For sorting containers
where each entry is an array (of characters), swap would have to copy
the data content.

Finally we should mention that the Python language uses a complicated
sorting algorithm called TimSort which is a smart mix of various
sorting algorithm.

References:

  * https://en.wikipedia.org/wiki/Timsort

Some small exercises

Removing adjacent duplicates

Removing duplicates from containers is a common programming task.
When we know in advance, that we do not want to store the same data
value more than once, and that the order of the stored values does
not matter, then we may consider using some form of sets or hash sets
. If insertion order matters, then the standard library may provide
some form of ordered or sorted sets for us. But for this exercise we
will assume that we have values stored in a sequence, and we want to
remove the adjacent duplicates. A typical use case for that is when
we have stored a path of 2D or 3D positions. When we insert, move or
delete a position value, it may occur that we get duplicates, with
zero spatial distance between the two neighbored positions. In that
case it is generally desired to remove the duplicate. A similar use
case may be a text file stored as a sequence of words, where adjacent
duplicated words may indicate a typo. To keep our example short and
simple, we will use as data type a seq[int]. Using other data types
or creating a generic proc should be not difficult for the reader.

# [1, 4, 4, 2, 5, 5, 5, 1] ==> [1, 4, 2, 5, 1]

Before you continue looking at our provided example code, you may
think about this task yourself, maybe take a piece of paper and a
pencil and sketch the algorithm. It is really not difficult, maybe
too easy for you when you have carefully studied the proceeding
sections of the book or when you have already some programming
experience. When we create a proc for this task we have to decide
first if the proc should work on the passed in seq in-place or if it
should return the processed result and leave the original data
unchanged. Often returning a copy is easier, but for our task the
common use case seems to be more an algorithm that works in place. So
we will provide the in place algorithm here and leave the version
returning a processed result as an optional exercise to the reader.
Note that returning a processed copy needs to allocate the seq for
the result, and maybe later the memory management system has to free
the result data again, which is some additional effort. So we may
guess that the in-place proc is faster, as long as we do not really
need the copy. When needed we can use the dup() macro of the sugar
module to use our in place proc as one that works on a copy and
returns this copy, without modifying the input.

The basic idea of our algorithm is that we iterate through the whole 
seq and pick an element at the current location only if it is not
identical to the proceeding element. For the case that our seq is
empty or contains only one single element, we have obviously noting
to do and can return immediately. So our code may look like

proc deTwin(s: var seq[int]) =
  if s.len < 2:
    return
  var i, d: int # d is the position where we copy the elements that we want to keep
  while i < s.high:
    inc(i)
    if s[d] != s[i]:
      inc(d)
      s[d] = s[i]
  s.setLen(d + 1)

var h = @[1, 4, 4, 2, 5, 5, 5, 1]
detwin(h)
echo h # [1, 4, 2, 5, 1]

We use two positions, the actual position in the input data denoted
as i, and the destination position d. Both start with the default
value zero. To keep full control over the iterating process, we do
not use a for iterator in this case, but a plain while loop for the
index of the actual position. In the while loop body we compare the
value at the current index position s[i] with the value that we
picked before, which is s[d]. If the values are not identical, we
pick the current value s[i], otherwise we just skip it. With picking
a value we mean that we copy it from index position i to index
position d. Of course this can only work, when d is never larger than
i, as otherwise we would destroy our still unprocessed input data at
positions s[i + 1]. The loop body is really simple, but getting the
indices right needs some care: We have to ensure that i and d starts
at the right positions, that we increase i and d when necessary, and
that the loop terminates when all input data is processed. Obviously
the first comparison should compare s[d == 0] with s[i == 1]. So d
and i can get initial values zero each, when we increase i already
each time at the start of the loop. The destination position d starts
with zero, as we always accept the first element, and d increases
only when we have accepted one more element. The loop is executed as
long as i is less than s.high. Finally we have to set the new length
of s to d + 1. The value d + 1 results from the fact that we always
accept the first element, and for each more accepted element d is
increased, so the total number of accepted elements is d + 1. The
carefully reader may wonder if the first two lines of the proc, where
we test for the trivial case, are really necessary, or if these cases
can be covered by our while loop already. Well, generally it is a
good idea to avoid unnecessary tests for trivial cases when possible,
as that tests may increase code size and cost some tiny bit of
performance. But in this case we have two trivial cases -- empty seq
and seq with only one element, which can be not covered well with a
loop with only one simple termination condition. And of course we
should try to make the condition of the while loop as simple as
possible, that is avoid additional boolean conditions with and or or
operators for best performance. Further you may wonder if our picking
strategy is really optimal, as for a seq with no adjacent duplicates
we still copy all the elements. Yes indeed, but for the general case
with duplicates we have to do the copy, and such a copy operation is
really fast. And additional tests with an if condition should cost
some performance. Maybe we could have used two loops in the proc
body, one that just accepts elements without a copy operation as long
as no duplicates are found, and a second loop like the one from above
which then has to do copy operations to move the elements to the
front. You may try that yourself and measure the performance for
various input data.

Array difference

The difference of two arrays or sequences A and B is the set (A - B)
of values that are contained in A but not in B.

[1, 2, 5, 2, 9, 7, 0] - [7, 4, 1, 10, 7] == [2, 5, 2, 9, 0]

Actually such difference of array or seq containers is not needed
that often, that is why that function may not be provided by the Nim
standard library.^[44] Building such kind of differences is much
easier and faster with sets or hash sets, so whenever possible we
should use these containers from the beginning when we know in
advance that we have to build differences. But sometimes we just have
arrays or sequences, and then we may notice that we need the
difference. When the order of the elements does not matter, we may
just convert both containers to sets or hash sets, build the
difference and then maybe convert that difference back to a seq. But
there are use cases where we really want to work with array or seq
containers, maybe because we want to iterate over the container, want
that values can be contained multiple times or always keep the
insertion order.

So let's create an algorithm to do this task. A naive strategy would
be to iterate over container A and delete each element that is also
contained in B. But that would be slow, and may not work at all, as
deleting elements while we iterate over the seq does generally not
work at all. We will create a proc called `-` which can be used as an
operator to build the difference of two arrays or sequences and which
returns the difference as a new seq, and a `-=` proc which removes
the elements of b from a in place and is also used as an operator.

import sets

proc `-`*[T](a, b: openArray[T]): seq[T] =
  let s = b.toHashSet
  result = newSeq[T](a.len)
  var i = 0
  for el in a:
    if el notin s:
      result[i] = el
      inc(i)
  result.setLen(i)

proc `-=`*[T](a: var seq[T]; b: openArray[T]) =
  let s = b.toHashSet
  var i, j: int # both start with default value zero
  while i < a.len:
    if a[i] notin s:
      a[j] = a[i]
      inc(j)
    inc(i)
  a.setLen(j)

proc main =
  let a = [1, 2, 5, 2, 9, 7, 0]
  let b = [7, 4, 1, 10, 7]
  echo a - b # @[2, 5, 2, 9, 0]
  echo b - a # @[4, 10]

  var x = @a
  x -= b
  echo x # @[2, 5, 2, 9, 0]

main()

To make the lookup for elements contained in b fast, we convert b to
a hash set, for which lookup time is in principle independent of the
size of the container, which is called O(1) in the big O notation. We
make the two procs generic and use the data type open array for the
two passed arguments so that our procs can be used for arrays as well
as for sequences. The exception is the first var parameter of the `-=
` proc, which has to be a seq obviously, as arrays have a fixed size
and can not shrink. For the `-` proc we pre-allocate the returned 
result variable with a size of a.len, so that we can avoid
re-allocations. Then we iterate over a with a for loop, and copy the
current element to the result seq when the value in not contained in
the hash set. We use the subscript operator [] to copy the picked
elements at position i in the result seq, which is faster than
starting with an empty result seq and appending the picked elements.
As we initialize the result seq with the size of container a, we have
finally to call result.setLen(i) to shrink the size to the number of
actually picked elements. The presented `-=` proc is a bit more
complicated, as we process seq a in place. We use an approach similar
as we did in the deTwin() proc in the previous section, that is we
use two index positions i and j, and copy elements from current
position i to position j if the value is not contained in the lookup
set. Again finally we have to set the size of seq a to the number of
picked elements.

While the two presented procs may be actually useful in same cases,
they are more presented as an exercise here. As a smart user of the
Nim forum showed us, we can get a very similar behaviour by use of
the filter() proc in combination with the => operator of the sugar
module:

import sequtils, sets, sugar

let a = [1, 2, 5, 2, 9, 7, 0]
let b = [7, 4, 1, 10, 7]

let bSet = b.toHashSet()
echo a.filter((x) => x notin bSet)

References:

  * https://forum.nim-lang.org/t/7753

Binary search

Maybe you can remember that some decades ago your parents have used
phone books and dictionaries build of paper sheets, filled with
printed text sorted alphabetically? Well, that alphabetically
ordering was done with a purpose: When searching for a name or a
word, we could just open the book somewhere in the middle. If the
name or word that we searched for was ordered alphabetically before
the content of the current page, then we continued searching for that
term in the lower half, otherwise in the upper half. That procedure
was continued until the searched entry was found, or until it was
observed that it was not available at all. This type of search in an
ordered data set is called binary search, half-interval search, 
logarithmic search or binary chop. As each repetition halves the
remaining data set, it is much faster than a linear search in
unordered data.^[45]

To use this type of search strategy on the computer, we store our
data sorted in an array or a seq. Creating a proc to do the search is
basically very easy, but we have to care for some details:

# 1 2 3 4 5 6 7 8 # search for v == 7
# a     p       b
#         a p   b
proc binarySearch(s: openArray[int]; v: int): int =
  var a, b, p: int
  a = s.low
  b = s.high
  while a <= b:
    p = (a + b) div 2
    if v > s[p]: # continue search in the upper partition
      a = p + 1
    elif v < s[p]: # continue search in the lower partition
      b = p - 1
    else: # we have a match
      return p
  return -1 # indicate no match

var d = [1, 3, 5, 7, 11, 13]

for i in 0 .. 15:
  let res = binarySearch(d, i)
  if res >= 0:
    echo "Found value ", i, " at position ", res

To keep our example code as simple as possible, we do our search on
an ordered array or seq of integers. The value which we search for is
passed as second integer argument to the proc called binarySearch().
The first tree lines of the example program shows some example data
consisting of the ordered numbers 1 .. 8. Here we use a consecutive
sequence of numbers, but the actual numbers are fully arbitrary, as
long as the sequence is ordered by the value of the numbers in
ascending order. Let a be the index of the lowest number in the seq,
and b be the index of the largest number. If the number we search for
is contained in the seq, than that number must be located at an index
position greater or equal to a and an index position less or equal to
index position b. For the index position p near the center the
obvious choice is (a + b) mod 2. For our example, we assume that we
search for a value of 7. The first index position of p is (0 + 7) div
2, which is 3 containing value 4, which is lower than the searched
value 7. So we would have to continue our search in the upper half,
setting the new lower bound of the range to search to p or p+1. The
upper boundary remains unchanged for this case, and we continue with
a new value p = (a + b) mod 2.

The program code follows this strategy in a straight way. We start
with a == s.low and b == s.high, and the loop continues as long as a
<= b. Note that we use as new boundaries not the value of p, but p +
1 if we continue with the upper half, and p - 1 when we continue with
the lower half of our data. We can do that offset of one, as we have
investigated position p already. This offset of one does not only
speed up things, as the new interval is smaller by one this way, but
actually guarantees that the interval size permanently shrinks and
the algorithm terminates always. Without that offset, for the case b
== (a+1) we would get a value p == (a + a + 1) mod 2, which is again 
a, and we might set the new a then to p, which is again the previous 
a. So the range would not always shrink, and our algorithm would not
really terminate for some data values. You may test that yourself
when you remove the offset -- the algorithm would not terminate for
some data. And finally you should try to make that proc generic, and
then maybe search for a word in an ordered list of words.

Click to see a possible solution

proc binarySearch[T](s: openArray[T]; v: T): int =
  var a, b, p: int
  a = s.low
  b = s.high
  while a <= b:
    p = (a + b) div 2
    if v > s[p]: # continue search in the upper partition
      a = p + 1
    elif v < s[p]: # continue search in the lower partition
      b = p - 1
    else: # we have a match
      return p
  return -1 # indicate no match

var d = ["Algol", "Basic", "C", "D", "Elexir", "Erlang", "F#", "Haskell"] # sorted!

for i in ["Oberon", "Nim", "Basic", "Ada", "Erlang"]:
  let res = binarySearch(d, i)
  if res >= 0:
    echo "Found value ", i, " at position ", res

References

  * https://en.wikipedia.org/wiki/Binary_search_algorithm

Integer to string conversion

Have you ever asked yourself what actually is happening when we print
the value of an integer variable on the screen, maybe by use of the 
echo() procedure? Before echo() can print the value, the integer has
to be converted to a text string somehow. Some people may think that
this conversion is trivial. But as we know already that in our
computer all the data is stored in abstract binary form, we know
better. But maybe there is some magic available to do this task? Well
when we regard the existence of C libs or the Nim standard library as
that magic solution, then the answer is yes. But in this section we
will assume that we will not use a C library or a function of the Nim
standard library for the conversion of integers to strings, but do it
our self. Indeed this conversion task is an interesting exercise,
from which we can learn a lot, much more than from using a gaming lib
and moving some sprites over the screen. Even when you already know
how to do it, you may learn some new. We will start with the
question, how we can convert an int i with a numeric value 0 <= i <=
9 to a single character digit matching this value, and then we will
present a first procedure to convert larger integers to strings.
After that we will try to improve that first procedure, we will make
it generic and will investigate which problems may occur on
restricted hardware like small microcontrollers and embedded systems.

As there is no magic available, let us recall how we can print the
characters 0 .. 9: Well, first we have to remember that the 256 ASCII
characters maps directly to the integers 0 .. 255, e.g. the character
A is mapped to the integer value 65. We can use the conversion
functions int() or ord(), and char() to convert between the two data
types, where these conversion functions does no real work at all, the
content of the variable is just interpreted as a different type. That
is, a plain cast would do the same for us:

var i: int8 = 65
echo char(i)
echo cast[char](i)
var c: char = 'A'
echo ord(c)
echo int8(c)
echo cast[int8](c)

A
A
65
65
65

The same conversions work of course for all the 256 ASCII characters,
which includes the decimal digits 0 .. 9. So one way to print the 10
digits is

var startPos = int('0')
var i: int
while i < 10:
  stdout.write(char(startPos + i))
  inc(i)
stdout.write('\n')

This works,because the 10 decimal digits follow each other in the
ASCII table. As we can not remember the position of the digit '0' in
that table, we get the position by int('0'). Note that int('0') and
ord('0') is basically the same here, we have no real preference and
use both alternately. The ord() function is generic, always returns
an int and works for ordinal types, enums with holes and distinct
ordinal types, while the int() functions have the advantage that we
have them for different sizes like int8() and also for unsigned
results. We strongly hope that you know well the difference between
the int value 0 and the decimal character digit '0' -- if not you may
read again the section about characters in part II of the book, see
Characters.

With these introduction, you may already have an idea how we can get
the decimal digit for the lowest decimal place of an arbitrary
integer value v: char(v mod 10 + ord('0')). This works, because v mod
10 is the numeric value of the lowest decimal place, that is a value
between 0 and 9, and when we add ord('0') we get the corresponding
position in the ASCII table. Finally we use char() to convert that
numeric value to a char data type, which is only a plain cast, the
compiler reinterprets the bit pattern as a character. So we are
mostly done. To get the following digits, we just divide the initial
integer value by ten to move all one position to the right, and then
we continue with the initial step. We repeat that until the division
by ten gives zero, then we are done. That division by ten may be
still confusing for you -- we know that in decimal notation a division
by ten is a shift right, but why does that work for a number which is
stored in binary form in the computer memory? Indeed it is a bit
confusing. The division by ten works, because it is a pure
mathematical, abstract division operation, fully independent from the
actual representation of the number. Imagine you have a number in the
range 10 .. 19. Now divide that number by ten. Independent how the
number is stored, we will get a new number in the range 0 .. 9, and
that value we can convert to a digit with the method shown above.

So following this strategy, we may get a first intToStr() procedure
that may look like this one:

proc intToStr(a: int): string =
  var v = a
  while true:
    result.add(char(v mod 10 + ord('0')))
    v = v div 10
    if v == 0:
      break

echo intToStr(0)
echo intToStr(1234)
echo intToStr(12345678901234.int)
echo intToStr(int.high)

As output we would get

0
4321
43210987654321
7085774586302733229

Not that bad, but unfortunately we get the digits in reversed order.
And for negative numbers it would not work yet. But that can be
easily fixed. What above code does should be obvious from the
discussion before: We copy the passed integer argument a into a local
variable v of same data type, so that we can modify it, and in the
while loop body we extract the lowest digit and then divide the value
by ten to shift it down to the right. We have to use a while true:
loop with a break statement, because we need at least one loop
execution to get at least one digit, but Nim does not support repeat
loops as known from languages like Pascal. In the loop body we apply
the discussed operation to get the digit of the lowest place, then
divide the actual value by ten and continue, as long that value is
not already zero. When it is zero we can leave the loop, as we do not
intend to print leading zeros.

Creating a proc that prints the digits in the correct order and that
can print the minus sign for negative values is straight forward:

proc intToStr(a: int): string =
  if a == int.low:
    return "-9223372036854775808"
  var v = a.abs
  var i: int
  var res: array[20, char]
  while true:
    res[i] = char(v mod 10 + ord('0'))
    v = v div 10
    inc(i)
    if v == 0:
      break
  if a < 0:
    res[i] = '-'
    inc(i)
  result = newString(i)
  let j = i - 1
  while i > 0:
    dec(i)
    result[j - i] = res[i]

echo intToStr(0)
echo intToStr(1234)
echo intToStr(int.high)
echo intToStr(-0)
echo intToStr(-1234)
echo intToStr(int.low + 1)
echo intToStr(int.low)

We use an array of character for temporary storing the decimal
places, and finally copy the digits into the result string. We
pre-allocate the string with the correct size, and use the subscript
operator [] instead of add() to insert the digits for performance
reasons. Initially we create a copy v with positive sign of the
passed integer argument a, and when the argument was initially
negative, then we add an additional minus sign to the temporary
array, which is finally also copied to the result string. All this is
not difficult, we have only to care that we get all the indices
right. A tiny problem is, that when the passed integer argument has
the value low(int), then applying abs() would generate an overflow
error, see section Binary Numbers in part II of the book if you
forgot it. We fix for that by returning just the correct string for
that unique negative value for now.

The above proc looks not that bad, but maybe we can improve it, maybe
we can avoid the temporary array? The actual problem is, that we do
not know how many total digits the integer argument will need in
advance, and so it is impossible to position all the digits at the
correct position in the result string. A possible solution is to use
a function that gives us the the number of decimal places of an
integer number. Indeed we have such a function available, it is 
math.log10(). Remember, log10(1) is zero, log10(10) is one, log10
(100) is two and so on. So basically what we need. The logarithm
function is not that slow on modern desktop computers, so it should
be OK to use it. At the end of this section we will consider how we
may replace it for tiny microcontrollers which do not provide a FPU.
The improved intToStr() proc may look like this one:

import math

proc intToStr(a: int): string =
  if a == int.low:
    return "-9223372036854775808"
  var v = a.abs
  var i, j: int
  if v > 0:
    i = math.log10(v.float).int
  if a < 0:
    j = 1
    inc(i)
  result = newString(i + 1)
  result[0] = '-'
  while i >= j:
    result[i] = char(v mod 10 + ord('0'))
    v = v div 10
    dec(i)

This proc is very similar to the one before. We call log10() to get a
measure for the number of needed digits. Remember that the logarithm
is undefined for the argument value zero, for that value we use the
default value i == 0. Actually in all cases i + 1 is the total number
of digits that we have to generate -- for the case that we have to
generate a minus sign we increase i by one. We pre-allocate a result
string with i + 1 positions, and put a minus sign at position zero,
which is overwritten in the while loop when the argument was not
negative. As we know the the total number of digits of our number we
can use the variable i to put the digits at the correct positions in
the while loop. The careful reader may wonder if log10(v.float).int
will really work for sure for all integer arguments of v, or if we
better should round the argument like log10(v.float + 0.5).int.
Indeed, with that rounding we should be save.

The next task is to avoid the initial test for int.low. We really
should remove that special case when we prepare to make the proc
generic later. A possible solution is, that we work with uint64
instead with int in the proc body, as in

import math

proc intToStr(a: int): string =
  var v: uint64
  if a == int.low:
    v = uint64(-(a + 1)) + 1
  elif a < 0:
    v = uint64(-a)
  else:
    v = uint64(a)
  var i, j: int
  if v > 0:
    i = math.log10(v.float + 0.5).int
  if a < 0:
    j = 1
    inc(i)
  result = newString(i + 1)
  result[0] = '-'
  while i >= j:
    result[i] = char(v mod 10 + ord('0'))
    v = v div 10
    dec(i)

When we add 1 to int.low, then we can invert the sign, and convert
the value to unit64. To the uint64 value we have to add again 1 to
get the initial sequence of digits. And now we can make the proc
generic:

import math

proc intToStr(a: SomeInteger): string =
  var v: uint64
  when a is SomeSignedInt:
    if int(a) == int.low:
      v = uint64(-(a + 1)) + 1
    elif a < 0:
      v = uint64(-a)
    else:
      v = uint64(a)
  else:
      v = uint64(a)
  var i, j: int
  if v > 0:
    i = math.log10(v.float + 0.5).int
  when a is SomeSignedInt:
    if a < 0:
      j = 1
      inc(i)
  result = newString(i + 1)
  result[0] = '-'
  while i >= j:
    result[i] = char(v mod 10 + ord('0'))
    v = v div 10
    dec(i)

echo intToStr(0)
echo intToStr(1234)
echo intToStr(int.high)
echo intToStr(-0)
echo intToStr(-1234)
echo intToStr(int.low)

echo intToStr(0.uint8)
echo intToStr(123.uint8)
echo intToStr(uint8.high)

echo intToStr(0.uint64)
echo intToStr(123.uint64)
echo intToStr(uint64.high)

echo intToStr(0.uint)
echo intToStr(123.uint)
echo intToStr(uint.high)

We use as parameter type SomeInteger, which allows signed and
unsigned ints of all byte sizes, and in the proc we test with is
SomeSignedInt: if we have to care for the sign and in case of value 
int.low for overflow. The advantage of this proc is, that it works
for all integer types, signed and unsigned. But one disadvantage is,
that always the data type uint64 is used, which may be not available
on microcontroller CPUs. Let us see how a proc for only unsigned
types may look:

import math

proc intToStr(a: SomeUnsignedInt): string =
  var i: int
  var v = a
  if v > 0:
    i = math.log10(v.float + 0.5).int
  result = newString(i + 1)
  while i >= 0:
    result[i] = char(v mod 10 + ord('0'))
    v = v div 10
    dec(i)

echo intToStr(0.uint8)
echo intToStr(123.uint8)
echo intToStr(uint8.high)

echo intToStr(0.uint64)
echo intToStr(123.uint64)
echo intToStr(uint64.high)

echo intToStr(0.uint)
echo intToStr(123.uint)
echo intToStr(uint.high)

That one is really simple and short, so maybe it would indeed make
sense to use this one for the unsigned types. And we do not need the 
unit64 type, so on a system with no native 8 byte integers that proc
should work fine.

Remember that whenever we use generic procs for the first time with a
new argument type, then a new instance of the proc customized for
that data type is instantiated. That is, when we call strToInt() at
least two times with an int32 and an int8 data type like intToStr
(myInt32) and intToStr(myInt8), then we get already two different
instances. So use of generic procs can increase the code size of our
final executable. To avoid that, we may use intToStr(myInt8.int32)
instead, which would just call the instance for the int32 argument
again.

All the previous examples have used log10() to determine the number
of digits for the passed argument value. On microcontrollers log10()
may be not available at all, or may be very slow. So let us
investigate at the end of this section how we can replace it. The
basic idea is, that we repeatedly divide the argument by ten, until
we get the result zero, counting the number of needed divisions. An
equally approach is to start with a variable with start value one and
multiply with ten, until the result is larger than our function
argument. As division is generally slower than multiplication, and on
microcontrollers a native div operation may be not available at all,
we will try to use multiply operations. So we may start with a proc
like

proc digits0(i: int): int =
  assert i >= 0
  result = 1
  var d = 10
  while d <= i:
    d *= 10
    inc(result)

Can you see the problem? What will happen when we pass int.high as
argument?

So a working proc is this:

proc digits(a: SomeInteger): int =
  assert a >= 0
  var i: uint64 = a.uint64
  result = 1
  when sizeof(a) == 8:
    const c = 10
    if i >= c:
      i = i div c
      result = 2
  var d: typeof(i) = 10
  while d <= i:
    d *= 10
    inc(result)

We do the math with an uint64 type in the proc. For the case that the
argument is an 8 byte type, we may get an overflow in the while loop,
which we prevent by doing one division before the loop already.
Actually for improved performance instead of a division by ten we may
do a division by a larger power of ten, and fix the start value for 
result accordingly. One disadvantage of that generic proc is again,
that an uint64 type is used for the math, which is fine on a desktop
PC, but may work bad on restricted hardware. So this variant seems to
be a better solution:

proc digits(a: SomeInteger): int8 =
  assert a >= 0
  var i = a
  result = 1
  if i >= 10:
    i = i div 10
    result = 2
  var d: typeof(i) = 10
  while d <= i:
    d *= 10
    inc(result)

echo digits(0)
echo digits(9)
echo digits(10)
echo digits(99)
echo digits(int.high)

echo digits(0.int8)
echo digits(int8.high)

echo digits(0.uint8)
echo digits(uint8.high)

echo digits(0.uint)
echo digits(uint.high)

Here we do a single div operation if the argument is larger than 9,
but do all the math with the same type as the argument type. The div
operation may be still slow on a microcontroller, but our intToStr()
proc has also used div operations. Indeed doing intToStr()
conversions on a tiny 8 bit microcontroller is not really a good
idea.

For determining the number of digits of integer numbers, you will
find many more solution in the internet. Sometimes this is called 
log10() for integer numbers. Some functions try to use the logarithm
with base 2, which is related to finding the highest set bit of a
number, some other functions use tabular data or a sequence of if or 
case statements. As the performance of that functions depends on the
actual hardware, there exists not really the best solution for all
cases.

For the intToStr() function you should also find very good solutions
in Nim's standard library. Note that it was not our goal in this
section to present a really good solution, the idea was more to show
you how such a tasks can be solved in principle, and how we can
improve or modify solutions, and how we can use Nim's generics to get
one function for multiple data types. Note that the presented procs
are only minimal tested, and are tested only on a 64 bit desktop OS.
So they may not work on systems where Nim's int type is 32 bit, or
for microcontrollers and embedded systems. But you have learned
enough now so you could fix it for that cases.

As possible exercises for the reader we may suggest to create a
similar proc called strToInt() that converts a numeric string to an
integer number, or to convert between strings and float numbers. The
first one is easy, you would build the int value by continuously
multiplying the digit value with its correct power of ten matching
its position in the string. The float conversion is more difficult,
in one weekend you may get some working code, but perfect solutions
like the ryu or dragonbox algorithm are very complicated.

No Game Programming?

No, not yet. We know that for many people game programming is the
initial motivation to start with computer programming at all, and so
a larger section about this topic would make indeed some sense. But
there are some reasons why we do not have this section currently.
Most important reason is, that we try to present in this book that
stuff, that is very fundamental and that is not presented at other
freely accessible places in a beginner friendly fashion. And there is
a lot of this which seems to be more important than games. The other
reason is, that for a section about games we would have to make a lot
of decisions in advance: 2D or 3D game, action game or strategy game.
Using a game engine, or only a simple libraray like cairo, sdl2,
raylib or gogot? When using a game engine, then the decision which
one we should use, and which Nim bindings set is not a simple
decision, and we should try to ensure that the bindings are actively
developed and so should work in a few years with Nim 2.0 still. And
finally there are already some nice tutorials for game programming
available, see for example

  * https://hookrace.net/blog/
    writing-a-2d-platform-game-in-nim-with-sdl2/

  * https://github.com/Ethosa/nodesnim

  * https://github.com/paranim/paranim

  * https://github.com/jiro4989/nimtetris

  * https://github.com/jiro4989/nimothello

  * https://forum.nim-lang.org/t/8080

  * https://github.com/dsrw/enu

  * https://github.com/def-/nimes

  * https://vladar4.github.io/nimgame2/

  * https://github.com/greenfork/nimraylib_now

  * https://github.com/pragmagic/godot-nim

  * https://github.com/nimgl/nimgl

  * https://github.com/Vladar4/sdl2_nim

  * https://github.com/ftsf/nico

  * https://github.com/StefanSalewski/nim-chess4

  * https://github.com/planetis-m/goodluck/

  * https://forum.nim-lang.org/t/8619

Actually game programming is not that difficult, when we have a nice
library with a good tutorial available. Game programming can be much
fun, which is great, but actually we do not learn that much when we
move some sprites over the screen. On the other hand, advanced game
programming, by using a big library like Godot, doing all with basic
libs like SDL2 or Raylib, or developing your own game engine based on
OpenGL or Vulkan, is a very demanding task.

So maybe we will add a section about game programming at the end,
when the rest of the book is done, or maybe when the next edition of
the book is published.

Part V: External Packages

In this part of the book we will present you some external packages,
which can easily be installed with Nim's package manager(s).

For packages registered in the Nimble data base, executing the nimble
install command

nimble install packageName

is sufficient, and you can also install unregistered packages, which
may be hosted at github.com or another platform with a command like

nimble install https://github.com/user/packageName

Note that we call nimble commands like install generally as ordinary
user, not as admin or root with administrator privileges. We told you
already in the introduction to this book, that we do not intend to
discuss the detailed use of nimble in this book, at least not for the
first edition. The Nimble package manager is described in detail in
https://github.com/nim-lang/nimble and also in the Manning book.
There you can also learn how you can create Nimble packages yourself,
and how you can register your own packages in Nimble's database so
that other people can find them more easy. While Nimble is Nim's
default package manager which is currently used by the majority of
the user base, there exist also the alternative implementation https:
//github.com/disruptek/nimph, and some lesser known ones like Nimp, 
Slim or Nifty.

We have already a few thousand external packages for Nim -- you may
use commands like nimble list or nimble search to list all registered
packages, or to search in the database for entries, or you can use
https://nimble.directory/ or the Github online search to find more
packages. You can also consult the list of curated Nim packages at
https://github.com/xflywind/awesome-nim.

While the use of external packages is really easy, there are some
important points to consider: External packages are not audited by
the Nim core team, so the quality of external packages can vary, and
in principle external packages can even contain malicious code which
may damage your computer when you install and use that package. Well,
as we use the nim and nimble commands as plain user without
administrator privileges, there is no real danger that the computer
OS can be damaged -- only our own user data may get corrupted or
damaged. But as we backup all of our important data regularly, there
is not that much danger, a SSD hardware crash seems to be more
likely. A more serious issue with external packages arise from the
fact that the packages may get outdated and abandoned and may stop
working with recent versions of the Nim compiler, or even may get
totally removed from the internet without prior announcements. So
when you should create a larger software project that depends on
external packages, then you should save a local copy of that package,
or you may even consider to make a private fork of that Github
package.

Some programming languages like Python are shipped already with a
very large collection of libraries, so that external packages are not
that often required at all. Other languages like C++ come basically
without any packages or a language specific package manager, so we
would use the package manager provided by the operating system to
install important C++ packages like Boost or CGAL, or install needed
libraries manually. Nim is between these two extremes -- it provides
already a large collection of modules with its standard library, but
has also a lot of external packages. Both, internal core modules, and
external packages have its merits. We mentioned already some
disadvantages of external packages, but actually they have also
benefits: They may be developed, updated and improved very fast, as
they are not strongly coupled to compiler release updates, and one
external package can easily be replaced by another similar one. A
large set of internal packages can on the other hand be a large
maintainment burden for the language core team -- the packages have to
be tested and maybe fixed for each new compiler version, and
replacing or removing legacy internal core packages can lead to a lot
of problems.

In this part of the book we will present a very small set of external
packages only. This is mostly done to tell you about the existence
and usefulness, and for some package because the currently available
documentation is not really beginner friendly.

We will start with a powerful package for the use of the Parsing
Expression Grammar (PEG), which is some form of an alternative to the
use of regular expressions for parsing tasks.

Parsing Expression Grammars

Parsing whole text files or single strings is a common programming
task, e.g. to process textual user input or to extract data from HTML
or CSV files. Traditional this is often done by the use of regular
expresions -- in part III of the book we show how it can be done by
use of the regex module.

PEGs, or Parsing Expression Grammars, are another formalism for
recognizing patterns in text by use of a set of rules. A PEG can be
used as an alternative to regular expressions for parsing, pattern
matching and text processing. The Parsing Expression Grammar was
introduced by Bryan Ford in 2004 and is closely related to the family
of top-down parsing languages introduced in the early 1970s. PEGs are
a derivative of the Extended Backus-Naur Form (EBNF) with a different
interpretation, designed to represent a recursive descent parser.

PEGs are not unlike regular expressions, but offer more power and
flexibility, and have less ambiguities. For example, a regular
expression inherently cannot find an arbitrary number of matched
pairs of parentheses, because it is not recursive, but a PEG can.

As PEGs can be constructed in a hierarchical way from individual
rules, it can be easier to create or understand them compared to
regular expressions.

While the use of regular expressions is very similar in different
programming languages or external tools like sed and grep, the API
for PEG libraries can be very different, and even the actual syntax
for building parsing rules can differ.

Nim's standard library includes already a simple pegs module, but we
will use the more advanced external NPeg package of Ico Doornekamp
instead. NPeg is a pure Nim library, that provides macros to compile
PEGs to Nim procedures which can parse strings and collect selected
parts of the input.

In this section we will try to explain the basic concepts of the PEG
use and give some examples. For a more formal and complete
description, you should refer to the linked Wikipedia article and
consult the API documentation of the npeg module.

Formally, a parsing expression grammar consists of a starting
expression, a set of parsing rules, and finite sets of terminal and
nonterminal symbols.^[46]

Each parsing rule has the form A - e, where A is a nonterminal symbol
and e is a parsing expression. An (atomic) parsing expression
consists of terminal or nonterminal symbols or an empty string. New
parsing expression can be constructed from existing ones by
concatenation (sequence), an ordered choice, by repetitions (zero-or
more, one-or-more, optional) of an existing expression, and by use of
the and and not predicate. The and-predicate expression &e invokes
the sub-expression e, and then succeeds if e succeeds and fails if e
fails, but in either case never consumes any input. The not-predicate
expression !e succeeds if e fails and fails if e succeeds, again
consuming no input in either case. Because these two predicates can
use an arbitrarily complex sub-expression to "look ahead" into the
input string without actually consuming it, they provide a powerful
syntactic look-ahead and disambiguation facility, in particular when
reordering the alternatives cannot specify the exact parse tree
desired.

NPeg is a pure Nim pattern matching library. It provides macros to
compile patterns and grammars (PEGs) to Nim procedures which will
parse a string and collect selected parts of the input. In this way 
npeg is an alternative to the use of the regex module, but npeg does
not support the optional replacement of matched patterns.

As understanding and using the PEG is really not that easy, and as
most readers may never have heard about PEG at all, we will start
with a few very simple examples. First let us parse just a few
decimal digits:

import npeg

let p = peg("str"):
  str <- +{'0'..'9'}

echo p.match("123").ok

The npeg module defines a few macros for processing PEG patterns. One
of them is the peg() macro, to which we pass as argument a starting
expression in form of a string, and which creates and return a Parser
object. In the body of the peg() macro we have to define all the
grammar rules that our PEG is built of. For our example we only need
one simple rule that is a repetition of the decimal digits zero to
nine.

The npeg module uses as terminal symbols single characters enclosed
in single quotes or strings enclosed in double quotes.

In the original PEG syntax a pair of square brackets is used to
specify character ranges like ['a'..'z'] for the lower case letters
of the alphabet, but the npeg module uses curly braces instead.

So in the npeg syntax {'0'..'9'} stands for a single decimal digit,
and the leading * indicates one or more repetitions. In the original
PEG syntax we would use square brackets instead and put the * after
the closing square bracket, that is ['0'..'9']*. We can also list the
characters of a character class separated by commas, e.g. {'a', 'z'}
for 'a' or 'z'. The symbol - assigns the parsing rule to the
nonterminal symbol str, which is already identical with the starting
expression.

The peg() macro returns a Parser object, which we can pass together
with a string that should be parsed to the match() function. The
function match() returns an instance of a MatchObject -- we use the ok
field of this object to check if the match was successful.

When we intend to use the npeg module, we have to know that this
module uses a syntax which is not fully identical to the original PEG
definition, which is used by the pegs module of Nim's standard
library: Originally character classes were created by enclosing
individual characters or character ranges in square brackets, similar
as done for regular expressions. But the npeg module uses a pair of
curly braces instead. In PEG repetitions are specified by *, + and ?
for zero or more, one or more, or one or zero, as in regular
expressions. In the original PEG design these characters were put
after an expression, while for the npeg syntax we have to put them in
front of an expression. In the original PEG syntax sequences of
expressions are just separated by spaces, while in npeg syntax a * is
used, and for the ordered choice npeg syntax uses the | instead of
the original slash (/) symbol. Like the original PEG syntax npeg uses
the symbols & and ! for the non capturing and and not predicates.
Additional npeg provides the symbol 1 to match all, 0 to match
nothing and an infix - operator -- P1 - P2 matches P1 if P2 does not
match. So an expression, which for example matches all characters but
a space can be easily written as 1 - ' '.

As next example we will create a PEG pattern that can match a simple
mathematical term built from decimal digits and the two operators +
and - for addition and subtraction:

import npeg

let p2 = peg("term"):
  term <- dig * *(op * dig)
  dig <- +{'0'..'9'}
  op <- {'+', '-'}

echo p2.match("1+23").ok

We said that the symbol * is used to indicate zero or more
repetitions of an expression. But for the npeg module this * is used
at the same time to construct sequences of expressions, that is to
concatenate expressions. In the code above we pass the string "term"
as starting expression to the peg() macro. In the macro body we
define three rules, which each assign an expression to the
nonterminal symbols term, dig and op. In the expression dig * *(op *
dig) the second * in front of the opening brace indicates an
arbitrary number of repetitions of the expression enclosed by the
round brackets, while the first and third * indicate the sequence or
concatenation operation. The following two rules just define a
sequence of one or more decimal digits, and the operator for addition
or subtraction.

Capturing data

The npeg module offers plain string captures and more flexible code
block captures.

String Captures

Let us assume that we want to split a line of text into words:

import npeg

let p = peg("line"):
  line <- +(space * >word)
  word <- +{'a'..'z'}
  space <- *' '

let m = p.match(" one   two three    ")
if m.ok:
  echo m.matchLen
  echo m.captures

16
@["one", "two", "three"]

The MatchResult returned by the peg() macro has the exported fields 
matchLen and captures, which we can read out in case of a successful
match. MatchLen tells us how many characters of the string has been
captured, and captures is a seq[string] containing the captured
strings.^[47]

Code Block Captures

Code block captures offer the most flexibility for accessing matched
data in NPeg. This allows you to define a grammar with embedded Nim
code for handling the data during parsing.

When a grammar rule ends with a colon :, the next indented block in
the grammar is interpreted as Nim code, which gets executed when the
rule has been matched. Any string captures that were made inside the
rule are available to the Nim code in the injected variable capture[]
of type seq[Capture]. Capture is an object with field s containing
the captured string and field si containing the index position of the
capture inside the original string.

The total sub-string matched by the code block rule is available as 
capture[0], and the individual captured strings are available with
indices > 0. In the indented code block, we can also use $n instead
of capture[n].s and @n instead of capture[n].si.

We could use the seq of captures to print the captured strings or to
copy it into some global variable. To avoid the need of global
variables, we can pass to the peg() macro a second argument, which is
a name and a data type separated by a colon, like peg(name,
identifier: Type). The second parameter is then available as an
ordinary variable in the code block.

For our next example we will assume that we have written a plain CAD
tool, that allows the user to enter textual commands like moveTo(x,
y).

import npeg, tables

type T = Table[string, string]

let p = peg("command", t: T):
  command <- >com * '(' * >pos * ',' * >pos * ')':
    # echo $1, $2, $3
    t["action"] = $1
    t["x"] = $2
    t["y"] = $3
  com <- "moveTo" | "lineTo"
  pos <- +{'0'..'9'}


var input: T
if p.match("moveTo(12,20)", input).ok:
  echo input["action"], ": ", input["x"], ", ", input["y"]

To keep the example simple and short, we assume that we have to
process only two different command, moveTo() and lineTo(), each
accepting x, y coordinates of integer form. We pass to the peg()
macro a second argument, which is the name and the data type of a 
Table instance. We have chosen for the key and value type of that
table the string data type, as we want to store the command name as
well as the x/y coordinates, so an integer value type would not work.
The macro body defines three rules -- command, com and pos. For the 
command rule we use an expression which starts with the command name,
followed in round brackets, the x/y coordinate pair. In front of the
nonterminal symbols com and pos we put the > operator to capture
these values. We put a colon after the command rule and can access
the captured values in the indented block, by use of the $N symbol.
For the com rule we specify the literal terminal symbols "moveTo" or 
"lineTo" as ordered choice with the | operator. Finally, the
expression for the pos rule is just a sequence of one or more decimal
digits.

Simple patterns

For simple patterns it may be not necessary to define multiple
parsing rules. In that case we can use the patt() macro instead of 
peg() and pass just a single one line pattern as argument.

For example, the pattern below splits a string by white space:

let parser = patt *(*' ' * > +(1-' '))
echo parser.match("   one two three ").captures

We took this example from the npeg API documentation verbatim. Here
the patt() macro uses Nim's command invocation syntax, so there is no
outer bracket after the macro name. The innermost bracket uses the
notation (1-' ') to match everything but a space, and the content of
the outer bracket starts with an arbitrary number of uncaptured
spaces.

"Look ahead" operators

The PEG syntax defines also the two noncapturing and and not
syntactic predicates, which uses the symbols & and ! and provide a
powerful syntactic lookahead and disambiguation facility. A common
use of the ! predicate is to terminate a parsing expression with !1.
Here the 1 matches everything, and !1 would only match when there is
nothing left to match, that is the string end is reached.

We will end our introduction to the parsing expression grammar and
the use of the npeg module here. To learn all the details about PEG
like restrictions, performance and memory consumption you should
consult the Wikipedia article or other dedicated literature. And for
advanced uses of the npeg module including the use of back
references, and all the available syntax elements, you have to study
its API documentation carefully.

References:

  * https://github.com/zevv/npeg

  * https://en.wikipedia.org/wiki/Parsing_expression_grammar

Part VI: Advanced Nim

In this part of the book we will try to explain the more difficult
parts of the Nim programming language: Macros and meta-programming,
asynchronous code, threading and parallel processing, and finally the
use of Nim's concepts. We will start with macros and
meta-programming, as that seems to be a really stable part of Nim's
advanced features. Nim's concepts just got a redesign, and for the
use of asynchronous code, threading and parallel processing there
exists currently various implementations, and all that may change
again when the Nim core devs should decide to actually use the CPS
(Continuation-Passing Style) based programming style for the
implementation of this.

Macros and Meta-Programming

Introduction

In computer science a macro (short for "macro instruction") is a rule
or pattern that specifies how a certain input should be mapped to a
replacement output. Meta-programming is a programming technique in
which computer programs have the ability to treat other programs as
their data. It means that a program can read, generate, analyze or
transform other programs, and even modify itself while running.

Legacy programming languages like C or assembly languages support
already some form of macros, which generally work directly on the
textual representation of the source code.

A common use of textual macros in assembly languages was to group
sequences of instructions, like reading data from a file or from the
keyboard, to make that operations easily accessible. The C
programming language uses the #define pre-processor directive to
introduce textual macros. Macros in C are generally single line text
substitutions which are processed by a pre-processor program before
the actual compiling process. Some examples of common C macros are

#define PI 3.1415
#define sqr(x) (x)*(x)

The basic C macro syntax is that the first continues character
sequence after the #define directive is replaced by the C
pre-processor with the rest of that line. The #define directive has
some basic parameter support which was used for the sqr() macro
above. C macros have the purpose to support named constants and to
support simple parameterized expressions like the sqr() from above,
avoiding the need to create actual functions. The C pre-processor
would substitute each occurrence of the string PI in the C source
file with the float literal 3.1415 and the term sqr(1+2) with (1+2)*
(1+2).

A Nim macro is a function that is executed at compile-time and
transforms a Nim syntax tree into a different tree. This can be used
to add custom language features and implement domain-specific
languages (DSL). While macros enable advanced compile-time code
transformations, they cannot change Nim's syntax.

The macro keyword is used similar to proc, func and template to
define a parameterized code block which is executed at compile time
and consists of ordinary Nim code and meta-programming instructions.
The meta-programming instructions are imported from the macros module
and are used to construct an Abstract Syntax Tree (AST) of the Nim
language. This AST is created by the macro body at compile time and
is returned by the macro as untyped data type. The parameter list of
macros accept ordinary (static) Nim data types and additional the
data types typed and untyped, which we already used for templates. We
will explain the differences of the various possible data types for
macro parameters later in more detail, after we have given a first
simple macro example. Note that Nim macros are hygienic by default,
that is symbols defined inside the macro body are local and do not
pollute the name space of the environment. As macros are executed at
compile time, the use of macros may increase the compile time, but
their use does not impact the performance of the final
executable -- in some cases the use of clever macros may even improve
the performance of our final program.

Macros are by far the most difficult part of the Nim programming
language. While in languages like Lisp macros integrate very well
into the language, for Nim the meta-programming with macros is very
different to the use of the language itself. Nim macros are very
powerful -- the current Nim async implementation is based on Nim
macros, and some advanced libraries for threading, parallel
processing or data serialization using JSON or YAML file formats make
heavily use of Nim macros. And many modules of the Nim standard
library provide some macros which extends the power of the Nim core
language. The famous with macro of the equality named module is only
one example for the usefulness of Nim's macros. And same small but
important parts of the high level GTK bindings are created with
macros, e.g. the code to connect GTK callback functions to GTK
signals. But this means not, that each Nim user really has to use
macros. For a few use cases we really need macros, for other use
cases macros may make our code shorter, maybe even cleaner. But at
the same time the use of macros can make it for other people harder
to understand the code, at least when we use exotic or our own
complicated macros. And learning advanced Nim macro programming is
not that easy. Nim macros have some similarity to the programming
language C++: When we follow the explanations in a C++ text book then
the C++ language seems to be not extremely difficult and even seems
to follow a more or less logical design. But then, when we later try
to write some actual code in C++ we notice that actually using the
languages is hard as long as we have not a lot of practice. For Nim
macros it is similar -- when we follow a talk of an experienced Nim
programmer about macro programming or when we read the code of an
existing macro written by the Nim core devs, then all seems to be not
that hard. But when we try to create macros our own for the first
time, it can be frustrating. Strange error messages, or even worse no
idea at all how we can solve a concrete task. So maybe the best start
with macros is to read the code of existing macros, to study the 
macros module to see what is available, and maybe to follow some of
the various tutorials listed at the end of this section. And finally
you would have to ask for help in the Nim forum, on IRC or the other
Nim help channels.

To verify that macros are really executed at compile time, we will
start with a tiny macro that contains only an echo statement in its
body:

import macros

macro m1(s: string): untyped =
  echo s

proc main =
  echo "calling macro m1"
  m1("Macro argument")

main()

When we compile above code the compiler prints the message "Macro
argument", as it processes the macro body. When we run the program,
we get only the output "calling macro m1" from the main() proc, as
the macro m1() does return an empty AST only. The carefully reader
may wonder, why the echo() statement in the macro body above works at
all, as the parameter of macro m1() is specified as ordinary string,
not as static[string]. So the type of s in the macro body should be a
NimNode. Well, maybe an echo() overload exists that can work with
NimNodes, or maybe, as we pass a string constant to macro m1(), in
this concrete case s is indeed an ordinary string in the macro body.
Maybe we should have used s: static[string] as parameter type, which
would give us the exact same results.

We said that macros have to always return an untyped result. This is
true, but as untyped is the only possible result type, that type can
currently be omitted. So you may see in the code of the Nim standard
lib a few macros which seems to return nothing. For our own macros we
really should use always untyped as result. And sometimes you may
even see macros where for parameters no data type is specified at
all. In that case the data type has the default untyped type.

As macros are executed at compile time, we can not really pass
runtime variables to it. When we try, we would expect a compiler
error:

import macros

macro m1(s: string): untyped =
  echo s

proc main =
  var str = "non static string"
  m1(str)

main()

But with the current compiler version 1.5.1 that code compiles and
prints the message "str", which is a bit surprising. To fix this, we
can change the parameter type to static[string], which guarantees
that we can indeed pass only compile time constants. Our last example
would give a compile error in this case, while the one before with
the string constant would work as expected.

Now let us create macros which actually creates an AST which is
returned by the macro and executed when we run our program. For
creating an AST in the macro body we have various options: we can use
the parseStmt() function or the "quote do:" notation to generate the
AST from regular program code in text form, or we can create the
syntax tree directly with expressions provided by the macros module,
e.g. by calls like newTree() or newLit() and such. The later gives us
the best control over the AST generation process, but is not easy for
beginners. The good news is that Nim now provides a set of helper
functions like dumpTree() or dumpAstGen() which shows us the AST
representation of a Nim source code block as well as the commands
which we can use to create that AST. This makes it for beginners much
easier to learn the basic instructions necessary to create valid
syntax trees and to create useful macros.

We will start with the simple parseStmt() function which generates
the syntax tree from the source code text string that we pass it as
argument. This seems to be very restricted, and maybe even useless,
as we can write the source code just as ordinary program text outside
of the macro body. That is true, but we can construct the text string
argument that we pass to the parseStmt() function with regular Nim
code at compile time. That is similar as having one program, which
generates a new source code string, saves that string to disk, and
finally compiles and runs that created program. Let us check with a
fully static string that parseStmt() actually works:

import macros

macro m1(s: static[string]): untyped =
  result = parseStmt(s)

proc main =

  const str = "echo \"We like Nim\""
  m1(str)

main()

When we compile and run above program, we get the output "We like
Nim". The macro m1() is called at compile time with the static
parameter str and returns an AST which represents the passed program
code fragment. That AST is inserted into our program at the location
of the macro call, and when we run our program the compiled AST is
executed and produces the output.

Of course executing a fully static string this way is useless, as we
could have used regular program code instead. Now let us investigate
how we can construct some program code at compile time. Let us assume
that we have an object with multiple fields, and we want to print the
field contents. A sequence of echo() statements would do that for us,
or we may use only one echo() statement, when we separate the field
arguments each by "\n". The with module may further simplify our
task. But as we have to print multiple fields, not an array or a seq,
we can not directly iterate over the values to process them. Let us
see how a simple text string based macro can solve the task:

import macros

type
  O = object
    x, y, z: float

macro m1(objName: static[string]; fields: varargs[untyped]): untyped =
  var s: string
  for x in fields:
    s.add("echo " & objName & "." & x.repr & "\n")
  echo s # verify the constructed string
  result = parseStmt(s)

proc main =
  var o = O(x: 1.0, y: 2.0, z: 3.0)
  m1("o", x, y, z)

main()

In this example we pass the name of our object instance as static 
string to the macro, while we pass the fields not as string, but as
list of untyped values. The passed static string is indeed an
ordinary Nim string inside the macro, we can apply sting operations
on it. But the field names passed as untyped parameters appear as so
called NimNodes inside the macro. We can use the repr() function to
convert the NimNodes to ordinary strings, so that we can use string
operations on them. We iterate with a for loop over all the passed
field names, and generate echo() statements from the object instance
name and the field names, each separated by a newline character. Then
all the statements are collected in a multi-line string s and are
finally converted to the final AST by the parseStmt() function. In
the macro body we use the echo() statement to verify the content of
that string. As the macro is executed during compile time, we get
this output when we compile our program:

echo o.x
echo o.y
echo o.z

And when we run it we get:

1.0
2.0
3.0

Well, not a really great result for this concrete use case: We have
replaced three echo() commands with a five lines macro. But at least
you got a feeling what macros can do for use.

Types of macro parameters

As Nim is a statically typed programming language, all variables and 
proc parameters have a well defined data type. There is some form of
exception from this rule for OR-types, object variants and object
references: OR-types are indeed no real exception, as whenever we use
an OR-type as the type of a proc parameter, multiple instances of the
proc with different parameter types are created when necessary. That
is very similar to generic procs. object variants and object
references built indeed some form of exception, as instances of these
types can have different runtime types that we can query with the 
case or with the of keyword at runtime. Note that object variants and
references (the managed pointers itself, not the actual data
allocated on the heap) occupy always the same amount of RAM,
independent of the actual runtime type. (That is why we can store
object variants with different content or references to objects of
different runtime types using inheritance in arrays and sequences.)

For the C sqr() macro from the beginning of this section, there is no
real restriction for the argument data types. The sqr() C macro would
work for all numeric types that support the multiply operation, from 
char data type over various int types to float, double and long
double. This behaviour is not really surprising, as C macros are only
a text substitution -- by the * multiply operator for our sqr() macro.
Actually the C pre-processor would even accept all data types and
even undefined symbols for its substitution process. But then the C
compiler would complain later.

Nim macros and Nim templates do also some form of code substitution,
so it is not really surprising that they accept not only well defined
data types, but also the relaxed types typed and untyped.

As parameters for Nim's macros we can use ordinary Nim data types
like int or string, compile time constants denoted with the static
keyword like static[int], or the typed and untyped data types. When
we call macros then the data types of the parameters are used in the
same way for overload resolution as it is done for procs and
templates. For example, if a macro defined as foo(arg: int) is called
as foo(x), then x has to be of a type compatible to int.

What may be surprising at first is, that inside the macro body all
parameter types have not the data type of the actual argument that we
have passed to the macro, but the special macro data type NimNode
which is defined in the macros module. The predefined result variable
of the macro has the type NimNode as well. The only exception are 
macro parameters which are explicitly marked with the static keyword
to be compile time constants like static[string], these parameters
are not NimNodes in the macro body, but have their ordinary data
types in the macro body. Variables that we define inside the macro
body have exactly that type that we give to then, e.g. when we define
a variable as s: string then this is an ordinary Nim string variable,
for which we can use the common string operations. But of course we
have always to remember that macros are executed at compile time, and
so the operations on variables defined in the macro body occur at
compile time, which may restrict a few operations. Currently macros
are evaluated at compile time by the Nim compiler in the NimVM
(Vitual Machine) and so share all the limitations of the NimVM: 
Macros have to be implemented in pure Nim code and can currently not
call C functions except those that are built in the compiler.

In the Nim macros tutorial the static, typed and untyped macro
parameters are described in some detail. We will follow that
description, as it is more detailed as the current description in the
Nim compiler manual. As these descriptions are very abstract, we will
give some simple examples later.

Static Macro Parameters

static arguments are a way to pass compile time constants not as a 
NimNode but as an ordinary value to a macro. These values can then be
used in the macro body like ordinary Nim variables. For example, when
we have a macro defined as m1(num: static[int]), then we can pass it
constants values compatible with the int data type, and in the macro
body we can use that parameter as ordinary integer variable.

Untyped Macro Parameters

untyped macro arguments are passed to the macro before they are
semantically checked. This means that the syntax tree that is passed
down to the macro does not need to make sense for the Nim compiler
yet, the only limitation is that it needs to be parsable. Usually,
the macro does not check the argument either but uses it in the
transformation's result somehow. The result of a macro expansion is
always checked by the compiler, so apart from weird error messages,
nothing bad can happen. The downside for an untyped macro argument is
that these do not play well with Nim's overloading resolution. The
upside for untyped arguments is that the syntax tree is quite
predictable and less complex compared to its typed counterpart.^[48]

Typed Macro Parameters

For typed arguments, the semantic checker runs on the argument and
does transformations on it, before it is passed to the macro. Here
identifier nodes are resolved as symbols, implicit type conversions
are visible in the tree as calls, templates are expanded, and
probably most importantly, nodes have type information. Typed
arguments can have the type typed in the arguments list. But all
other types, such as int, float or MyObjectType are typed arguments
as well, and they are passed to the macro as a syntax tree.^[49]

Code Blocks as Arguments

In Nim it is possible to pass the last argument of a proc, template
or macro call as an indented code block following a colon, instead of
an ordinary argument enclosed in the parentheses following the
function name. For example instead of echo("1 + 2 = ", 1 + 2) we can
also write

echo("1 + 2 = "):
  1 + 2

For procs this notation makes not much sense, but for macros this
notation can be useful, as syntax trees of arbitrary complexity can
be passed as arguments.

Now let us investigate in some more detail which data types a macro
accepts. This way we hopefully get more comfortable with all these
strange macro stuff. For our test we create a few tiny macros with
only one parameter which does noting more than printing a short
message when we compile our program:

import macros

macro m1(x: static[int]): untyped =
  echo "executing macro body"

m1(3)

This code should compile fine and print the message "executing macro
body" during the compile process, and indeed it does. The next
example is not that easy:

import macros

macro m1(x: int): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

var y: int
y = 7
m1(y)

This compiles, but as the assignment [.code]y = 7 is executed at
program runtime, while the macro body is already executed at compile
time, we should not expect that the echo() statement in the macro
body prints the value 7. Instead we get just y for both echo() calls.
Now let us investigate what happens when we use typed instead of int
for the macro parameter:

import macros

macro m1(x: typed): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

var y: int
y = 7
m1(y)

We get the same result again, both echo() statements prints y. The
advantage of the use of typed here is, that we can change the data
type of y from int to float and our program still compiles. So the 
typed parameter type just enforces that the parameter has a well
defined type, but it does not restrict the actual data type to a
special value. The previous macro with int parameter type would
obviously not accept a float value.

Now let us see what happens when we pass an undefined symbol to this 
macro with typed parameter:

import macros

macro m1(x: typed): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

m1(y)

This will not compile, as the macro expects a parameter with a well
defined type. But we can make it compile by replacing typed with 
untyped:

import macros

macro m1(x: untyped): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

m1(y)

So untyped macro parameters are the most flexible ones, and actually
they are the most used. But in some situations it is necessary to use
typed parameters, e.g. when we need to know the parameter type in the
macro body.

Quote and the quote do: construct

In the section before we learned about the parseStmt() function which
is used in a macro body to compile Nim code represented as a
multi-line string to an abstract syntax tree representation. Macros
uses as return type the "untyped" data type, which is compatible with
the NimNode type returned by the parseStmt() function.

The quote() function and the quote do: construct has some similarity
with the parseStmt() function: It accepts an expression or a block of
Nim code as argument and compiles that Nim code to an abstract syntax
tree representation. The advantage of quote() is that the passed Nim
code can contain NimNode expressions from the surrounding scope. The 
NimNode expressions have to be quoted using backticks.

As a first very simple example for the use of the quote do: construct
we will present a way to print some debugging output.

Assume we have a larger Nim program which works not in the way that
we expected, so we would add some echo() statements like

var currentSpeed: float = calcSpeed(t)
echo "currentSpeed: ", currentSpeed

Instead of the echo() statement we would like to just write show
(currentSpeed) to get exactly the same output. For that we need
access not only to the actual value of a variable, but also to its
name. Nim macros can give us this information, and by using the quote
do: construct it is very easy to create our desired showMe() macro:

import macros

macro show(x: untyped): untyped =
  let n = x.toStrLit
  result = quote do:
    echo `n`,": ", `x`

import math
var a = 7.0
var b = 9.0
show(a * sqrt(b))

When we compile and run that code we get:

a * sqrt(b): 21.0

In the macro body we use the proc toStrLit() from the macros module
which is described with this comment: "Converts the AST n to the
concrete Nim code and wraps that in a string literal node" So our
local variable n in the macro body is a NimNode that now contains the
string representation of the macro argument x. We use the NimNode n
enclosed with backtics in the quote do: construct. It seems that
writing this macro was indeed not that difficult, but actually it was
only that easy because we have basically copied the dump() macro from
the sugar module of Nim's standard library.

Let us investigate our show() macro in some more detail to learn more
about the inner working of Nim macros. First recall that macros
always have a return value of data type untyped, which is actually a 
NimNode. The quote do: construct gives us a result which we can use
as return value of our macro. Sometimes we may see macros with no
result type at all, which is currently identical to the untyped
result type. As the macro body is executed at compile time, the quote
do: construct is executed at compile time as well, that is that the
code block which we pass to the quote do: construct is processed at
compile time and the quoted NimNodes in the block are interpolated at
compile time. For our program from above the actual echo() statement
in the block is then finally executed at program runtime. To prove
how this final echo() statement looks we may add as last line of our 
macro the statement "echo result.repr" and we would then get the 
string "echo "a * sqrt(b)", ": ", a * sqrt(b)" when we compile our
program again.

Building the AST manually

In the two sections before we used the functions parseStmt() and 
quote() to build the AST from a textual representation of Nim code.
That can be convenient, but is not very flexible. In this section we
will learn how we can build a valid AST from scratch by calling
functions of the macros module. That is not that easy, but this way
we have the full power of the Nim meta-programming available.

Luckily the macros module provides some macros like dumpTree() and 
dumpAstGen() which can help us get started. We will create again a
macro similar to the show() macro that we created before with the 
quote do: construct, but now with elementary instructions from the 
macros module. This may look a bit boring, but this plain example is
already complicated enough for the beginning, and it shows use the
basics to construct much more powerful macros later.

The core code of our debug() macro would look in textual
representation like

var a, b:int
echo "a + b", ": ", a + b

That is for debugging we would like to print an expression first in
its string representation, and divided by a colon the evaluated
expression. The dumpTree() macro can show us how the Nim syntax tree
for such a print debug statement should look:

import macros

var a, b: int

dumptree:
  echo "a + b", ": ", a + b

When we compile this code we get as output:

 StmtList
  Command
    Ident "echo"
    StrLit "a + b"
    StrLit ": "
    Infix
      Ident "+"
      Ident "a"
      Ident "b"

So the Nim syntax tree for the echo() statement from above is a
statement list consisting of an echo() command with two string
literal arguments and a last argument which is built with the infix +
operator and the two arguments a and b. So we can see how the AST
that we would have to construct would have to look, but we still have
no idea how we could construct such an AST in detail. Well, the 
macros module would contain the functions what we need for that, but
it is not easy to find the right functions there. The dumpAstGen() 
macro can list us exactly the needed functions:

import macros

var a, b: int

dumpAstGen:
  echo "a + b", ": ", a + b

Compiling that code gives us:

 nnkStmtList.newTree(
  nnkCommand.newTree(
    newIdentNode("echo"),
    newLit("a + b"),
    newLit(": "),
    nnkInfix.newTree(
      newIdentNode("+"),
      newIdentNode("a"),
      newIdentNode("b")
    )
  )
)

This is a nested construct. The most outer instruction constructs a
new tree of Nim Nodes with the node type statement list. The next
construct creates a tree with node kind command, which again contains
the ident node with name echo, which again contains two literals and
the infix + operator.

Indeed we can use the output of the dumpAstGen() macro directly to
create a working Nim program:

import macros

var a, b: int

#dumpAstGen:
#  echo "a + b", ": ", a + b

macro m(): untyped =
  nnkStmtList.newTree(
    nnkCommand.newTree(
      newIdentNode("echo"),
      newLit("a + b"),
      newLit(": "),
      nnkInfix.newTree(
        newIdentNode("+"),
        newIdentNode("a"),
        newIdentNode("b")
      )
    )
  )

m()

When we compile and run that code, we get the output:

a + b: 0

So the AST from above is fully equivalent to the one line echo()
statement. But now we would have to investigate how we can pass an
actual expression to our macro and how we can use that passed
argument in the macro body -- first print its textual form, and then
the evaluated value separated by a colon. And there is one more
problem: That nested macro body from above is not really useful for
our final dump() macro, as we would like to be able to construct the 
NimNode, that is returned by the dump() macro step wise: Add the echo
() command, then the passed expression in string form, and finally
the evaluated expression. So let us first rewrite above macro in a
form where the AST is constructed step by step. That may look
difficult, but when we know that we can call the newTree() function
with only one node kind parameter to create a empty tree of that
kind, and that we can later use the overloaded add() proc to add new
nodes to that tree, then it is easy to guess how we can construct the
macro body:

 import macros

var a, b: int

#dumpAstGen:
#  echo "a + b", ": ", a + b

macro m(): untyped =
  nnkStmtList.newTree(
    nnkCommand.newTree(
      newIdentNode("echo"),
      newLit("a + b"),
      newLit(": "),
      nnkInfix.newTree(
        newIdentNode("+"),
        newIdentNode("a"),
        newIdentNode("b")
      )
    )
  )

macro m2(): untyped =
  result = nnkStmtList.newTree()
  let c = nnkCommand.newTree()
  let i = nnkInfix.newTree()
  i.add(newIdentNode("+"))
  i.add(newIdentNode("a"))
  i.add(newIdentNode("b"))
  c.add(newIdentNode("echo"))
  c.add(newLit("a + b"))
  c.add(newLit(": "))
  c.add(i)
  result.add(c)

m2()

First we create the tree empty three structures of node kinds
statement list, command and infix operator. Then we use the
overloaded add() proc to populate the threes, using procs like 
newIdentNode() or newLit() to create the nodes of matching types as
before. When we run our program with the modified macro version m2()
we get again the same output:

a + b: 0

The next step to create our actual dump() macro is again easy -- we
pass the expression to dump() as an untyped macro parameter to the 
macro, convert it to a NimNode of string type and use that instead of
the newLit("a + b") from above. In our second macro, where we used
the quote do: construct, we applied already toStrLit() on an untyped 
macro parameter, so we should be able to reuse that to get the string
NimNode. Instead we would have to apply the stringify operator
additional on that value. But a simpler way is to just apply repr()
on the untyped macro argument to get a NimNode of string type. And
finally, to get the value of the evaluated expression in our dump()
macro, we add() the untyped macro parameter directly in the command
three -- that value is evaluated when we run the macro generated code.

import macros

var a, b: int

macro m2(x: untyped): untyped =
  var s = x.toStrLit
  result = nnkStmtList.newTree()
  let c = nnkCommand.newTree()
  c.add(newIdentNode("echo"))
  c.add(newLit(x.repr))
  #c.add(newLit($s))
  c.add(newLit(": "))
  c.add(x)
  result.add(c)

m2(a + b)

Again, we get the desired output:

 a + b: 0

So our dump() macro called still m2() is complete and can be used to
debug arbitrary expression. Note that this macro works for arbitrary
expressions, not only for numerical ones. We may use it like

m2(a + b)
let what = "macros"
m2("Nim " & what & " are not that easy")

and get the output

a + b: 0
"Nim " & what & " are not that easy": Nim macros are not that easy

Now let us extend our debug() macro so that it can accept multiple
arguments. The needed modifications are tiny, we just pass instead of
a single untyped argument an argument of type varargs[untyped] to the
debug macro, and iterate in the macro body with a for loop over the 
varargs argument:

import macros

macro m2(args: varargs[untyped]): untyped =
  result = nnkStmtList.newTree()
  for x in args:
    let c = nnkCommand.newTree()
    c.add(newIdentNode("echo"))
    c.add(newLit(x.repr))
    c.add(newLit(": "))
    c.add(x)
    result.add(c)

var
  a = 2
  b = 3
m2(a + b, a * b)

When we compile and run that code we get:

a + b: 5
a * b: 6

The Assert Macro

As one more simple example we will show how we can create our own 
assert() macro. The assert() has only one argument, which is a
expression with a boolean result. If the expression evaluates to true
at program runtime, then the assert() macro should do nothing. But
when the expression evaluates to false, then this indicates a serious
error and the macro shall print the expression which evaluated to 
false, and then terminate the program execution. This is basically
what the assert() macro in the Nim standard library already does, and
the official Nim macros tutorial contains such an assert() macro as
well.

Arguments for our assert() macro may look like "x == 1 +2",
containing one infix operator and one left-hand and one right-hand
operand. We will show how we can use subscript [] operators on the 
NimNode argument to access each operand.

As a first step, we use the treeRepr() function from the macros
module to show us the Nim tree structure of a boolean expression with
an infix operator:

import std/macros

macro myAssert(arg: untyped): untyped =
  echo arg.treeRepr

let a = 1
let b = 2

myAssert(a != b)

When we compile that program, then the output of the treeRepr()
function shows us, that we have passed as argument an infix operator
with two operands at index position 1 and 2.

Infix
  Ident "!="
  Ident "a"
  Ident "b"

Now let us create an assert() macro which accept such a boolean
expression with an infix operator and two operands:

import std/macros

macro myAssert(arg: untyped): untyped =
  arg.expectKind(nnkInfix) # NimNodeKind enum value
  arg.expectLen(3)
  let op = newLit(" " & arg[0].repr & " ") # operator as string literal NimNode
  let lhs = arg[1] # left hand side as NimNode
  let rhs = arg[2] # right hand side as NimNode
  result = quote do:
    if not `arg`:
      raise newException(AssertionDefect,$`lhs` & `op` & $`rhs`)

let a = 1
let b = 2

myAssert(a != b)
myAssert(a == b)

The first two function calls expectKind() and expectLen() verify that
the macro argument is indeed an infix operator with two operands,
that is the total length of the argument is 3. The symbol nnkInfix is
an enum value of the NimNodeKind data type defined in the macros
module -- that module follows the convention to prepend enum values
with a prefix, which is nnk for NimNodeType in this case. In the 
macro body we use the subscript operator [0] to access the operator,
and then apply repr() on it to get its string representation. Further
we use the subscript operators [1] and [2] to extract the two
operands from the macro argument and store the result each in a 
NimNode lhs and rhs. Finally we create the quote do: construct with
its indented multi-line string argument and the interpolated NimNode
values enclosed in backtics. The block after the quote do: construct
checks if the passed arg macro argument evaluates to false at
runtime, and raises an exception in that case displaying the
reconstructed argument.

We have to admit that this macro is not really useful in real life,
as it is restricted to simple boolean expressions with a single infix
operator. And what it does in its body makes not much sense: The
original macro argument is split in tree parts, the infix operator
and the two operands, which are then just joined again to show the
exception message. But at least we have learned how we can access the
various parts of a macro argument by use of subscript operators, how
we can use the treeRepr() function from the macros module to inspect
a macros argument, and how we can ensure that the macro argument has
the right shape for our actual macro by applying functions like 
expectKind() and expectLen() early in the macro body.

Pragma Macros

All macros and templates can also be used as pragmas. They can be
attached to routines (procs, iterators, etc), type names, or type
expressions. In this section we will show a small example how a proc
pragma can be used to print the proc name whenever a proc annotated
with that pragma is called:

import std/macros

dumpAstGen: # let us see how the NimNode for an echo statement has to look
  proc test(i: int) =
    var thisProcName = "test"
    echo thisProcname
    echo 2 * i

macro pm(arg: untyped): untyped = # a pragma macro
  expectKind(arg, nnkProcDef) # assert that macro is applied on a proc
  let node = nnkCommand.newTree(newIdentNode("echo"), newLit($name(arg)))
  insert(body(arg), 0, node)
  result = arg

proc myProc(i: int) {.pm.} =
  echo 2 * i

proc main =
  myProc(7)

main()

We start with the dumpAstGen() macro applied on a test() proc which
contains an echo() statement. So when we compile that code, we get an
initial idea how a NimNode that shall print the proc name should
look. To use pragma macros, we annotate the proc with the macro name
enclosed in the pragma symbols {..}. The annotated proc is then
passed to the pragma with that name in form of a syntax tree. Our
goal is to to add a NimNode to this tree that prints the proc name of
the passed AST. To do that we have to know two important points: For
the proc that is passed as untyped data type to our macro, we can use
the function body() to get the AST representation of the body of the
passed proc, and we can use name() to get the name of that proc. The
functions body() and name() are provided by the macros module of
Nim's standard library. In our macro pm() we first verify, that the
passed argument is really of node kind ProcDef. Then we create a new 
NimNode, which calls the echo() function with the proc name as
parameter. And we insert that node at position 0 into the body of the
passed proc. Finally we return the modified AST.

When we run our program, we get this output in the terminal window:

$ ./t
myProc
14

Pragma Macro for Iterator

Let us assume we have an object type which has some fields which are
all sequences with the same base type, and we need an iterator to
iterate over all the container elements. Indeed this may happen when
the different seqs contain subclasses of the same parent class as in

type
  Group = ref object of Element
    lines: seq[Line]
    circs: seq[Circ]
    texts: seq[Text]
    rects: seq[Rect]
    pads: seq[Pad]
    holes: seq[Hole]
    paths: seq[Path]
    pins: seq[Pin]
    traces: seq[Trace]

iterator items(g: Group): Element =
  for el in g.lines:
    yield el
  for el in g.rects:
    yield el
  for el in g.circs:
    yield el
  ....

Maybe we do not want to write all the for loops in the iterator body
manually. One solution is to create a pragma macro, which creates the
for loops in the iterator body for us:

import std/macros

type
  O = object
    a, b, c: seq[int]

macro addItFields(o: untyped): untyped =
  const fields = ["a", "b", "c"]
  expectKind(o, nnkIteratorDef)
  # echo o.treeRepr
  # echo o.params.treeRepr
  let objName = o.params[1][0]
  for f in fields:
    let node =
      nnkStmtList.newTree(
        nnkForStmt.newTree(
          newIdentNode("el"),
          nnkDotExpr.newTree(
            #newIdentNode("o"),
            newIdentNode($objName),
            # newIdentNode("b")
            newIdentNode(f)
          ),
          nnkStmtList.newTree(
            nnkYieldStmt.newTree(
              newIdentNode("el")
            )
          )
        )
      )
    insert(body(o), body(o).len, node)
  result = o
  #echo result.repr

iterator items(o: O): int {.addItFields.} =
  discard

#dumpAstGen:
#  iterator xitems(o: O): int =
#    for el in o.a:
#      yield el

var ox: O
ox.a.add(1)
ox.b.add(2)
ox.c = @[5, 7, 11, 13]

for l in ox.items:
  echo l

We start again with a dumpAstGen() call which shows us the shape of
the for loop node. In that node we only have to replace two 
newIdentNode() calls so that the fields names can be provided by
iterating over an array of strings, and the object name is taken from
the iterator parameter. To get the object name, we first use 
o.treeRepr to see the whole parameter structure, and then 
params.treeRepr to get the structure of the parameters passed to our
iterator. Using subscript operators we get the actual object name. We
insert each new node that we create in the for loop with a call of 
insert(body(o), body(o).len, node) as new last node in the body of
the iterator. We can create a more flexible variant of our above 
macro, when we pass the actual field names as additional parameter to
the pragma macro:

import std/macros

type
  O = object
    a, b, c: seq[int]

macro addItFields(fields: openArray[string]; o: untyped): untyped =
  expectKind(o, nnkIteratorDef)
  let objName = o.params[1][0]
  for f in fields:
    let node =
      nnkStmtList.newTree(nnkForStmt.newTree(newIdentNode("el"),
          nnkDotExpr.newTree(newIdentNode($objName),newIdentNode($f)),
          nnkStmtList.newTree(nnkYieldStmt.newTree(newIdentNode("el")))))
    insert(body(o), body(o).len, node)
  result = o

iterator items(o: O): int {.addItFields(["a", "b", "c"]).} =
  discard

var ox: O
ox.a.add(1)
ox.b.add(2)
ox.c = @[5, 7, 11, 13]

for l in ox.items:
  stdout.write l, ' '

When we run this macro or the one before, we get

1 2 5 7 11 13

Macros to generate new operator symbols

Earlier in the book we have already learned how we can define new 
procs and templates which can be used as operators. In this section
we will learn how we can create a macro that does not only create an
operator that can work on existing variables, but that can be used to
create new variables. In Nim we use the var or let keyword to create
new variables. Some other languages allow to create new variables on
the fly by using just "=", ":=" or "!=" to create new variables.

import std/macros

dumpAstGen:
  var xxx: float

macro `!=`(n, t: untyped): untyped =
  let nn = n.repr
  let tt = t.repr
  nnkStmtList.newTree(
    nnkVarSection.newTree(
      nnkIdentDefs.newTree(
        # newIdentNode("xxx"),
        newIdentNode(nn),
        # newIdentNode("float"),
        newIdentNode(tt),
        newEmptyNode()
      )
    )
  )

proc main =
  myVar != int
  myVar = 13
  echo myVar, " ", typeof(myVar)

main()

Again dumpAstGen() shows us the structure of the needed AST. We use 
repr() to get the string representation of the two macro arguments
and replace in the dumpAstGen() output the arguments of the 
newIdentNode() calls with that values. When we compile and run above
program we get

$ ./t
13 int

For the case that we should really intend to use such a macro in our
own code, we should of course add some code to the macro to check
that the passed arguments have the correct content.

References:

  * https://nim-lang.org/docs/manual.html#macros

  * https://nim-lang.org/docs/tut3.html

  * https://nim-by-example.github.io/macros/

  * https://hookrace.net/blog/introduction-to-metaprogramming-in-nim/

  * https://flenniken.net/blog/nim-macros/

  * https://dev.to/beef331/demystification-of-macros-in-nim-13n8

Process execution

In this section we will discuss how we can use multiple threads or
Nim's async/await framework to avoid blocking IO (input/output)
operations and to enable parallel code execution on multiple physical
CPU cores. The various forms of not strictly linear and sequential
program execution are also called multitasking or multi-threading.
Threading is generally the splitting of one path of actions into
various sub-parts, which can be processed in parallel or concurrent.
On a CPU with multiple physical cores, threads can be distributed
between them, while on a CPU with only one core, all threads have to
run obviously alternating on that single core, which is called
concurrency. Parallel processing requires always dedicated physical
hardware, that is multiple CPUs, or a multi-core CPU consisting of
two or more independent units called cores.

As the CPUs of recent desktop computers often have already a few
dozen of cores, and GPUs may have thousands of them, it has become
more and more important to distribute computing tasks between all
these cores to gain optimal performance. Dedicated programming
languages like Chapel or Pony have been developed for this task, and
most modern programming languages supports it. For older languages
like C extension like OpenMP for threading support have been
developed.

The various forms of asynchronous operation were introduced due to
the fact that some input and output operations and network requests
can be very slow compared to the data processing rate of the CPU. It
would be very wasteful when the CPU has to be idle while a slow
network data transfer or a floppy disk operation is performed.
Actually asynchronous operation was already done long before the
first multi-core CPUs were available.

While Nim has already good support for threading and asynchronous-
and parallel processing, all this is still some work in progress, so
things may further improve in future.

When we launch a computer program on our desktop PC, then the
operating system creates an instance of a new process, sometimes also
called a task, to execute the application. Each process is strongly
separated from other processes that may also running on the computer,
each process has its own memory regions (RAM) that it may use, and
when one process should crash for some reason other processes are not
concerned. Processes can have various states defined by the OS, this
includes some form of running, idle, ready, waiting or halted. A
process executes one or multiple threads, which can run concurrent or
parallel on multiple physical CPU cores. All the threads of one
single process can use common resources and access common variables,
which enables data exchange between threads, with some restrictions.
Data exchange between different, separated processes is not that
easy, but it is also possible by use of inter-process-communication
protocols. Early PC operating systems executed only one process at a
time, sometimes the user was able to switch between multiple launched
processes. Modern operating systems do a fast switching between all
the ready processes, so that the user gets the feeling that all of
them are running in parallel, even when the CPU has only one physical
core. The fast switching between processes is called multitasking or
concurrent execution. Unfortunately these two terms are a bit
misleading, as they seems to imply true parallel execution on
multiple physical CPU cores. But the term concurrency actually only
indicates the fast switching process -- for a few micro-second one
process may be executed, then an automatic task switch occurs, which
includes saving and restoring of all the CPU registers and states,
and the next process is executed again for a few micro-second. This
form of concurrency was already a big progress for the desktop PC, as
it was possible to run processes with a heavy work load, like a
compiler, while the user was still able to use his text editor or web
browser without noticing serious delays for key and mice input or
display updates. Concurrency is generally supported by smart hardware
which can interrupt the current work of the CPU to temporary execute
a different code segment. Hardware like disc controllers or network
cards have its own data buffers or can access parts of the RAM
directly by DMA (Direct Memory Access) and notice the CPU by so
called interrupt signals when a buffer is full (or empty) or when
another condition is met, e.g. when new network data are available.
This interrupt system can drastically improve performance and
throughput, as active waiting in polling busy loops for new network
or disk data can be avoided -- the CPU is free to process one of the
other waiting processes until interrupt signals indicate filled/empty
data buffers or other conditions that needs active CPU intervention.

This form of (hardware interrupt driven) concurrency needs generally
some software support, e.g. the Linux kernel may use the epoll system
for I/O event notifications. Initially it was a common practice to
connect so called callback functions to interrupt driven signals,
e.g. a callback function was invoked whenever some network data
packages has arrived. Some C programs and system libraries work still
this way, for example the glib library of the GTK GUI toolkit. But
use of callbacks can become difficult and confusing for large
applications, sometimes it was called a "callback hell". So languages
like Java, JavaScript or Python introduced a framework called async/
await to simplify the process of writing non blocking asynchronous
software. The async/await framework actually hides the use of
callbacks or use of system functions like epoll from the user. This
asynchronous programming style has gained some popularity due to the
fact that many programs perform a lot of network communication, where
data is transferred often slow compared to the processing power of
the CPU. The Nim standard library provides an async/await framework
which can be used in a similar way as that of Python, and the
external Chronos package of Status corp. offers one more similar
package. Additional, there was a discussion of some of the Nim
developers to support or replace the async/await framework with a
more flexible CPS based one. We should mention, that async/await has
it drawbacks -- its internal working is difficult, its usage is not
always easy, and the user has to be careful when using asynchronous
and synchronous functions together. Async/await was definitely the
best option when desktop PCs had only one single CPU core, but with
the arrival of multi-core CPUs the importance of asynchronous
operations has become less important, as using many threads running
in parallel has become an alternative solution. It is said that
asynchronous program execution has less overhead than just using
parallel processing on multiple cores, that may be one reason why
asynchronous programming is still very popular.

Note that Nim's async/await framework is not a direct component of
the Nim language, but is provided by libraries which are created by
use of Nim's macro and meta-programming capacities. While the async/
await system of the standard library do not support parallel
execution directly, but is executed only on a single thread, it is
generally possible to use async/await with threads running in
parallel. As an example for that you may see https://github.com/dom96
/httpbeast.

For Nim we have many different ways to do parallel program execution,
and for the async/await framework of Nim's standard library the 
chronos alternative implementation is available. Creating new
threads, which are executed in parallel when the CPU has multiple
physical cores, is supported by the threads module. Additional the
Nim standard library provides the threadpool module, which can create
a pool of threads, which may be used by the spawn construct or the 
parallel keyword. Additional, external packages like weave can be
used for high performance parallel processing. And finally, when we
use the C compiler backend, we may also use the parallel construct of
the OpenMP C library.

Some other programming languages like Lua or Go offer also virtually
(green) threads, or coroutines and fibers, and some languages use the
CPS system for a very flexible parallel and asynchronous framework.
Maybe Nim will support that also in the future.

The biggest problem of high performance parallel data processing is
the exchange of data between threads, which has to be performed with
much care to avoid data corruption by uncoordinated random access or
race conditions. For this mutexes, locks and atomic operations can be
used to control the access of common variables, or Channels can be
used to send data from one thread to another one. Another problem for
parallel thread execution can result from the Garbage Collector. For
a system design, where a single GC accesses all data of a process, it
can be necessary to stop all the threads of a process while the GC
does its work. Nim is using for each thread a separate heap area and
a thread local GC, so other threads can continue their work while the
GC cleans up the data of one single thread. The problem of passing
data between treads still exists, but the new ARC/ORC memory
management system may further improve the situation.

In the following sections of the book, we will first demonstrate a
few ways to use multiple threads, which will run in parallel when
there are more than one physical CPU core available. After that, we
will investigate basic async/await operations and show how we can
send data from one thread to another by use of the channels module.

Note: Whenever we intent to use threads in Nim, that is when we
import the threadpool or the threads module, we have to compile our
program with the option --threads:on.

Module threadpool

Creating new threads is always some overhead, so it can make sense to
create a pool of threads, which we then can use to executes parts of
our program.

Using spawn to execute a proc by one thread of the pool

As a first very simple example we will show how we can use the spawn
procedure of the threadpool module to request the execution of a
regular proc. This way we create not really a new thread, but we add
our proc to a list of procs to execute. When one of the threads in
the pool is idle, then our proc is immediately executed by a thread,
otherwise the execution of our proc may be delayed until a thread is
ready to execute it. All the threads of the pool are distributed
among the physical cores of the CPU, so we can really execute procs
in parallel. We have to compile the code using spawn() with the 
--d:threads=on option:

import std/threadpool
proc sum(i: int): int =
  var j = 0
  while j < i:
    inc(j)
    result += j

proc main =
  var a: FlowVar[int] = spawn sum(1e7.int)
  var b = spawn sum(1e7.int)
  echo ^a , " ", ^b

main()

The spawn() function executes an ordinary Nim function by a thread of
the pool. Note that syntactically we do not pass a function and the
functions arguments to spawn, but an expression, which is the the
actual call of the proc! Spawn() returns immediately a variable of 
FlowVar[T] type, which is a container type that can store the result
of our passed function. In the example above we used FlowVar[int] as
our proc sum returns an integer value, but of course the generic 
FlowVar[T] type works for other data types as well, including
sequence and object types. As the instances of FlowVar[T] type are
returned immediately by spawn(), these container variables may be
empty initially. We may then use functions like isReady() from the 
threadpool module to test if the FlowVar[T] variable contains already
the result data, or we can do a blocking wait for the result of our 
proc with the ^ operator. The ^ operator applied to the FlowVar[T]
variable waits for the thread to finish the execution of our proc and
then returns the actual result. If the thread is already finished
when we apply the ^ operator, we get the result immediately. As ^
does a blocking wait, it may look as there would not be much benefit,
but of course we can launch a number of threads with spawn, which can
be processed in parallel, and then we wait with ^ on all of the
results.

In the example above we use a plain proc which sums up the first i
natural numbers, very similar to our very first example program in
part I of the book. We use spawn() to launch two instances of that 
proc, and then wait for the results with the ^ operator applied on
the flowvar. If your PC has more than one physical CPU core, then
both proc instances should be running in parallel, taking only the
total time of one single run. You may compile and launch above code
with nim c --threads:on t.nim; time ./t to see the execution time,
then comment out the second spawn call as well as the echo() call for
Flowvar b and compile and run timed again. Times should be nearly
identical when your PC has at least two CPU cores, indicating true
parallel execution. Of course, launching multiple times the same proc
with the same data makes not much sense, but in real life we could
launch it with different data, or we could use different procs.

As one more example for the use of spawn(), let us investigate how we
can avoid the blocking behaviour of the readLine() proc that we used
earlier in the book. Without special care, a call of readLine()
blocks the main thread of our process, so our program would not be
able to do some useful work or to update the display until the user
terminates his textual input request by pressing the return key. One
possible option to avoid a blocking request for user input may be the
use of the async/await framework, but that may not work well for the
current Nim implementation. So let us just use spawn to execute 
readLine() on one of the threads of the pool:

import std/threadpool
from std/os import sleep
proc doSomeWork =
  echo "not really working that hard..."
  sleep(1000) # sleep 1000 ms

proc main =
  var userInput: FlowVar[string] = spawn readLine(stdin)
  while not userInput.isReady:
    doSomeWork()
  echo "You finally entered: ", ^userInput

main()

In this example we use spawn() to execute the readLine() function of
Nim's standard library by a thread of the treadpool module. We use
the function isReady() to test if the user input is already
available, and call a worker procedure if there is no input yet. As
we have no real work to do, that proc just echos a messages and calls
os.sleep() to create a delay. Note that we use the echo() call in 
doSomeWork() only to show what is going on -- it is obvious that the
repeated printed message would interfere with the user input echoed
by the terminal window. Actually this example is not really that
nice, but it shows you the use of isReady() and at least one possible
way to request user input without blocking the whole app.

The parallel statement

With the parallel statement the threadpool module offers one more way
to use threads to process data in parallel. While the parallel
statement is already available in Nim since many years, it was
recently labeled as experimental feature, so we have to use the 
{.experimental.} pragma to use it. And the detailed description is
currently only available in the experimental section of the manual:
https://nim-lang.org/docs/manual_experimental.html#
parallel-amp-spawn-parallel-statement

With the parallel statement is is easily possible to process large
data, e.g. arrays or sequences, in parallel. The compiler proves the
data access for us to avoid data races or otherwise invalid
operations. As a very simple example we will sum up the elements of
an integer array by use of two threads running in parallel -- when
more then one physical CPU core is available:

import std/threadpool
{.experimental: "parallel".}

proc sum(i, j: int; a: array[8, int]): int =
  for k in i .. j:
    result += a[k]

proc main =
  var a: array[8, int]
  for i in a.low .. a.high:
    a[i] = i

  var s1, s2: int
  parallel:
    s1 = spawn sum(0, 3, a)
    s2 = spawn sum(4, 7, a)
  echo s1, " + ", s2

main()

Inside the parallel block, we use again spawn to launch a function
which is then executed by a thread of the threadpool. The sum()
function in our example code sums up a range of array elements. When 
spawn() is used inside a parallel block, then its semantic is
different: Instead of a FlowVar[T] spawn() now returns directly the
result of the called proc. We can save these results in ordinary
variables and access them freely after the parallel block. In the
above case we would finally sum up the individual result to get the
total sum of all the array elements.

Our example code above is kept very simple by intent to clearly show
the principle use. You may try to modify it to work on sequences with
arbitrary runtime sizes instead of a fixed sized array, and to use
more than two threads. For all the details of the threadpool module
you should of course consult is documentation.

Using the threads module to create new threads

When for some reason we can not use the threadpool module, or we need
more control over the various threads, then we can create our own
treads:

proc sum(i: int) {.thread.} =
  var j, result: int
  while j < i:
    inc(j)
    result += j
  echo result

proc main =
  var th1, th2: Thread[int]
  createThread(th1, sum, 1e7.int)
  createThread(th2, sum, 1e7.int)
  joinThreads(th1, th2)

main()

The createThread() procedure is provided by the threads module, which
is part of the system module -- for that reason we do not have to
explicitly import it. The proc that we want to execute in its own,
newly created thread has to be annotated with the {.thread.} pragma
and has to use one single parameter. We pass the generic Thread[T]
variable, the proc to execute and the proc parameter to createThread
(). The Thread variable must have the same generic type as the
parameter of the proc that we want to execute. In our example that
parameter type is a plain integer, but of course we can use other
data types including objects, tuples or container types like
sequences.

As createThread() does not return a result, we call echo() in our sum
() proc to show what is going on. Actually calling echo() from within
a proc running as a thread may be not a good idea, as multiple echo()
calls from different threads may interfere. We may use the locks
module to make the output operation atomic, but to keep our example
short and simple we ignore that problem for now. The code above
creates two newly created threads, which in our case run the same 
proc with the same data. If there is more than one CPU core
available, then the two threads should be executed in parallel by the
OS. After launching our new treads, we can use the joinThreads()
procedure to wait for the termination of all treads -- we should
generally do that before our app terminates itself.

Using Channels for communication between Threads

When we use the threadpool and spawn() to execute a function by one
of the threads of the pool, we get immediately the result of the
executed function back when the work of the function is done.

Threads created with the createThread() function of the threads
module do not directly return a result but may be executed for a long
time period, often for the whole lifetime of the main process.
Generally it is necessary to exchange messages and data between these
types of threads -- among multiple child threads themselves or among
them and the process's main thread. For this message- and
data-exchange Channels can be used. Nim's Channels use internally a 
queue for sending data from one thread to another thread. A queue is
a first-in-first-out (FIFO) data structure -- items put in first will
also be extracted first. That way the receiving thread will receive
the items in the same order as the sending thread has sent them.

The generic Channel[T] data type and the functions to use it are
provided by the system module, so we do not need to import them.
Channels should be used only for Threads of the threads module, but
not for the hidden threads of the threadpool module. Channels allow
to send messages and data only in one direction, for bidirectional
communication we would need two separate channels. Variables of the 
Channel data type are generally defined at the global scope, to avoid
problems with the thread local garbage collector, and the generic
type of the Channels determines the data type of the messages that we
can send through the Channel. The sent data is deeply copied by the
Channel, which may be not that efficient for large data packages.

In the code below we will present a very simple example for the use
of one single Channel. The proc sum() sums up again the first n
natural integer numbers, but this time the function sums up the
numbers in chunks, and send the sum of each chunk over the Channel to
the parent thread:

var ch: Channel[int]
proc sum(i: int) {.thread.} =
  var j, res: int
  while j < i:
    inc(j)
    res += j
    if j mod 4 == 0:
      ch.send(res)
      res = 0
  ch.send(res) # send the remainder
  ch.send(0) # send zero to indicate termination

proc main =
  var th: Thread[int]
  ch.open()
  createThread(th, sum, 10)
  while true:
    let r = ch.recv()
    if r == 0:
      break
    echo "Received: ", r
  joinThreads(th)
  ch.close()

main()

# Expected output:
Received: 10
Received: 26
Received: 19

The proc sum() sums up continuously 4 more numbers, and then sends
the partial sum into the channel. The generic Channel[int] variable 
ch is defined in global scope. In the main() proc we create the child
thread, open the Channel and read the Channel data with calls to recv
() until we get a zero value as terminating condition. Finally we
call joinThreads() to ensure that the child thread was really
terminated and call close() on the channel to close it. Note that in 
sum() we use an additional send() call to send the last partial sum
which may have less than 4 sumands and so may not have been sent.
Instead of this additional send() call in the while loop a condition
like if j mod 4 == 0 or i == j: could be used of course. When we are
done we send the zero value to indicate to the parent thread that we
are done. This way the parent thread will not wait for more data that
never got send. In the main() proc we use recv() to read the data
from the Channel. Recv() would block if data is not yet available.
Instead we could use tryRecv() with returns a tuple, with the field
dataAvailable indicating if there is already something to read
available. The open() function accepts as second optional argument
the number of items that can be buffered in the internal items queue
of the channel. If that limit is reached, further calls to send()
would block until the reading thread has read the next item. If we
restrict the maximum number of items in the Channel, we may use
instead of send() which may block when the channel is full, trySend()
which just returns false for this case without blocking.

Of course the code example from above makes not much sense, as there
is no real useful work done in parallel, and as there is no reason
for sum() to not just sum up all the elements immediately. But the
example should show you the basic use of Channels, including the need
for having a terminating condition.

Race conditions

A race condition may occurs when two or more threads attempt to read
and write to a shared resource at the same time. Such behavior can
result in data corruption or unpredictable results that are difficult
to debug. Let us consider this tiny example, where two threads
increase the value of a global integer variable:

var counter: int

proc incCounter(i: int) {.thread.} =
  for j in 0 ..< i:
    var local = counter
    local += 1
    counter = local

const N = 1000

proc main =
  var th1, th2: Thread[int]
  createThread(th1, incCounter, N)
  createThread(th2, incCounter, N)
  joinThreads(th1, th2)

main()
echo N, ": ", counter

In the code above the two threads are running concurrent, and in
parallel when your CPU has at least two physical cores. Each thread
increases the global counter variable N times, so one may expect a
final result of 2 * N. But at least when the threads are running in
parallel the actual result will be a random value between N and 2 *N.
The problem is, that the threads do not increase the global counter
in one atomic step, but create a local copy, increase the value of
the copy and write the value back. When the other thread had modified
the global counter variable in between, that modification is
overwritten. When the two threads would run not in parallel but
concurrent on only one CPU core, then the actual result may depend on
the way how the OS does the actual task switching.

These kind of problems are sometimes called race conditions, because
the actual behaviour is determined by the order in which the various
threads access the data. In the example code the actual problem
results from the copying into the local variable and later copying
the value back -- a plain inc() executed on the global variable may
work. We used the local copy here to make the problem visible.
Whenever we would work in such an unordered way onto more complicated
data like strings or objects, we would get corrupted data. This
example should raise your awareness to all the problems which may
occur when multiple threads access global data in an uncontrolled
way.

We have already learned about Channels, which provide a way to
exchange data between threads without the use of global variables.
Other methods to protect global variables from uncontrolled access
which can lead to corrupted states are locks, mutexes or semaphores.
We will give an example to due access control by use of Locks in the
next section. In the example above we used as global data a primitive
value data type. Even more problems may occur, when we try to use
global references data types: In the past the Nim standard library
provided special functions like allocShared() to allocate pointer and
reference data types that can be accessed from multiple treads. But
as Nim's thread handling may change and improve in the future
further, we will not try to discuss all these details here. It should
be enough that you have a feeling for the problems that my arise from
executing multiple threads with shared data -- for the details you
should consult the documentation of Nim's standard library and the
compiler manual.

Guards and Locks

While Nim's Lock data type and the corresponding functions are
defined in the locks module of the standard library, that module
contains only minimal explanations, so we have to consult the
experimental section of the compiler manual.

In computer science, a lock or mutex (from mutual exclusion) is a
synchronization primitive that enforces limits on access to a
resource when there are many threads of execution. Before that
resource is accessed, the lock is acquired, and after the resource is
accessed, it's released. The simplest type of lock is a binary
semaphore. It provides exclusive access to the locked data. Following
this definition from Wikipedia, Nim's locks seems to be actually
binary semaphores.

Nim's Locks are generally used together with the guard pragma, which
we can attach to a global variable that is accessed from more than
one thread. With the guard pragma attached, each thread has to first
acquire the lock before it is allowed to access that variable. If the
lock is already acquired by another thread, acquire blocks until that
other thread releases the lock to indicate that it is done with its
access. Of course these possible blocking can decrease the total
performance, so each thread should acquire the lock only when it
needs really access to the protected data and release the lock as
soon as possible.

We can use the template withLock to access the guarded global
variable in a block -- withLock() acquires the given lock, and
releases the lock again at the end of the block. Accessing a guarded
variable outside a withLock() block would give a compile time error.

import std/locks
var lock: Lock
var counter {.guard: lock}: int

proc incCounter(i: int) {.thread.} =
  for j in 0 ..< i:
    withLock lock:
      var local = counter
      local += 1
      counter = local

const N = 1000

proc main =
  var th1, th2: Thread[int]
  createThread(th1, incCounter, N)
  createThread(th2, incCounter, N)
  joinThreads(th1, th2)

main()
echo N, ": ", counter

When you now run the above code, counter should always have the
desired value 2 * N. Note that replacing the withLock with a plain 
acquire() and release() pair seems not to work for locks that are
used as guards -- but actually there is no reason to do that, the 
withLock block is easier to use and ensures that acquire() and 
release() is always used in matching pairs.

Exceptions in Threads

Whenever a proc that is running as its own thread is raising an
uncaught exception, then the whole process is terminated and a stack
trace with the corresponding error message is displayed in the
terminal window. This applies not only to the threads of the threads
module, but also when the spawn() function is applied to run
functions by one of the threads in the pool.

References

  * http://callbackhell.com/

  * https://journal.stuffwithstuff.com/2015/02/01/
    what-color-is-your-function/

  * https://en.wikipedia.org/wiki/Green_threads

  * https://en.wikipedia.org/wiki/Continuation-passing_style

  * https://github.com/status-im/nim-chronos

  * https://github.com/mratsim/weave

  * https://en.wikipedia.org/wiki/Lock_(computer_science)

Code execution with async/await

The async/await framework allows asynchronous code execution by use
of only one single thread -- the currently active thread can suspend
itself when waiting for data or other events.

Async/await is mostly used for IO bound tasks, where a significant
amount of time is spent by waiting for data to become available. In
such a scenario multi-threading, even when the various threads run
parallel on multiple physical CPU cores, would not really help to
improve the throughput or performance.

The initial idea of asynchronous operations was to avoid blocking the
CPU for longer time periods during slow network and IO (input/output)
requests. Indeed that made much sense in times when we read data from
floppy disks or magnetic tapes, and send data with 300 baud modems.
And when true parallel thread execution was not possible at all as
CPUs had only one core, and computers with more than one CPU where
very expensive and not used by ordinary people.

Today, with network data rates of up to one Gbit/s for our
smartphones or home networks, and SSD devices which have data
transfer rates of multiple Gbit/s, it is not that easy to motivate
the use of asynchronous operations at all. Still for server
applications like online shops or communication platforms used by
thousands of people simultaneously, where network throughput is the
limiting factor and delays have to be avoided, the use of async/await
may actually provide the best performance. And it can be combined
with threading and parallel program execution when needed.

Asynchronous program execution can work with only one thread running
on a single CPU core due to the fact that some hardware like network
cards or disk controllers can read or write small data blocks
autonomously without active CPU support, using their own data buffers
or writing to parts of the system RAM by use of DMA (direct memory
access). These hardware can signal to the CPU when buffers are full
or empty, or when all data transfer is completed, so that the CPU may
copy the buffer content or start to process or display the data. As
this way the external hardware interrupts the current CPU work, these
signals are called hardware interrupts. Operation systems generally
provide various levels of support for these interrupt-driven data
transfer operations, e.g. the epoll framework of the Linux kernel or
Kqueue, the scalable event notification interface in FreeBSD.

For user programs, one solution for doing asynchronous IO by watching
for hardware interrupt signals is to connect callback functions to
these interrupt signal. That way the program can launch an IO
operation, and perform other work, until that work is interrupted by
a call of the callback function. As most programming languages
support the use of callback functions, these form of asynchronous IO
is widely supported by software libraries, e.g. the glib library of
the GTK GUI toolkit. As doing asynchronous IO with callback can get
complicated when we have a lot of nested IO operations, the async/
await workflow was introduced, which allows asynchronous code to be
written in a synchronous style.

The async/await pattern is a syntactic feature of many programming
languages that allows an asynchronous, non-blocking function to be
structured in a way similar to an ordinary synchronous function. It
is semantically related to the concept of a coroutine and is often
implemented using similar techniques, and is primarily intended to
provide opportunities for the program to execute other code while
waiting for a long-running, asynchronous task to complete, usually
represented by promises or similar data structures.^[50] The
programming language F# (pronounced F sharp) introduced a
asynchronous workflows with await points already in 2007 for the
version 2.0 of the language. And in 2012 Microsoft released C# in
version 5 with async/await support. Later languages like Haskell,
Python, JavaScript and TypeScript, Kotlin, Dart, Julia, Zig, Swift
and Rust used the async/await pattern, and since 2020 it is also
available for C++.

Is async/await faster than multi-threading?

For IO-bound task the use of async/await can actually have
performance benefits.

The multi-threaded program execution, that we described in the
previous sections, is some form of preemptive multitasking, where
switching between the active threads occurs at arbitrary time
intervals controlled by the OS. But the async/await pattern is a form
of co-operative multitasking, which provides the user with full
control of the code execution. We can pause the code execution by
using the await keyword when it really makes sense. e.g. when we have
to wait for new data packets or events, and immediately enable
execution of a different code path.

As for this form of co-operative multitasking only the code execution
path is changed, but no switching between threads is necessary,
additional overhead can be avoided, and typical problems of
multi-threading, like the passing of data between different threads
or race conditions do no exist.

So at least in theory the co-operative multitasking controlled by the
async/await pattern is more efficient, and for maximum performance it
can be combined with threading and parallel program execution.

Nim's asynchronous dispatcher

The core elements of Nim's async/await framework are provided by the
modules std/asyncdispatch and std/asyncfutures.

These modules provide a dispatcher, a generic Future[T] type
implementation, and the async macro which allows asynchronous code to
be written in a synchronous style with the await keyword.

The asyncdispatch module implements a global dispatcher (technically
one per thread) that is responsible for running the procedures that
are registered with it.^[51]

Build on top of these two modules there exists various modules for
asynchronous communication: Module std/asyncnet implements a
high-level asynchronous sockets API and std/asynchttpserver
implements a high performance asynchronous HTTP server. Some other
modules like std/httpclient support synchronous and asynchronous data
transfers.

Nim's async/await framework is not part of the language itself, but
implemented with macros and meta-programming and use of Nim's
iterators. The underlying implementation is based on epoll on Linux,
IO Completion Ports on Windows and select on other operating systems.

Currently Nim's async/await uses only one single thread on its own,
but applications can combine it with multiple parallel running
threads. As an alternative implementation we could use https://
github.com/status-im/nim-chronos, which provides similar
functionality.

Asynchronous procedures

Asynchronous procedures are marked by the {.async.} pragma and must
return a generic Future[T] type or return no result at all. In the
later case a Future[void] is assumed. A Future, also called Promise
in other languages, is a generic container type which holds a value
which is not yet available, but which may be available in the future.
So a Future has some similarity with the generic FlowVar type that we
used as return types for threads of Nim's threadpool.

Inside asynchronous procedures the keyword await can be used to call
other asynchronous procedures or procedures which return a Future
type.

The await keyword will suspend the code execution until the awaited 
Future completes. After completion the asynchronous procedure will
resume its execution. During the period when an asynchronous
procedure is suspended other asynchronous procedures will be run by
the dispatcher.

The generic Future[T] data type

The Future[T] data type, which is also called Promise, Delay or 
Deferred in other programming languages, acts as a proxy for a result
that is initially unknown or unavailable.

We can think of a Future[T] as a container; initially it's empty, and
while it remains empty we can't retrieve its value. At some unknown
point in time, something is placed in the container and it is no
longer empty and we can read out its value. That is where the name 
Future comes from.

Every asynchronous operation in Nim returns a Future[T] object, where
the T corresponds to the type of value that the Future promises to
store in the future. We don't have to know that many details of the
internal structure or behaviour of the Future[T] data type, but we
can easily experiment with it without involving any actual
asynchronous I/O operations. The code below shows the behavior of a
simple Future[string] object:

import std/asyncdispatch

proc cb(f: Future[string]) =
  echo "executing callback: ", f.read

let f1: Future[string] = newFuture[string]()
echo f1.finished

f1.callback = cb

f1.complete("Nim and its future")
#f1.fail(newException(ValueError, "Future failed"))

We can create a new instance of the generic Future[T] data type with
the newFuture[T]() constructor, we can query if the instance variable
is already finished, and we can attach a callback function. Finally,
we can call complete() on it to set its value, which then
automatically calls the attached callback function. Or we can call 
fail() on it to set an exception, which later is raised when someone
tries to read its value.

Simple example

We will start our explanations with a very simple asynchronous
program, which will not do an actual asynchronous data transfer yet,
but an asynchronous sleep (wait). The asynchronous sleep called 
sleepAsync() actually behaves very similar to the asynchronous data
transfer functions, that is the execution of the actual code path is
suspended until a hardware condition is fulfilled, and the dispatcher
continues with the code execution.

import std/[asyncdispatch, times]
from std/os import sleep

let to = epochTime()

proc tick(t: string): Future[void] {.async.} =
  for i in 0 .. 1:
    os.sleep(100) # sleep 100 ms
    echo "tick ", t, ((epochTime() - to) * 1000).int, "ms"

let f1: Future[void] = tick(" AAA ")
let f2 = tick(" BBB ")

echo "total time elapsed: ", epochTime() - to

In the code example above we import the asyncdispatch module and have
attached the {.async.} pragma to our tick() procedure. As the tick() 
proc does not return any actual data, we use Future[void] as return
type -- actually we could leave out the return type for this case. We
call tick() two times with different string arguments and use the
function epochTime() to measure the total execution time of our
program. When we compile and run the code we get this output:

tick  AAA 100ms
tick  AAA 200ms
tick  BBB 300ms
tick  BBB 400ms
total time elapsed: 0.4007678031921387

The result is not really surprising, as for each call of proc tick()
the loop in its body is executed two times, generating a 100 ms delay
for each iteration. But the output will drastically change when we
call instead of the ordinary sleep() function the function sleepAsync
() provided by the asyncdispatch module:

import std/[asyncdispatch, times]
from std/os import sleep

let to = epochTime()

proc tick(t: string): Future[void] {.async.} =
  for i in 0 .. 1:
    await sleepAsync(100) # suspend code execution for 100 ms
    echo "tick ", t, ((epochTime() - to) * 1000).int, "ms"

let f1: Future[void] = tick(" AAA ")
let f2 = tick(" BBB ")

echo f1.finished, ' ', f2.finished
echo "time: ", epochTime() - to

waitFor f1
# waitFor f1 and f2 # wait for both futures to finish

echo f1.finished, ' ', f2.finished
echo "total time elapsed: ", epochTime() - to

false false
time: 9.72e-05
tick  AAA 100ms
tick  BBB 100ms
tick  AAA 200ms
tick  BBB 200ms
true true
total time elapsed: 0.20061

The two calls of the tick() proc each returns nearly instantly a 
Future[void] object, no waiting happens here. The use of the await
keyword in the proc body causes the proc to suspend its execution and
control flow returns to the calls site immediately. But at the same
time the asynchronous tick() proc got registered by the dispactcher
loop, so that it can resume its execution.

The returned Future object encapsulates the actual return type of the
call -- in this case only void -- and gives us a reference that we can
use to ask the dispatcher whether our call has completed or not.

But futures can't get resolved by themselves, we need to actually run
the dispatcher in order for any of the code registered with it to
resume its execution. Remember all of this is still running in a
single thread of execution. There are many ways to run the
dispatcher, but in this case it is done by the waitFor call. When we
run waitFor, the dispatcher will run in a loop until the given future
is completed, and the proc which has returned that future is removed
from the dispatcher loop. WaitFor actually calls poll() in a loop
until the future is finished, and then returns the generic value of
the future -- in the code above waitFor returns no actual result, as
we used a Future of void type.

We can use the operators and or or to combine multiple futures, in
this way we can wait until all of them or at least one of them
completes. Note that the dispatcher loops stops when waitFor()
succeeds, so when we wait only for one future and that one finished,
then the dispatcher loop stops and other futures may stay unfinished.

We can use the function finished() to check if a future variable is
already finished. When a future is finished it means that either the
value that it holds is now available or it holds an error instead.
The latter situation occurs when the operation to complete a future
fails with an exception. We can distinguish between the two
situations with the failed() function. Future objects can also store
a callback procedure which will be called automatically once the
future completes, see the example in the previous section.

In our example code above we called waitFor f1 -- this is necessary to
actually execute the dispatcher loop, and to wait for the future f1
to complete. We could have used waitFor f1 and f2, or waitFor f1 or
f2 to wait for completion of both futures or one of them. The result
would be identical in this case, as the proc that returns f1 and f2
is identical and returns always after 2 loop iterations.

The important result of this modified code is, that the proc
execution alternates, and the total runtime of the program is only 
0.2 ms. The reason for this is, as we already explained, that the use
of the await keyword in our tick() proc suspends the execution, and
so immediately the next call of tick() with "BBB" as argument is
executed.

As one more simple example let us investigate this code, where two
different asynchronous procs are executed:

import std/[asyncdispatch, times]

let to = epochTime()

proc numbers() {.async.} =
  for i in 1 .. 3:
    await sleepAsync(250)
    echo i, ' ', ((epochTime() - to) * 1000).int, " ms"

proc letters() {.async.} =
  for i in 'a' .. 'e':
    await sleepAsync(400)
    echo i, ' ', ((epochTime() - to) * 1000).int, " ms"

var
  n = numbers()
  l = letters()
echo "start: ", ((epochTime() - to) * 1000).int, " ms"
waitFor sleepAsync(1500)
echo "done: ", ((epochTime() - to) * 1000).int, " ms"

As both asynchronous procs use different arguments when they call 
sleepAsync(), they are not executed strictly alternating, so the
numbers 2 and 3 are printed with no letter in between:

start: 0 ms
1 250 ms
a 400 ms
2 500 ms
3 751 ms
b 801 ms
c 1202 ms
done: 1501 ms

In this example we do not call waitFor() directly on our actual
asynchronous procs, but on sleepAsync() from asyncdispatch. As the 
procs numbers() and letters() got registered by the dispatcher, they
are executed by the dispatcher loop, but only as long as determined
by waitFor sleepAsync(1500). So the execution of the dispatcher loops
stops already before letters() is really done, and letters d and e
got never printed. The fact that the printed time values can be a few
ms larger than the actual specified sleep times should not surprise
us, as some more code is executed in our procs, and as the dispatcher
loop itself may need some tiny execution time. When an exact timing
should be required, we may use the std/times module to read the exact
time and adjust the actual delays. Also note that async/await as a
co-operative approach of multitasking also means that long running
tasks can delay the execution of other tasks unexpectedly: Imaging
that in our code above the numbers proc would contain a lot of
additional code that takes more than 250 ms to run -- that would
confuse the whole timing scheme. As async/await is most often not
used to create actual delays, but for asynchronous network and IO
operations, we will not discuss the problems of exact timing here in
detail. The linked paper of P. Munch discusses this topic in some
more detail and offers some possible solutions for more accurate
timings.

File download

The module std/httpclient of Nim's standard library provides
procedures for synchronous and asynchronous file transfer. Let us
start with this simple synchronous example to download two small text
files from an URI:

import std/httpclient
let client = newHttpClient()
echo client.getContent("http://ssalewski.de/tmp/texttestpage1.txt")
echo client.getContent("http://ssalewski.de/tmp/texttestpage2.txt")

We have uploaded the two plain text files to that location in
advance, and when we compile and run above code we should get:

This is a plain two
lines test page.

This is one more two
lines test page.

Nim's API documentation for std/httpclient shows us how we can do the
download in an asynchronous way -- at least for one single file:

import std/[asyncdispatch, httpclient]
proc asyncProc(): Future[string] {.async.} =
  let client = newAsyncHttpClient()
  return await client.getContent("http://ssalewski.de/tmp/texttestpage1.txt")
echo waitFor asyncProc()

In this example we use an asynchronous HTTP client, for which the
overloaded proc getContent() returns a Future[string] in this case.
The call of waitFor waits for the download to finish and returns the
actual content of the future, which is a string containing the page
content.

With the knowledge which we get from our previous example with 
sleepAsync(), we can easily modify above code to download two files
asynchronous:

import std/[asyncdispatch, httpclient]
proc asyncProc(url: string): Future[string] {.async.} =
  return await newAsyncHttpClient().getContent(url)

let f1 = asyncProc("http://ssalewski.de/tmp/texttestpage1.txt")
let f2 = asyncProc("http://ssalewski.de/tmp/texttestpage2.txt")

waitFor f1 and f2 # this returns Future[void]
echo f1.read
echo f2.read

The combination f1 and f2 actually creates a new future of void type.
We use two variables f1 and f2 of string type, and read the content
with the read() proc when both futures are completed.

A chat server application

In the API documentation of the std/asyncnet module we find this
example for a very basic chat server application:

import std/[asyncnet, asyncdispatch]

var clients {.threadvar.}: seq[AsyncSocket]

proc processClient(client: AsyncSocket) {.async.} =
  while true:
    let line = await client.recvLine()
    if line.len == 0: break
    for c in clients:
      await c.send(line & "\c\L")

proc serve() {.async.} =
  clients = @[]
  var server = newAsyncSocket()
  server.setSockOpt(OptReuseAddr, true)
  server.bindAddr(Port(12345))
  server.listen()

  while true:
    let client = await server.accept()
    clients.add client

    asyncCheck processClient(client)

asyncCheck serve()
runForever()

The purpose of a chat server is, that multiple clients can connect to
a running server, and then all messages that a client sends to the
server got resend to all other connected clients. So one user can
talk to all the other connected users.

A chat server has to perform two primary tasks:

  * Listen for new connections from potential clients

  * Listen for new messages from clients that have already connected
    to the server

All the messages that the server receive will need to be sent to
every other client that is currently connected to it.

We have not a working client app yet, but in the case that you have
the telnet program installed on you computer, you can already use
that one to test this server. Telnet sends messages unencrypted , so
its use is generally not recommended to send messages over the
internet, but for testing purposes on the local net we may use it. If
the telnet app is not installed on your computer, you may install it
with the package manager of your OS -- for Gentoo Linux we would run
"emerge -av telnet-bsd". An alternative may be the use of the busybox
app, which provides the telnet functionality as well.

If you have a telnet app available, you may open three terminal
windows: On the first one you compile and run the server app -- you
will see no output in that window. In the two other terminals you
type telnet localhost 12345 each. That should start the telnet app
which connects to our running server. When you now type in some text,
that text is echoed to both telnet windows:

$ telnet localhost 12345
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
We use Nim.
We use Nim.

^]
telnet> quit
Connection closed.

Note that terminating the telnet app is not that simple -- you may
have to type CTRL ] first, then you get the telnet prompt, where you
type quit to terminate the app.

Before we will try to explain the details of above server app, we
should summarize a few facts about network communication. Then, at
the end of this section, we will create a simple client app, which
you can use instead of telnet to send messages to the server.

Data transfer over a network

For our chat application we will use the TCP protocol, with so called
network sockets as endpoints. The use of sockets and the TCP protocol
is a common practice in network communication. We will not try to
explain any details here, so citing some definitions from Wikipedia
should be enough for now:

A computer network is a set of computers sharing resources located on
or provided by network nodes. The computers use common communication
protocols over digital interconnections to communicate with each
other.^[52]

The Internet protocol suite, commonly known as TCP/IP, is the set of
communications protocols used in the Internet and similar computer
networks. The current foundational protocols in the suite are the 
Transmission Control Protocol (TCP) and the Internet Protocol (IP).
The Internet protocol suite provides end-to-end data communication
specifying how data should be packetized, addressed, transmitted,
routed, and received.^[53]

A network socket is a software structure within a network node of a
computer network that serves as an endpoint for sending and receiving
data across the network.^[54]

In Nim, a network socket is represented by the Socket data type,
defined in the std/net module. We can create new Socket instances
with a call of newSocket() or newAsyncSocket() for synchronous or
asynchronous communication.

Sockets have some similarity with file descriptors -- instead file
operations like read, write and open, for socket instances we have
the operations recv(), send(), connect(), bindAddr() and listen().
The functions recv() and send() are used to receive or to send data
packages.

TCP is a connection-oriented transport protocol that allows the
socket to be used as a server or as a client. A newly created TCP
socket is neither until the bindAddr() or connect() procedure is
called. The former transforms it into a server socket, and the latter
into a client socket.

By default, the newSocket() constructor will create a TCP socket, but
we could pass more options to the newSocket() constructor for other
socket types or to customize the socket instance.

As we want to create a non blocking, asynchronous server app, we
create our socket instance with a call to newAsyncSocket() of default
TCP type, and then bind it with a call of bindAddr() to a socket
address, that is the combination of an IP address and a port number.
The IP address is a string, it may consist of four or six 8-bit
numbers each separated by a period, or of a symbolic name like 
"google.com". As we want to test our server only on our local
network, we use the default IP address "localhost". The port numbers
are unsigned 16 bit numbers in the range from 0 to 2^16-1, where the
numbers 0 .. 1023 are reserved for special tasks and can generally be
used only with administrator privileges. For a real world app the
used port numbers are important, as server-client communication works
only when both use the same port number. For our experiments we will
use the number 12345 from the initial example, as this one is easily
to remember. As the Port type is a distinct unsigned 16 bit data
type, we have to use the notation Port(12345) for the second
parameter of bindAddr().

We will start our explanations with a simplified code example, where
we have removed the sending of messages to all the clients, and we
have replaced some new function calls like runForever() or asyncCheck
() with similar substitutes that we already know:

import std/[asyncnet, asyncdispatch]

proc processClient(client: AsyncSocket) {.async.} =
  while true:
    let line = await client.recvLine()
    if line.len == 0: break
    echo line

proc serve() {.async.} =
  echo "start serve()"
  let server = newAsyncSocket()
  server.setSockOpt(OptReuseAddr, true)
  server.bindAddr(Port(12345))
  server.listen()
  while true:
    let client = await server.accept()
    let f1: Future[void] = processClient(client)

let f: Future[void] = serve()
echo "back at main scope"
waitFor sleepAsync(320000)

We have two asynchronous procs, serve() and processClient(), which
are both marked with the {.async.} pragma and return a Future[void]
instance each. Our program starts by calling the serve() proc. That 
proc creates an asynchronous socket, binds it to localhost and port 
12345, and starts listening for new connections. At the beginning of
the infinite while true loop await server.accept() is called to
accept new client connections. As no client tries to connect to the
server yet, control is immediately returned back, and the message 
"back at main scope" is printed. Without the last line in our code
with the waitFor statement our program would terminate now. It is
very important that we remember that the call of serve() does not
only call that asynchronous proc, but also add it to the global
dispatcher loop. And with the last line in our code we actually run
this dispatcher loop. We have used waitFor sleepAsync(320000) instead
the original runForever() to make the code look not too
foreign -- running 320 seconds should be good enough for our initial
tests. Note that as long as no client connects to the server, proc 
processClient() is not executed at all. But when a client connects,
then processClient() is called for that client, and an instance of
this processClient() proc with the current client as argument is
added to the global dispatcher loop. This way a new instance of the 
processClient() proc is added to the dispatcher loop whenever one
more client connects to the server. This results to the fact that we
have for each client its own instance of a processClient() proc in
the dispatcher loop, that is executed periodically and so can receive
data for that client. This way all connected clients are served,
although we do not have an actual list of all the clients that we
iterate!

The actual code in proc processClient() is not difficult: await
client.recvLine() tries to receive a textual message from the client,
and gives control back to the dispatcher loop when there is no data
available. And when there is data, then we just print it for now. The
test for line length zero makes some sense and is necessary to
determine when a client disconnects.

When we have managed to understand the simplified code from above,
understanding the original example is easy: We use a sequence with
all the connected clients, as we want to forward each message that we
get from one client to all other connected clients. So the serve() 
proc adds each new client to that seq, and proc processClient()
iterates over that seq and send the received message to all the
connected clients, followed by a "\c\L" to separate the messages. And
instead of waitFor sleepAsync() runForever() is used, and instead of
assigning the results of the procs serve() and processClient() of 
Future[void] type to an unused variable, or to discarding them, these
results are passed to asyncCheck. AsyncCheck is used to provide us
with some error messages if something goes wrong -- it sets a callback
on the future argument which raises an exception if the future should
finish with an error state.

We hope that you do not wonder about the two infinite "while true"
loops any more -- for the async/await pattern such loops makes sense
of course, as each await returns control back to the global
dispatcher loop. And the server would run this loop until it is
terminated by CTRL-C or another OS intervention.

We have intentionally left out some less important points in the
above explanations: The call of server.setSockOpt(OptReuseAddr, true)
should prevent a common problem when apps using sockets are
terminated and restarted: Socket instances are not immediately
removed by the OS when an app terminates, as data packages for that
socket may be still traveling. So a restart of the app may produce
error message like "Socket address is already in use". Another point
is that we used not the IP address string "localhost", but leave out
that parameter, which seems to default to the empty string in that
case. Well, generally the default should make some sense, you may
test with "localhost" yourself to see if that may make a difference.
Finally, we append the string "\c\L" to the messages that we send out
to all of our clients. That is a carriage-return linefeed string,
which is commonly used in network communication to separate messages.
You may still wonder about the capital "L" -- well should be identical
to "\l", you can verify it yourself.

The carefully reader may also wonder if initialization of the client
list with clients = @[] is really necessary. No, should not be
necessary for recent Nim versions, maybe that is a legacy from old
Nimrod days. And is the threadvar pragma in var clients
{.threadvar.}: seq[AsyncSocket] really needed? Our guess would be no,
as the async/await pattern used in this server app is executed only
on the single main thread of the process. But as we are not sure we
have leave it in.

The client application

The client has to connect to the server, and then to watch for
keyboard input from the user and for arrival of new messages from the
server at the same time. So again we have to care to prevent blocking
operations. Unfortunately reading user input in a terminal window is
always blocking, and there is currently no input method available
that is directly supported by Nim's async/await framework. But we
presented earlier in the book already a way to avoid the blocking
behavior of the readLine() proc by use of Nim's threadpool library.
We will use that method again for the realLine() calls, and combine
it with the async/await pattern for sending messages to the server
and for watching for other messages from the server. Actually our
client example program follows closely the client program from Mr.
Picheta, the creator of Nim's async/await framework, which he
sketched in the Manning book years ago:

import std/[threadpool, asyncdispatch, asyncnet]

proc doConnect(socket: AsyncSocket, serverAddr: string) {.async.} =
  echo("Connecting to ", serverAddr)
  await socket.connect(serverAddr, Port(12345))
  echo("Connected!")
  while true:
    let line = await socket.recvLine()
    echo "Received Message: ", line

proc main =
  echo("Chat application started")
  var socket = newAsyncSocket()
  asyncCheck doConnect(socket, "localhost")
  var messageFlowVar = spawn stdin.readLine()
  while true:
    if messageFlowVar.isReady():
      asyncCheck socket.send(^messageFlowVar & "\c\l")
      messageFlowVar = spawn stdin.readLine()
    asyncdispatch.poll()

The structure of this client implementation is a bit different from
the server one. The main reason is, that we have to use Nim's 
threadpool and spawn to avoid the blocking behaviour of the readLine
() proc. Note that our main() proc is not marked with the [.var]
{.async.} pragma and contains no await statement. Only the doConnect
() proc, which connects to the server and then watches for messages
send by the server is marked with the async pragma and awaits the new
messages. The main() proc creates the new asynchronous socket and
then calls the asynchronous proc doConnect(), which actually connects
to the server and enters an infinite loop watching for messages from
the server. When doConnect() calls await, control flow returns
immediately to our main() proc. But doConnect() has become a
component of the dispatcher loop, so its infinite while loop with the
await statement will gain control back later. In the main() proc we
then use spawn to execute readLine() on one thread of the threadpool,
and enter a different infinite while loop. This loops checks if user
input is available, and calls poll() to ensure that the global
dispatcher loop is executed. If there is user input available, that
message is send to the server, and spawn is called again waiting for
the next user input.

Of course you may wonder of this client structure really makes sense.
At least it seems to work. But you may be right -- the use of spawn is
an important component to avoid the blocking terminal input issue,
and the dispatcher loop seems to do not that much contribution.

Feel free to experiment with modified client app structures yourself.

Final remarks

Of course, whenever you should intent to create a real world chat
application there are a lot of other task to solve and points to
discuss: Is the client/server architecture really the best solution,
or may the clients just talk directly to each other, without the use
of a central server? Then there is the problem with the actual port
numbers, as routers and firewalls may block that ports. And finally,
you may intent to send not only plain strings as we did, but
structured message -- maybe add a time and sender name to each
message, and send the content encrypted over the internet. For
encrypting the messages you should find some ideas in Nim' standard
library or in external packages, and sending structured messages is
not difficult: For example we used the JSON format in an earlier
section of the book to save structured objects to disk and reload it
later. The object content was encoded as human readable text, which
you can send of course over the net without any problems. You just
have to define a protocol for the message exchange: Create Nim
objects that contain all the data you want to exchange, like sender
name, time, and the actual message content. Then use one of the procs
provided by the json module to encode the object before you send it,
and encode it again on the receiving side. The json module provides
for example the % operator to convert various data types to JSON
strings or JSON objects, and the parseJson() proc to convert the text
string back into Nim data types. When you have some free time and are
interested in that topic, you can try that yourself, it should be not
difficult. Maybe we will give later in the last part of the book a
concrete example for such an app -- but maybe that is just too trivial
and boring? What you may try as a small exercise is this: We send the
verbatim message over the net, that is exactly what the user typed
in, and we send it to all clients, including the one who initially
send it. So the sender always gets a echoed copy of its input. A
simple exercise for you would be to add a user name to each message,
so that all clients can see who wrote it. And you can use that user
name to identify messages that you send yourself, to suppress the
echoed copy.

Another interesting point is, what happens actually when connected
clients disconnect. There should be at least one serious problem: The
server stores all the connected clients in a list, and sends messages
to all of them. But what happens when a client vanished? Sending
messages to disconnected clients is not really a good idea, so the
server may remove clients from the list when they disconnect, or at
least mark them as disconnected. And when do we have to call close()
on a client that is disconnecting? We have not used close() at all
now, should we use it in the server or in the client app? We will not
try to cover all these details in this book -- when you really should
intent to do some form of network programming, you should consult
some dedicated literature.

For a real world Nim application for network data exchange you may
also investigate this Twitter clone: https://github.com/zedeus/nitter

References:

  * https://en.wikipedia.org/wiki/Asynchronous_I/O

  * https://en.wikipedia.org/wiki/Async/await

  * https://peterme.net/asynchronous-programming-in-nim.html

Concepts

to be done... 

Appendix

Acknowledgments

Special thanks go to Mr. Jim Wilcoxson (https://github.com/hashbackup
) who did proof reading of the first dozen pages of the book and gave
some advice for English grammar and spelling. Thanks go also to Mr.
Marek Lach (https://github.com/marek-lach) for fixing some more
spelling and grammar errors.

ASCII Table

proc print(i: int) =
  let c =
    if i > 31 and i < 128: char(i) else: ' '
  stdout.write("  ", c, "  ")

proc main =
  echo "Visible ASCII Characters\n"
  stdout.write("     ")
  for i in 0 .. 15:
    if i < 10:
       stdout.write(" +")
    else:
      stdout.write("+")
    stdout.write(i, "  ")
  stdout.write('\n')
  var i = 0
  while i < 128:
    if i < 10:
      stdout.write("  ")
    elif i < 100:
      stdout.write(" ")
    stdout.write(i, ' ')
    for j in 0 .. 15:
      print(i + j)
    stdout.write('\n')
    inc(i, 16)

main()

Div and Mod operation

type
  T = array[-5 .. 4, int]
  T2 = array[-5 .. 4, T]

var t: T2

for d in 0 .. 1:
  if d == 0:
    echo "\nResult of i div j"
  else:
    echo "\nResult of i mod j"
  for i in -5 .. 4: # row
    for j in -5 .. 4: # col
      if i == -5 and j == -5:
          t[i][j] = int.high
      elif i == -5:
        t[i][j] = j
      elif j == -5:
        t[i][j] = i
      else:
        if j == 0:
          t[i][j] = int.high
        else:
          if d == 0:
            t[i][j] = i div j
          else:
            t[i][j] = i mod j

  for i in -5 .. 4:
    for j in -5 .. 4:
      if t[i][j] >= 0:
        stdout.write(" ")
      if t[i][j] == int.high:
        stdout.write("  ")
      else:
        stdout.write(t[i][j], " ")
    echo ""

Text styles

We use semantic markup with these text styles:

  * New text: This is new stuff

  * Recent text: This was recently updated

  * First use: term

  * Italic: This is italic

  * Operators: + - & shl

  * Keywords: var ref object import while

  * Use of proc in text: proc

  * Use of macro in text: macro

  * Data types: float int Table

  * String data type: string

  * Array data type: array

  * Function calls: setLen()

  * Variables: i, j, length

  * Module names: strutils, system, io

  * Literals: 100, false, 3.14

  * Constants: fmWrite

  * Code in text: while a > 0 and not done:

  * Terminal text: nim c -gc:arc test.nim

For the term "proc", which is a Nim keyword, we use no keyword
markup, as it occurs so often. Maybe we will later write just
procedure instead. And we do the same for the macro keyword.

ChangeLog

November 2021

We have added a few more simple examples and exercises:

  * Removing adjacent duplicates

  * Array difference

  * Binary search

  * Integer to string conversion

  * No Game Programming?

Februar 2022

  * [Regular Expressions]

  * [Part V: External Packages]

  * Templates (extended)

  * Iterators (extended)

---------------------------------------------------------------------
1. In the past, some form of analog computers existed, some worked
mechanically, some used analog voltages or currents as input and
output signals. Indeed, one important device which is still very
common in analog electronics is the summing amplifier, which can sum
up multiple electric voltages.
2. Well, when we watch very carefully, we will notice that the signal
is not really digital -- when we switch on, the filament may take a
few milliseconds to heat up, and when we switch off, the filament
takes again a few milliseconds to cool down.
3. Of course even digital electric signals can not really "jump" from
one digital state to another, but the transition time is much shorter
than the time duration of the steady state, so the signal has a
rectangular shape when we watch it on an oscilloscope, it looks like
__--__--__.
4. Well, before the OS is loaded and starts execution, often another
tiny program called a Boot Manager is launched. Boot managers are
used to select different operating system to boot, maybe Linux or
Windows, or to pass parameters as the hard disk boot partition number
to the OS.
5. The search for the reason why a program does not do exactly what
was hoped for by its creators is called debugging. That term is still
a legacy from the very first computers in the 50's, where logical
circuits where built by mechanical relays, for example a logical and
operation was built by two relays in series connection. To let the
current flow, both of them would have to be in the conducting state.
And it was told that sometimes insects walked onto the electric
contacts of the relays and blocked them. Today, misbehavior of
computer programs is rarely due to hardware faults, but the term
"bugs" for errors and "debugging" for finding and fixing the errors,
was kept.
6. Exactly speaking, Ruby and Python do not really interpret the
source code, but compile it on the fly to byte-code, which is then
interpreted. And there exists some variants of Ruby and Python that
compile with some success to native machine code. Crystal is a
variant of Ruby, with some significant differences, that compiles to
fast native machine code.
7. Indeed an experimental LLVM backend is already available by third
party contributors.
8. Indeed the Nim compiler compiles itself three times in this time
period to ensure a stable result. Incremental compilation may further
reduce recompile times soon.
9. This syntax is well known from the D programming language, where
it was called Uniform Function Call Syntax (UFCS).
10. If you still wonder why this works that way in decimal and binary
system: Remember how we determine the value of a literal number. We
sum the digits multiplied with powers of the base. And if we multiply
an arbitrary number with the base, each of these powers increases
obviously by one. Write it on a piece of paper when it is not yet
clear to you.
11. If you have a piece of paper and a pencil at hand, you may test
some properties of signed binary numbers represented in two's
complement: take binary 0, apply the two's complement operation to
get the negative of it. Note, we ignore overflow here when we add the
1! That was easy. Can we verify that all negative numbers in two's
complement can really be identified by its set topmost bit? Maybe
that fact is not really obvious, as we not only invert all bits of
the positive number, but also add 1. OK, let us consider the
non-negative numbers 0 .. 127 for an 8-bit word. All those bit
patterns have the topmost bit cleared and all bit combinations used
in the other 7 bits. Inverting these patterns gives us a pattern with
the leftmost bit set and again all bit combinations used in the other
7 bits. Fine so far, the topmost bit is set, but we still have to add
1 to complete our two's complement operation. But the only case where
adding 1 changes the topmost bit is when the 7 other bits are all
set, and that is only the case when the initial value before bit
inversion was zero. So the leftmost bit remains set for all numbers
except initial zero, and zero maps to zero again!
12. For the current Nim implementation, signed overflow generates an
overflow exception, while unsigned types just wrap around. For C it
is similar -- for C99 it is defined that unsigned int types wrap
around, while the behavior for signed ints is undefined and depends
on the actual implementation of the C compiler.
13. Actually the code above would not compile, as the cos() function
has to be imported from the math module. A first line like "from std/
math import cos" would fix that, but we leave that out by intend for
now.
14. Here len() is a predefined function of the Nim standard library,
len() is short for length, and that function returns the actual
length of a text string as an integer value.
15. When we are using the term "size" here, this means how much space
an instance of that type occupies in the RAM of the computer. A type
of size 4 would occupy 4 bytes of the RAM of your computer.
16. The fact that in the current Nim implementation of A. Rumpf float
is identical to float64 should be seen as an implementation detail.
For other implementations the float size may depend on OS and CPU.
17. Well to some degree -- Inf + 1.0 is still Inf, but for Inf / Inf
the result is not that obvious... 
18. Here we use already the for loops, which will be introduced later
in the book.
19. Arrays are homogenous, fixed size containers, we will learn the
details later.
20. We will explain later in this book what pointers are, so if you
have no idea for what pointers are used in computer programming, then
just ignore it for now.
21. Well we have a backspace key on our keyboard, but generally it
does not insert a backspace character but deletes the character to
the left of the cursor when we are editing text. And the return key,
well, it indeed inserts a newline character, but at the same time in
our editor the cursor moves to the next line. Most of the time we
desire a character that generates a newline when we run our program,
but not when we enter our source code.
22. The base types can be sum types, we will discuss them later.
23. It can be difficult to remember if we have to write [8, int
24. A start index of 1 may appear to be more natural, some text books
use start indices 1, and indeed one advantage of start index 1 is,
that in his case the length and the highest position index of the
container are identical. But for systems programming languages like C
it is a common practice to start at zero.
25. Well with some luck the RAM area after the currently used memory
block is still unused. For this case the OS may offer functions like
realloc() to just increase our memory block size, so we can avoid the
copying of the contained data.
26. Actually we can not really delete elements from an array, as it
has a fixed size -- but of course we can just ignore elements at the
end of the array.
27. In some special cases it may be useful to just overwrite entries
with special marker values, instead of actual deleting them. That
operation is fast, order is maintained, and for operations like data
output we can just ignore the marker entries. But this is really only
a good solution is special cases, maybe just using a different
container type like a list or hash table is a preferable solution.
28. Sometimes the compiler may refuse to accept the addr() function,
for example for variables defined with the let keyword. For that case
we may have to use the function unsafeAddr().
29. The name of this temporary variable is fully arbitrary and does
not indicate a special meaning. We could have used el as abbreviation
for element, t or temp to indicate that it is only a temporary
variable. We use the name "node" here because that is a common term
for the elements in linked lists, trees or similar dynamically
created data structures. We could have used the single letter n
instead of "node", but n is already used for the actual friends name
entered by the user.
30. There may be situations, where we want to pass a value parameter
to a proc, but still need to modify it in the proc body. For that
case we can make just a mutable copy, and the copy can even have the
same name as the proc parameter, like var myPar = myPar.
31. We learned already about pointers, and passing var parameters to
procedures and functions is closely related to pointers: The compiler
passes indeed the address of the var parameter. But we have not to
care about these details.
32. Of course compiler become smarter every day, so that restriction
may disappear in future.
33. Automatically inlining of procs from imported modules is not
easy, as the C compiler backend does only inline functions from other
source code files when link time optimization is enabled. With the
inline pragma applied, the Nim compiler copies that proc from the
imported module to the module where it is actually used, so the C
compiler can inline it.
34. https://nim-lang.org/docs/manual.html#
iterators-and-the-for-statement-firstminusclass-iterators
35. https://en.wikipedia.org/wiki/Generic_programming
36. In principle, as the compiler may always optimize that.
37. And in case you find out for what these SkipTable is used, let us
know :-)
38. The carefully reader may notice that we forget to process the
float value for the corner radius in this example.
39. A nimMax() is supposed to be faster than separate min() and max()
calls, as minMax() would have to iterate through all the items in RAM
only once.
40. https://en.wikipedia.org/wiki/Metacharacter
41. https://en.wikipedia.org/wiki/Regular_expression
42. Unfortunately, we have used the term STACK in two different
meanings in this section: For our recursive sorting proc we are
talking about the CPU stack, which is used to pass proc parameters
and store proc local data -- and can overflow. In our nonrecursive 
proc we use a variable which is also called stack, but that is only
an ordinary buffer variable. The term stack is common for such
buffers where we can push() and pop() values.
43. From introductory courses to Computer Science, people generally
remember QuickSort best, just because of its name and its good
performance, as long as we ignore the worst case scenario and the not
stable sorting order.
44. Ruby provides such an operator: https://ruby-doc.org/core-3.0.0/
Array.html#method-i-2D
45. Well, for small data set, maybe up to a few dozen entries, a
linear search may be the fastest still. We have observed this
behaviour already for our various sorting strategies -- logarithmic,
interval halving strategies pays of only for not too tiny data sets.
46. https://en.wikipedia.org/wiki/Parsing_expression_grammar
47. Of course the example from above works only for lines containing
only lower case words -- we will present a better line splitting
example soon.
48. This definition is from the Nim macros tutorial written by A.
Doering, a former paid Nim core developer
49. This definition is from the Nim macros tutorial written by A.
Doering, a former paid Nim core developer
50. https://en.wikipedia.org/wiki/Async/await
51. https://peterme.net/asynchronous-programming-in-nim.html
52. https://en.wikipedia.org/wiki/Computer_network
53. .https://en.wikipedia.org/wiki/Internet_protocol_suite
54. https://en.wikipedia.org/wiki/Network_socket
Last updated 2022-02-25 22:17:31 +0100