U8 async IO

Simple example:

let io = require("io");

async function testReadLines() {
    let input = await io.openRead("../test/test.txt");
    let n = 1;
    for await (let b of input.lines) {
        console.log(`${n++}: ${b}`);
    }
}

Concepts

The async IO is based on 4 fundamendal concepts:

• U8 InputStream efficiently performs the buffered read operations, character decoding, iterating over the streams and reading them at whole.
• U8 OuputStream efficiently performs the buffered write operations, character encoding, etc.
• U8 Stream Sources help to build the particular input and output streams, for example, io.openRead(url) returns an U8 InputStream that can be used to read the contents of the files.
• U8 handles. Under the hood of a stream there are handles – the objects that provide actual async read and/or write operations with binary buffers (we use Uint8Array instances). These could be the instances of any handle-like classes, which provide at least async read(size) and/or async write(typedArray) operations.

So, the applications perform uniform read/write operations with streams that provide consistent interface for any I/O, and the sources construct particular streams connecting them ot handle-like objects performing actual I/O to various subsystems, like network sockets, files, pipes and so on.

This mimimizes effort of connecting to any new data sources: just provide a handle-like object for it and connect to some existing source with URI schema, or add a new source connector.

File and simple URI streams

Open read stream from a file or URI

let input = await io.openRead(uri); 

• uri could be eithera local FS file name, like ~/docs/daily_notes.md, or any URI the system is capable to connect to.

returns u8 InputStream to read from this source.

Open write stream to a file or URI

let output = openWrite(url, mode = "w", 
                       {bufferLength = chunkSize, umask = 0o644}={});

• uri could be either a local FS file name, like ~/docs/daily_notes.md, or any URI system is capable to connect to.
• mode: one of "w" for write, "a" for append.
• umask: a regular Unix access mask, default is 0o644 whan means -rw-r--r-- in unix terms.

returns u8 OutputStream to write to this target.

Supported URI protocols

Under construction:

• process pipes, network sockets, http downloads.

Input Stream

Input streams provide convenient and uniform asyncronous methods to read any data sources in may ways. Input streams implement read buffering and do read data in blocks where possible.

Note that all read methods share and advance current stream position. It means for example that allBytes() will read not the whole file but the rest of it from the current position.

Construction

Usually, application software does not construct streams but use u8 sources for it. Use constructor only to connect new source.

let input = new InputStream(handle, buferLength = defaultBufferSize);

• handle: handle-like object providing async read(maxSize) returning proxy resolving to Unit8Array with loaded data.

Binary operations

Read portion

let array = await inputStream.read(maxSize);

Reads up to maxSize bytes. Can return less if the stream closes before the data is received.

Read single byte

returns typed array or undefined on end of stream (EOS) reached.

let value = await inputStream.nextByte()

returns next byte or undefined on EOS.

Read stream to the end

let contents = await inputStream.allBytes();

Read stream from the current position to the end and return it as an Unit8Array, or undefined on EOS.

Binary iterator

Iterate remainig bytes one by one.

for await (let value of inputStream.bytes) { /* value is a byte */ }

String operations

All string operations currently use utf8 encoding. More encoding could be added later on the request.

Read remainig stream as string

let text = await inputStream.allAsString();

returns contents decoded from utf8 or undefined on EOS.

Get next line

Read stream bytes until the EOS or line ending character and decode it as utf8 string. Line ending is not included in the result.

let line = await inputStream.nextLine();

returns the line string or undefined on EOS.

Lines iterator

for await (let line of inputStream.lines) { /* line is a string */ }

OutputStream

This is a utility class that simplifies and unifies write operations over some writable handle-like obejct.

Construction

Usually, application software does not construct streams but use u8 sources for it. Use constructor only to connect new source.

const output = new OutputStream(handle, buferLength = defaultBufferSize);

• handle: handle-like object providing async read(maxSize) returning proxy resolving to Unit8Array with loaded data.

Binary operations

await output.write([1,2,3]); // write bytes 0x01, 0x02 and 0x03

writes bunary data from array which should contain byte-size numbers e.g. (0..255) or an Unit8Array instance.

The promise resovles when all data are written. It is safe to call several writes without awaiting them an dthen await last or all.

Common operations

await output.close();

closes the stream. further operation will fail.

U8 Handles

Handles are low-lewel async I/O objects that perform either reading or writing binary data, or both. Handle-like object is used to construct u8 InputStream and u8 OutputStream.

To create custom I/O handle provide an object that perform any or both of:

async function read(maxSize); // returns Uint8Array or undefied on EOS
async function write(uint8ArrayData);

It is a good idea to provide also the u8 source that will take care of constructing custom hanle and return connected stream.