Page: U8 async IO
2019-11-21 11:11
U8 async IO
Simple example:
let io = require("io");
async function testReadLines() {
let input = await io.openRead("../test/test.txt");
let n = 1;
for await (let b of input.lines) {
console.log(`${n++}: ${b}`);
}
}
Concepts
The async IO is based on 4 fundamendal concepts:
- U8 InputStream efficiently performs the buffered read operations, character decoding, iterating over the streams and reading them at whole.
- U8 OuputStream efficiently performs the buffered write operations, character encoding, etc.
- U8 Stream Sources help to build the particular input and output streams, for
example,
io.openRead(url)
returns an U8 InputStream that can be used to read the contents of the files. - U8 handles. Under the hood of a stream there are
handles
– the objects that provide actual async read and/or write operations with binary buffers (we useUint8Array
instances). These could be the instances of any handle-like classes, which provide at leastasync read(size)
and/orasync write(typedArray)
operations.
So, the applications perform uniform read/write operations with streams that provide consistent interface for any I/O, and the sources construct particular streams connecting them ot handle-like objects performing actual I/O to various subsystems, like network sockets, files, pipes and so on.
This mimimizes effort of connecting to any new data sources: just provide a handle-like object for it and connect to some existing source with URI schema, or add a new source connector.
File and simple URI streams
Open read stream from a file or URI
let input = await io.openRead(uri);
uri
could be eithera local FS file name, like~/docs/daily_notes.md
, or any URI the system is capable to connect to.
returns u8 InputStream to read from this source.
Open write stream to a file or URI
let output = openWrite(url, mode = "w",
{bufferLength = chunkSize, umask = 0o644}={});
uri
could be either a local FS file name, like~/docs/daily_notes.md
, or any URI system is capable to connect to.mode
: one of"w"
for write,"a"
for append.umask
: a regular Unix access mask, default is 0o644 whan means-rw-r--r--
in unix terms.
returns u8 OutputStream to write to this target.
Supported URI protocols
protocol | direction | description |
---|---|---|
file:/ file:/// |
rw | local filesystem access limited only by the process owner rights |
Under construction:
- process pipes, network sockets, http downloads.
Input Stream
Input streams provide convenient and uniform asyncronous methods to read any data sources in may ways. Input streams implement read buffering and do read data in blocks where possible.
Note that all read methods share and advance current stream position. It means
for example that allBytes()
will read not the whole file but the rest of it from
the current position.
Construction
Usually, application software does not construct streams but use u8 sources for it. Use constructor only to connect new source.
let input = new InputStream(handle, buferLength = defaultBufferSize);
handle
: handle-like object providingasync read(maxSize)
returning proxy resolving toUnit8Array
with loaded data.
Binary operations
Read portion
let array = await inputStream.read(maxSize);
Reads up to maxSize bytes. Can return less if the stream closes before the data is received.
Read single byte
returns typed array or undefined
on end of stream (EOS) reached.
let value = await inputStream.nextByte()
returns next byte or undefined on EOS.
Read stream to the end
let contents = await inputStream.allBytes();
Read stream from the current position to the end and return it as an
Unit8Array
, or undefined
on EOS.
Binary iterator
Iterate remainig bytes one by one.
for await (let value of inputStream.bytes) { /* value is a byte */ }
String operations
All string operations currently use utf8 encoding. More encoding could be added later on the request.
Read remainig stream as string
let text = await inputStream.allAsString();
returns contents decoded from utf8 or undefined
on EOS.
Get next line
Read stream bytes until the EOS or line ending character and decode it as utf8 string. Line ending is not included in the result.
let line = await inputStream.nextLine();
returns the line string or undefined on EOS.
Lines iterator
for await (let line of inputStream.lines) { /* line is a string */ }
OutputStream
This is a utility class that simplifies and unifies write operations over some writable handle-like obejct.
Construction
Usually, application software does not construct streams but use u8 sources for it. Use constructor only to connect new source.
const output = new OutputStream(handle, buferLength = defaultBufferSize);
handle
: handle-like object providingasync read(maxSize)
returning proxy resolving toUnit8Array
with loaded data.
Binary operations
await output.write([1,2,3]); // write bytes 0x01, 0x02 and 0x03
writes bunary data from array which should contain byte-size numbers e.g. (0..255) or an Unit8Array instance.
The promise resovles when all data are written. It is safe to call several writes without awaiting them an dthen await last or all.
Common operations
await output.close();
closes the stream. further operation will fail.
U8 Handles
Handles are low-lewel async I/O objects that perform either reading or writing binary data, or both. Handle-like object is used to construct u8 InputStream and u8 OutputStream.
To create custom I/O handle provide an object that perform any or both of:
async function read(maxSize); // returns Uint8Array or undefied on EOS
async function write(uint8ArrayData);
It is a good idea to provide also the u8 source that will take care of constructing custom hanle and return connected stream.