[Pas2js] [feature request + ideas] Implementing TStream based classes (TBytesStream, TFileStream, TStringStream and so on)

[silvioprog]

Hi Mattias, sorry for the late answer.

I have bad and good news. Also I have an idea about a possible
'full-offline cross-browser virtual-fs' logic. :-)

... and speed vs compatibility.
>
> How do you propose to replace the methods ReadBuffer and WriteBuffer?
>

The bad news:

Unfortunately, the local filesystem specs were discontinued around 2014 by
W3C: "Work on this document has been discontinued and it should not be
referenced or used as a basis for implementation
<https://www.w3.org/TR/file-system-api/>". Another message (from Eric U):
http://lists.w3.org/Archives/Public/public-webapps/2014AprJun/0010.html .

However, the File()/Blob() objects were kept. A File object can be accessed
via an '<input type=file>' or as item of a FileList from an '<input ...
multiple>'. I've done some tests and both works properly, but for security
reasons (which I totally agree) "you can only access a file with the user's
consent". Thus, I can't imagine how we could easily handle it via
<streamed-pascal-classes>.SaveTo/LoadFromFile(), due to user need
to 'click' in a HTML control. :-(

The good news:

Both File/Blob objects allows to create a URL object, that can assigned to
a control (even hidden) and finally downloaded to the user downloads
without his consent. It could be useful for our SaveToFile() methods.
Please take a look at this project (implementing the W3C's saveAs()
FileSaver interface): https://eligrey.com/demos/FileSaver.js . It allows
you to specify the name of the file to be saved (the example uses an image,
text and a rich text) .

At the end of this email I'm going to show how it could be useful + local
buffering...

Does that mean streams require at least ECMAScript6?
>

After checking how pas2js already handles the TBytes (array of byte) type,
we don't need ArrayBuffer anymore. :-) The growing buffer logic and the
error handling can be easily done in purely Pascal code, as you suggested.
Advantages by using TBytes: it will be compatible in any ES (1st edition or
higher). :-)

What about the nodejs platform? As you stated below it provides an API,
> that gives more complete and more direct access.
>

The nodejs' fs is awesome. We should use it for that platform. And ...

What transpiler parameter?
>

... switched using something like this:

{$IFDEF BROWSER} // when the -Tbrowser parameter is passed
... browser logic ...
{$ENDIF}
...
{$IFDEF NODEJS} // when the -Tnodejs parameter is passed
... nodejs logic ...
{$ENDIF}

I'm not sure if these macros are available when passing the
'-T<script-style>' parameter, but we could easily add it.

A big challenge is the asynchronous file system access in the
> browser. How do you want to handle this?


Indeed. But I have an idea which I would like to check if you agree:

The following native technique would be great, but it is Chrome-specific
and it's unlikely to be implemented by other browsers and is no longer
being standardized with the W3C:
https://www.html5rocks.com/en/tutorials/file/filesystem . (It have the
latest Chromium in my Linux and the 'FileSystem API' availability is 'No'
in the http://html5test.com checker). However, we could use only this logic
idea. Supposing we would use a browser storage to implement the TFileStream:

- since its last parameter is a Word type, we could add  two new types
pas2js-specific without breaking the original TFileStream ABI:
fmUseSessionStorage/fmUseLocalStorage (that can be flagged as 'fmCreate or
fmUseSessionStorage', 'fmOpenReadWrite or fmUseLocalStorage' and so on);
- the write(), read(), seek() logics would be handled in the
session/localstorage (according the fmUse* above);
- the flush() logic would read the buffer from session/localstorage and
downloaded to the user downloads (using the FileSaver.js idea);
- if any fmUseSessionStorage/fmUseLocalStorage is passed, it assumes
fmUseSessionStorage be default to make the browser storage clear.

The session/localstorage are good because they don't need a user agreement
to be used, and both are already consolidated in several browsers (even
mobile ones). Perhaps they are even more useful than Service Workers
<https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers>
(SW)
to handle big files, because SW is a very recent technology and I'm not
sure if it provides anything to be used in our seek() logic. IMHO, the
session/localstorage has good limits for our buffering. This link shows a
table + info about them and 'offline' data limits:
https://www.html5rocks.com/en/tutorials/offline/quota-research .

Use Cases of the idea:

- when a file or directory is selected for upload, it copies the files into
a local browser data and uploads a chunk at a time, rather than sending a
file at a time in a XHR;
- uploads/downloads can be restarted after browser crashes, network
interruptions, etc. using an efficient streaming + seek;
- the ability to organize and configure projects by creating directories,
files etc.;
- edited files could be accessed by client-side applications;

Some topics above were cut from this article
<https://www.html5rocks.com/en/tutorials/file/filesystem/> and its HTML5
Terminal project <http://www.htmlfivewow.com/demos/terminal/terminal.html>
example. It uses the unstandardized fs, but it can be implemented using
session/localstorage with some work.

What do you think? Maybe I'm dreaming out loud. 😅

--
Silvio Clécio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/pas2js/attachments/20181110/29ed905e/attachment.html>