[fpc-pascal] Best way to transfer data between applications?

Tue Oct 30 09:08:57 CET 2012

2012/10/30 印場 乃亜 <shiruba at galapagossoftware.com>:
> Hi,
>
> Wow, Class Factories - I am having nightmarish flashbacks to my college C++ classes.
>
> Okay, Thank you for all of the suggestions.  I will try to compare the suggestions, and see what I can come up with for my situation.
> Just for further information:
> 1. I wasn't really planning on using the network, so using something that has to pack/unpack the data from FPC to other
formats might be overkill.

For the data delivery part, you can use TSimpleIPCServer/TSimpleIPCClient

> 2. I am happy to pass records, Objects aren't really necessary.  (If you pass an object in binary, the internal pointers would be referencing the old copy anyway).
>

The problem with records is that the data sending/receiving function
must know the record structure in advance, or use a very inflexible
record strucuture (one size fits all), or use a much more complex
protocol with multiple entry points selected by a prefix byte or
something similar.

in the end of the day, classes tends to be simpler to pass around.

Think about this :

Class TA has a property named pname, another property named pvalue.

Instance A is an instance of TA.
Instance B is an instance of TA.

Instance A has the property pname set to 'somename'
Instance A has the property pvalue set to 10

Instance B has the property pname set to 'somename'
Instance B has the property pvalue set to 10

Is A=B ?

If you simply look at the pointer (classes are internally pointers to
data structures in memory), then no, they are different.

But if you look at the values, then they are the same.

What i want to mean ?

If you can, using some IPC/whatever, make a new instance on the other
application, of the same class, with the same values, then (for a very
wide interpretation of the meaning of transporting) you transported
the class from one application to another...even if they are not
exactly the same instance...

that means that you dont need shared memory nor any fancy trick, just
a simple protocol like

where classname is used as an index on a class factory and
property1values are used to set the class properties

will do the trick.

(There are some languages that can LITERALLY transfer an instance
between two applications, but Pascal cant due to being statically
typed).

> I was actually considering using the GUI messaging system like exists in Windows, but:
> 1. It doesn't really exists on non-windows systems (I read some about one being created for Lazarus on Linux)

TSimpleIPCServer/TSimpleIPCClient is portable across all FPC targets.

> 2. I seem to recall that it only allowed passing relatively simple messages.  I thought perhaps I could use shared memory and then send a message telling the client application to read the record.

Shared memory is usefull when you need to transfer large blocks of
data fast between two instances. I believe pipes are simpler for
simpler tasks (And thats what TSimpleIPCServer/TSimpleIPCClient use).

TSimpleIPCServer/Client are standard components being part of the FCL.
(Freepascal component library). At least they come standard with the
compiler.

>
> Maybe once I examine all the options, I will make a FAQ ;)
>
> Thank you,
>    Noah Silva
>
> On 2012/10/30, at 13:37, "Jorge Aldo G. de F. Junior" <jagfj80 at gmail.com> wrote:
>
>> Ah, and you will need a class factory to register the tpersistent
>> class creators...
>>
>> And i said my unit is neater ?
>>
>> here it is :
>>
>> http://code.google.com/p/pascal-actor-model/source/browse/trunk/rttiobjects.pas
>>
>> I use it on a lot of things...
>>
>> All you need is to declare published properties that you want streamed
>> and use TRTTIObject.SaveRTTIToStream.
>>
>> It will save on a binary format where the first string is the class
>> name. You should have a class factory on the other side that registers
>> all classes that can be exchanged between applications.
>>
>> Pascal being statically typed (and binary creating instead of bytecode
>> creating) means that you cannot "slurp" an object from one application
>> to another (IE.: Stream an unknown object around).
>>
>> Each class should be know in advance by both applications.
>>
>> 2012/10/30 Jorge Aldo G. de F. Junior <jagfj80 at gmail.com>:
>>> Simple solution :
>>>
>>> Use TSimpleIPC components + my own RTTIObjects unit.
>>>
>>> All messages must be descendants of TRTTIObject. (You can use
>>> TPersistent, but i believe my unit is neater for this specific task).
>>>
>>> Well, using only standard components i believe you should use
>>> TSimpleIPC components + TPersistent classes
>>>
>>> All you need is a way to stream the TPersistent into a string or
>>> stream then pass that resulting string/stream to the simple ipc
>>> components.
>>>
>>> That will do what you are asking, without much fuss...
>>>
>>> 2012/10/30 印場 乃亜 <shiruba at galapagossoftware.com>:
>>>> Hi,
>>>>
>>>> I read about MSGPACK before, but it seems to be a binary version of JSON, right?
>>>> I actually don't care about the protocol much (and since it's local, Binary vs. text wouldn't make a lot of difference in transfer speed - though using FPC  native data structures would be faster since they wouldn't have to be converted to an external format).
>>>>
>>>> The problem isn't just the transfer format, but the whole process.
>>>>
>>>> For example, I want to have something like the unix message (Signal) system that allows me to send not just an Integer, but any random data.
>>>>
>>>> For example, let's say I have a daemon running that receives chat messages (or something) and then it wants to send the data to the GUI, so it would call something like:
>>>>
>>>> SendDataToApplication(Gui_App, ChatDataRecord);
>>>>
>>>> Obviously, I could manually encode that data into JSON or CSV (or msgpack) and manually send it to the other app by IPC and manually decode it on the other side.  What I want is some system that will know that the incoming message is of type TCharDataRecord (or whatever) and automatically let me use that type (and not forced typecasting with pointers, etc. - but properly).  Obviously there would be limitations to any solution - for example, I would have to have the same types available on both sides, etc.
>>>>
>>>> I will take a look at msgpack, though, to make sure it can't do the above, though.  What little JSON I have done in FPC, I have done manually thus far, so perhaps I need to learn the capabilities of the real units before asking more.
>>>>
>>>> Mainly my question was asking if there is a "standard" way to do this, since it seems like a common thing to need to do.
>>>>
>>>> Thank you,
>>>>    Noah Silva
>>>>
>>>> On 2012/10/29, at 18:00, ik <idokan at gmail.com> wrote:
>>>>
>>>>> On Mon, Oct 29, 2012 at 5:52 AM, 印場 乃亜 <shiruba at galapagossoftware.com> wrote:
>>>>>> Greetings,
>>>>>>
>>>>>> I am familiar with the basic underlying methods available for transferring data between processes on Windows and Unix, i.e. Pipes, Shared memory, and TCP/IP - but what I am not familiar with is any higher level functionality that may be available on FPC.
>>>>>>
>>>>>> As an example:  I have one application with a daemon that uses the IPC component to write to a file in CSV format, and then the user application reads this (GPS location) data via IPC.  Then I have to re-convert this string data into a series of floating point values manually, though.  The IPC component doesn't seem to be reliable on all platforms either (it sometimes blocks on OS X, and at least the debug client doesn't seem to work at all on Windows 7).
>>>>>>
>>>>>> Another disadvantage is that the sequence of launching the applications matters, and what's more, it seems there can only be one "client" per "server" in many cases.
>>>>>>
>>>>>> More to the point, if I want to pass around structures, records, etc.between processes - what is the best day?  For example, Javascript uses JSON.  I know there are JSON libraries and various other libraries for Pascal - but I don't know what is the most standard way.  (I think this kind of marshalling/unmarshalling may be more standard in Java and C#, but I don't know the "normal" way in Delphi/FPC).
>>>>>>
>>>>>> I would prefer to use built-in functionality, rather than learn yet another library - and if learning a library, I would prefer to use one with lots of users that is actively maintained.  Likewise, I would prefer to actually "pass" the data, rather than just pass a pointer to it.  I plan to have the processes run on the same machine, so I don't need a solution that works with networking, though that would be fine, of course.
>>>>>>
>>>>>> Along the same lines, a convenient way to call functions/procedures with parameters from the "other" process would be greatly appreciated.  (i.e. something like RPC that handles OOP).
>>>>>
>>>>> I'm writing at the moment support for msgpack (http://msgpack.org/) in
>>>>> plain FPC (object pascal style):
>>>>> https://github.com/ik5/fp-msgpack
>>>>>
>>>>> I will also implement the rpc version.
>>>>>
>>>>> MsgPack allow you to take data, including JSON and convert it into
>>>>> bytes that represent the data.
>>>>> It contain smaller version of the same way that JSON does it for example.
>>>>>
>>>>>
>>>>>>
>>>>>> Thank you,
>>>>>>    Noah Silva
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> fpc-pascal maillist  -  fpc-pascal at lists.freepascal.org
>>>>>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>>>>> _______________________________________________
>>>>> fpc-pascal maillist  -  fpc-pascal at lists.freepascal.org
>>>>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>>>>
>>>> _______________________________________________
>>>> fpc-pascal maillist  -  fpc-pascal at lists.freepascal.org
>>>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>> _______________________________________________
>> fpc-pascal maillist  -  fpc-pascal at lists.freepascal.org
>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>
> _______________________________________________
> fpc-pascal maillist  -  fpc-pascal at lists.freepascal.org
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal