[fpc-pascal] Best way to transfer data between applications?
印場 乃亜
shiruba at galapagossoftware.com
Tue Oct 30 14:34:46 CET 2012
Hi,
Wow I am surprised this is continuing so long - I take it other people have interest in the subject too.
On 2012/10/30, at 17:08, "Jorge Aldo G. de F. Junior" <jagfj80 at gmail.com> wrote:
> 2012/10/30 印場 乃亜 <shiruba at galapagossoftware.com>:
>> Hi,
>>
>> Wow, Class Factories - I am having nightmarish flashbacks to my college C++ classes.
>>
>> Okay, Thank you for all of the suggestions. I will try to compare the suggestions, and see what I can come up with for my situation.
>> Just for further information:
>> 1. I wasn't really planning on using the network, so using something that has to pack/unpack the data from FPC to other
> formats might be overkill.
>
> For the data delivery part, you can use TSimpleIPCServer/TSimpleIPCClient
Which is what I have been doing so far - though I had to look into the implementation and use lower level calls in some cases to stop the app from blocking on reading - at least on OS X. (It would sometimes think there was data waiting, and try to read it, which would freeze the app). Also I have mainly been using CSV. It doesn't seem easily possible to have multiple clients hook to a single server either - though maybe I am missing something.
>
>> 2. I am happy to pass records, Objects aren't really necessary. (If you pass an object in binary, the internal pointers would be referencing the old copy anyway).
>>
>
> The problem with records is that the data sending/receiving function
> must know the record structure in advance, or use a very inflexible
> record strucuture (one size fits all), or use a much more complex
> protocol with multiple entry points selected by a prefix byte or
> something similar.
>
I think that last one if what the debug server protocol does.
> in the end of the day, classes tends to be simpler to pass around.
>
wow... but if you pass a class, then the VMT, etc. needs to be re-done somehow. (i.e. the new copy needs to be actually instantiated, the constructor re-run, etc.).
> Think about this :
>
> Class TA has a property named pname, another property named pvalue.
>
> Instance A is an instance of TA.
> Instance B is an instance of TA.
>
> Instance A has the property pname set to 'somename'
> Instance A has the property pvalue set to 10
>
> Instance B has the property pname set to 'somename'
> Instance B has the property pvalue set to 10
>
> Is A=B ?
By my definition, yes, but not by pointer logic. (Unless the language does some fancy COW stuff).
>
> If you simply look at the pointer (classes are internally pointers to
> data structures in memory), then no, they are different.
>
> But if you look at the values, then they are the same.
>
> What i want to mean ?
>
> If you can, using some IPC/whatever, make a new instance on the other
> application, of the same class, with the same values, then (for a very
> wide interpretation of the meaning of transporting) you transported
> the class from one application to another...even if they are not
> exactly the same instance...
>
Right, if you can easily do this.
> that means that you dont need shared memory nor any fancy trick, just
> a simple protocol like
>
> <classname> <property1value> <property2value> ... <etc>
>
> where classname is used as an index on a class factory and
> property1values are used to set the class properties
>
> will do the trick.
>
> (There are some languages that can LITERALLY transfer an instance
> between two applications, but Pascal cant due to being statically
> typed).
>
Well, it could with some compiler magic I think, if both sides had access to the same type. Actually, I suppose it would be more RTL magic than compiler magic, I suppose. The types are known at compile time (plus there is RTTI), so if the applications shared a common type library, it should seemingly be possible. Of course this approach would be relatively brittle since if anything changed about the type in one place, everything would have to be recompiled with the same change. Not much different from linking, I suppose.
>> I was actually considering using the GUI messaging system like exists in Windows, but:
>> 1. It doesn't really exists on non-windows systems (I read some about one being created for Lazarus on Linux)
>
> TSimpleIPCServer/TSimpleIPCClient is portable across all FPC targets.
Supposedly, but see above.
>
>> 2. I seem to recall that it only allowed passing relatively simple messages. I thought perhaps I could use shared memory and then send a message telling the client application to read the record.
>
> Shared memory is usefull when you need to transfer large blocks of
> data fast between two instances. I believe pipes are simpler for
> simpler tasks (And thats what TSimpleIPCServer/TSimpleIPCClient use).
Right, and I am talking about relatively small amounts of data, not movies or something. The reason I thought of SHM is that there would be no need to decode the structure into a string and re-encode it, etc. Instead, the application could somehow pass a message and a pointer. The receiving application then takes over management of that record (or whatever), and the sending application forgets about it (and allocates a new one next time it needs to send something).
Of course the devil is in the details, which is why I was asking about standard solutions.
>
> TSimpleIPCServer/Client are standard components being part of the FCL.
> (Freepascal component library). At least they come standard with the
> compiler.
>
Which is why I am using them now.
Actually, in case anyone is searching this mail archives for similar solutions in the future, let me say a little more about *why* I am trying to communicate between two applications on the same machine:
1. Using threads in FPC is a pain and has proven unreliable for me. (There needs to be some auto-sunchronization...)
2. I like to separate the logic between the core and the GUI (and what better way than to compile them separately?)
3. I might want to compile the core as 64bit with the GUI as 32bit (which is required on OS X right now...)
4. I might want to have multiple applications (or multiple users) to access the same core without weighing down the machine by running multiple copies of it.
5. The core might be in C or such, while the client is in FPC. (I do this on Mac OS where I write the core in ObjectiveC for easy access to the Cocoa libraries, and then export the data I need to FPC via a named pipe).
6. I might want to have different security on the two "applications". Interestingly OS X has taken this approach recently, but this is slightly different special type of app, and I don't think we have direct support for it in FPC yet.
In all but one of those cases, it's FPC to FPC, so I don't much care the format. I suppose in reality, something like RecordToBlob -> IPC -> BlobToRecord would work fine, as long as the BlobToRecord could detect the original type and return that type.
>>
Thank you,
Noah Silva
>> Maybe once I examine all the options, I will make a FAQ ;)
>>
>> Thank you,
>> Noah Silva
>>
>> On 2012/10/30, at 13:37, "Jorge Aldo G. de F. Junior" <jagfj80 at gmail.com> wrote:
>>
>>> Ah, and you will need a class factory to register the tpersistent
>>> class creators...
>>>
>>> And i said my unit is neater ?
>>>
>>> here it is :
>>>
>>> http://code.google.com/p/pascal-actor-model/source/browse/trunk/rttiobjects.pas
>>>
>>> I use it on a lot of things...
>>>
>>> All you need is to declare published properties that you want streamed
>>> and use TRTTIObject.SaveRTTIToStream.
>>>
>>> It will save on a binary format where the first string is the class
>>> name. You should have a class factory on the other side that registers
>>> all classes that can be exchanged between applications.
>>>
>>> Pascal being statically typed (and binary creating instead of bytecode
>>> creating) means that you cannot "slurp" an object from one application
>>> to another (IE.: Stream an unknown object around).
>>>
>>> Each class should be know in advance by both applications.
>>>
>>> 2012/10/30 Jorge Aldo G. de F. Junior <jagfj80 at gmail.com>:
>>>> Simple solution :
>>>>
>>>> Use TSimpleIPC components + my own RTTIObjects unit.
>>>>
>>>> All messages must be descendants of TRTTIObject. (You can use
>>>> TPersistent, but i believe my unit is neater for this specific task).
>>>>
>>>> Well, using only standard components i believe you should use
>>>> TSimpleIPC components + TPersistent classes
>>>>
>>>> All you need is a way to stream the TPersistent into a string or
>>>> stream then pass that resulting string/stream to the simple ipc
>>>> components.
>>>>
>>>> That will do what you are asking, without much fuss...
>>>>
>>>> 2012/10/30 印場 乃亜 <shiruba at galapagossoftware.com>:
>>>>> Hi,
>>>>>
>>>>> I read about MSGPACK before, but it seems to be a binary version of JSON, right?
>>>>> I actually don't care about the protocol much (and since it's local, Binary vs. text wouldn't make a lot of difference in transfer speed - though using FPC native data structures would be faster since they wouldn't have to be converted to an external format).
>>>>>
>>>>> The problem isn't just the transfer format, but the whole process.
>>>>>
>>>>> For example, I want to have something like the unix message (Signal) system that allows me to send not just an Integer, but any random data.
>>>>>
>>>>> For example, let's say I have a daemon running that receives chat messages (or something) and then it wants to send the data to the GUI, so it would call something like:
>>>>>
>>>>> SendDataToApplication(Gui_App, ChatDataRecord);
>>>>>
>>>>> Obviously, I could manually encode that data into JSON or CSV (or msgpack) and manually send it to the other app by IPC and manually decode it on the other side. What I want is some system that will know that the incoming message is of type TCharDataRecord (or whatever) and automatically let me use that type (and not forced typecasting with pointers, etc. - but properly). Obviously there would be limitations to any solution - for example, I would have to have the same types available on both sides, etc.
>>>>>
>>>>> I will take a look at msgpack, though, to make sure it can't do the above, though. What little JSON I have done in FPC, I have done manually thus far, so perhaps I need to learn the capabilities of the real units before asking more.
>>>>>
>>>>> Mainly my question was asking if there is a "standard" way to do this, since it seems like a common thing to need to do.
>>>>>
>>>>> Thank you,
>>>>> Noah Silva
>>>>>
>>>>> On 2012/10/29, at 18:00, ik <idokan at gmail.com> wrote:
>>>>>
>>>>>> On Mon, Oct 29, 2012 at 5:52 AM, 印場 乃亜 <shiruba at galapagossoftware.com> wrote:
>>>>>>> Greetings,
>>>>>>>
>>>>>>> I am familiar with the basic underlying methods available for transferring data between processes on Windows and Unix, i.e. Pipes, Shared memory, and TCP/IP - but what I am not familiar with is any higher level functionality that may be available on FPC.
>>>>>>>
>>>>>>> As an example: I have one application with a daemon that uses the IPC component to write to a file in CSV format, and then the user application reads this (GPS location) data via IPC. Then I have to re-convert this string data into a series of floating point values manually, though. The IPC component doesn't seem to be reliable on all platforms either (it sometimes blocks on OS X, and at least the debug client doesn't seem to work at all on Windows 7).
>>>>>>>
>>>>>>> Another disadvantage is that the sequence of launching the applications matters, and what's more, it seems there can only be one "client" per "server" in many cases.
>>>>>>>
>>>>>>> More to the point, if I want to pass around structures, records, etc.between processes - what is the best day? For example, Javascript uses JSON. I know there are JSON libraries and various other libraries for Pascal - but I don't know what is the most standard way. (I think this kind of marshalling/unmarshalling may be more standard in Java and C#, but I don't know the "normal" way in Delphi/FPC).
>>>>>>>
>>>>>>> I would prefer to use built-in functionality, rather than learn yet another library - and if learning a library, I would prefer to use one with lots of users that is actively maintained. Likewise, I would prefer to actually "pass" the data, rather than just pass a pointer to it. I plan to have the processes run on the same machine, so I don't need a solution that works with networking, though that would be fine, of course.
>>>>>>>
>>>>>>> Along the same lines, a convenient way to call functions/procedures with parameters from the "other" process would be greatly appreciated. (i.e. something like RPC that handles OOP).
>>>>>>
>>>>>> I'm writing at the moment support for msgpack (http://msgpack.org/) in
>>>>>> plain FPC (object pascal style):
>>>>>> https://github.com/ik5/fp-msgpack
>>>>>>
>>>>>> I will also implement the rpc version.
>>>>>>
>>>>>> MsgPack allow you to take data, including JSON and convert it into
>>>>>> bytes that represent the data.
>>>>>> It contain smaller version of the same way that JSON does it for example.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Noah Silva
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> fpc-pascal maillist - fpc-pascal at lists.freepascal.org
>>>>>>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>>>>>> _______________________________________________
>>>>>> fpc-pascal maillist - fpc-pascal at lists.freepascal.org
>>>>>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>>>>>
>>>>> _______________________________________________
>>>>> fpc-pascal maillist - fpc-pascal at lists.freepascal.org
>>>>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>>> _______________________________________________
>>> fpc-pascal maillist - fpc-pascal at lists.freepascal.org
>>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>>
>> _______________________________________________
>> fpc-pascal maillist - fpc-pascal at lists.freepascal.org
>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
> _______________________________________________
> fpc-pascal maillist - fpc-pascal at lists.freepascal.org
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
More information about the fpc-pascal
mailing list