[fpc-devel] fphttpclient cannot download a file from w3.org

[Ondrej Pokorny]

Thank you Michael for pointing me to the headers - the problem was 
indeed the missing headers.

INDY sends these 2 headers automatically:
Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
     User-Agent=Mozilla/3.0 (compatible; Indy Library)

When I add both of them to TFPHTTPClient, I get a valid response (both 
are needed). I only used "FPC/3.1 (Free Pascal Compiler)" for the 
user-agent instead of the INDY one :)

Best
Ondrej


On 01.07.2019 17:59, Ondrej Pokorny wrote:
>
> Thank you Michael for the response. I'll check what headers INDY sends.
>
> Ondrej
>
>
> On 01.07.2019 14:57, Michael Van Canneyt wrote:
>>
>> I tested:
>>
>> If I run a strace, it just hangs on the read operation:
>>
>> connect(3, {sa_family=AF_INET, sin_port=htons(80), 
>> sin_addr=inet_addr("128.30.52.100")}, 16) = -1 EINPROGRESS (Operation 
>> now in progress)
>> select(4, NULL, [3], NULL, {tv_sec=3, tv_usec=0}) = 1 (out [3], left 
>> {tv_sec=2, tv_usec=884913})
>> getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
>> fcntl(3, F_GETFL)                       = 0x802 (flags 
>> O_RDWR|O_NONBLOCK)
>> fcntl(3, F_SETFL, O_RDWR)               = 0
>> sendto(3, "GET /TR/2002/REC-xmldsig-core-20"..., 112, 0, NULL, 0) = 112
>> recvfrom(3,
>>
>> If I do the same with 'wget' it also hangs but in the end gets through..
>>
>> But if you add some headers, you get a proper error response.
>>
>> There is little I can do about that. I can only surmise Indy & wget 
>> send different
>> headers (E.G.user-agent) and the server reacts differently on that.
>>
>>
>>
>> Michael.
>>
>> On Mon, 1 Jul 2019, Ondrej Pokorny wrote:
>>
>>> Hello,
>>>
>>> I cannot download 
>>> http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/xmldsig-core-schema.xsd 
>>> with TFPHTTPClient. Delphi INDY downloads it just fine. Does anybody 
>>> know what I am missing?
>>>
>>> Test code:
>>>
>>> program Project1;
>>> uses Classes, SysUtils, fphttpclient;
>>> procedure DownloadFile(const aURI: string; out outStream: TStream);
>>> var
>>>   xClient: TFPHTTPClient;
>>> begin
>>>   outStream := nil;
>>>   xClient := TFPHTTPClient.Create(nil);
>>>   try
>>>     xClient.AllowRedirect := True;
>>>     outStream := TMemoryStream.Create;
>>>     xClient.Get(aURI, outStream);
>>>     outStream.Position := 0;
>>>   except
>>>     on E: Exception do
>>>     begin
>>>       FreeAndNil(outStream);
>>>       Writeln(E.ClassName);
>>>       Writeln(E.Message);
>>>     end;
>>>   end;
>>>   xClient.Free;
>>> end;
>>> var
>>>   S: TStream;
>>> begin
>>> DownloadFile('http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/xmldsig-core-schema.xsd', 
>>> S);
>>>   S.Free;
>>>   Readln;
>>> end.
>>>
>>> Fails with exception:
>>>
>>> EHTTPClient
>>> Error reading data from socket
>>>
>>> #0 fpc_raiseexception(0x145fb14, 0x0, 0xffffffff) at 
>>> ..\inc\except.inc:158
>>> #1 FILLBUFFER(0x145fb14) at fcl-web\src\base\fphttpclient.pp:731
>>> #2 READSTRING(0x15e1c30, 0x0) at fcl-web\src\base\fphttpclient.pp:749
>>> #3 READRESPONSEHEADERS(0x15e1c30) at 
>>> fcl-web\src\base\fphttpclient.pp:854
>>> #4 READRESPONSE(0x15e1c30, 0x15b5dc0, 0x145fdb8, 0, false) at 
>>> fcl-web\src\base\fphttpclient.pp:1132
>>> #5 DONORMALREQUEST(0x15e1c30, {PROTOCOL = 0x15b5e4c 'http', USERNAME 
>>> = 0x0, PASSWORD = 0x0, HOST = 0x15b5ecc 'www.w3.org', PORT = 0, PATH 
>>> = 0x1601c9c '/TR/2002/REC-xmldsig-core'..., DOCUMENT = 0x15bc58c 
>>> 'xmldsig-core-schema.xsd', PARAMS = 0x0, BOOKMARK = 0x0, 
>>> HASAUTHORITY = true}, 0x449d58 'GET', 0x15b5dc0, 0x145fdb8, 0, 
>>> false, false) at fcl-web\src\base\fphttpclient.pp:1226
>>> #6 DOMETHOD(0x15e1c30, 0x449d58 'GET', 0x443060 
>>> 'http://www.w3.org/TR/2002'..., 0x15b5dc0, 0x145fdb8, 0) at 
>>> fcl-web\src\base\fphttpclient.pp:1283
>>> #7 HTTPMETHOD(0x15e1c30, 0x449d58 'GET', 0x443060 
>>> 'http://www.w3.org/TR/2002'..., 0x15b5dc0, 0x145fdb8, 0) at 
>>> fcl-web\src\base\fphttpclient.pp:1397
>>> #8 GET(0x15e1c30, 0x443060 'http://www.w3.org/TR/2002'..., 
>>> 0x15b5dc0) at fcl-web\src\base\fphttpclient.pp:1439
>>> #9 DOWNLOADFILE(0x443060 'http://www.w3.org/TR/2002'..., 0x15b5dc0) 
>>> at project1.lpr:14
>>> #10 main at project1.lpr:30
>>>
>>> Ondrej
>>>
>>> _______________________________________________
>>> fpc-devel maillist  - fpc-devel at lists.freepascal.org
>>> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
>>>
>>
>> _______________________________________________
>> fpc-devel maillist  -fpc-devel at lists.freepascal.org
>> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
>
> _______________________________________________
> fpc-devel maillist  -  fpc-devel at lists.freepascal.org
> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-devel/attachments/20190702/edb5beee/attachment.html>