[SOLVED] Can I make 64bit system-calls in 32bit progs for 64bit file
timestamps?
lazarus / 32-Linux

----------------------------------------------------------------------
Hartmut · 2021-01-26 16:17

In unit "baseunix" there are 2 functions fpUtime() and fpStat() to set
and get the timestamp of a file. In a 32-bit program both functions
use 32-bit variables to pass the timestamps and so they are restricted
to the years 1901..2038. For a couple of reasons I want some more.

In a 64-bit program both functions use 64-bit variables to pass the
timestamps and so more then enough years are possible. But for a
couple of reasons I would like to use this in a 32-bit program, if
possible. What I want to avoid is having to call console programs like
"touch" and "ls -l" e.g. via "TProcess" each time to accomplish this.

So what I want to know is, if / how it is possible to call the
existing 64-bit versions of fpUtime() and fpStat() from a 32-bit
program. I use Linux Ubuntu 64-bit. My imagination is, because a
32-bit program obviously can make system calls to a 64-bit OS, that
there must exist some kind of "interface" to connect those 2 worlds,
but I have no idea how that "works" and if / how it's possible to use
this.

I had a look into the sources of unit "baseunix" in FPC 3.2.0 and
found:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Function
fpUtime(path:pchar;times:putimbuf):cint;begin  fputime:=do_syscall(sys
call_nr_utime,TSysParam(path),TSysParam(times));end; // and for
32-bit:function Do_SysCall(sysnr,param1,param2:TSysParam):TSysResult;
register; external name 'FPC_SYSCALL2';// with:const syscall_nr_utime
= 30;type TSysResult = longint;     TSysParam  = Longint;     putimbuf
is a pointer to a record of 2 x longint // while for 64-bit:function
Do_SysCall(sysnr,param1,param2:TSysParam):TSysResult;  external name
'FPC_SYSCALL2';// with:const syscall_nr_utime = 132;type TSysResult =
int64;     TSysParam  = int64;     putimbuf is a pointer to a record
of 2 x int64
Function fpStat() also "ends" with a do_syscall(). But from this point
I don't know to "follow" what happens in and after the do_syscall() is
invoked...

If there is no direct way, maybe it is possible to create a unit or
some kind of "library", which does the job? Or that something like
this already exists which I can use?

Thanks in advance for your help.

Update: because this Topic got a little longer I added the answer and
a short summary in reply #35

----------------------------------------------------------------------
PascalDragon · 2021-01-27 08:45

Quote from: Hartmut on January 26, 2021, 05:17:08 pmSo what I want to
know is, if / how it is possible to call the existing 64-bit versions
of fpUtime() and fpStat() from a 32-bit program. I use Linux Ubuntu
64-bit. My imagination is, because a 32-bit program obviously can make
system calls to a 64-bit OS, that there must exist some kind of
"interface" to connect those 2 worlds, but I have no idea how that
"works" and if / how it's possible to use this.
No, from a 32-bit process you only have access to the 32-bit
compatible interface of the kernel.

What you need to do instead is to use the correct system call that
provides 64-bit timestamps (fpStat should already do that, if it
doesn't please report a bug). In case of utime or utimes you should
look at the system calls utimensat and utimensat_time64 (and maybe
also report a bug so that the y2038 problem is handled correctly).

Quote from: Hartmut on January 26, 2021, 05:17:08 pmFunction fpStat()
also "ends" with a do_syscall(). But from this point I don't know to
"follow" what happens in and after the do_syscall() is invoked...
There is nothing to follow, because all do_syscall does is invoke the
kernel with the specified arguments and to correctly retrieve the
return value. Anything in between is handled directly by the kernel so
you'd need to look at the syscall interface of the kernel (been there,
done that).

----------------------------------------------------------------------
Hartmut · 2021-01-27 11:22

Thanks a lot PascalDragon for your reply. But I did not understand all
you wrote because I'm not an expert for those things like you.

Quote from: PascalDragon on January 27, 2021, 09:45:43 amNo, from a
32-bit process you only have access to the 32-bit compatible interface
of the kernel.
Does this mean, that a 64-bit Linux provides 2 completely separate
"interfaces": one complete set with all system calls only for 64-bit
programs and a second complete set with all system calls only for
32-bit programs?

QuoteWhat you need to do instead is to use the correct system call
that provides 64-bit timestamps (fpStat should already do that, if it
doesn't please report a bug).
Hmm: when I call fpStat() in a 32-bit program (FPC 3.2.0) it returns a
record, where the 3 timestamps are in 32-bit variables, as also shown
in https://www.freepascal.org/docs-html/current/rtl/baseunix/stat.html
(type Cardinal = LongWord = 32-bit). Is that the bug which I should
report?

QuoteIn case of utime or utimes you should look at the system calls
utimensat and utimensat_time64 (and maybe also report a bug so that
the y2038 problem is handled correctly).
I did not find utimensat and utimensat_time64 in unit "baseunix". Then
I googled for both and found hard stuff but did not find something
what helped me to proceed: it seems that 'utimensat' uses only 32-bit
variables (in a 32-bit program) and that 'utimensat_time64' exists
only in kernel >= 5.1, but I have 4.15 (Ubuntu 18.04).

So I searched the sources of FPC 3.2.0 for both words and found only 2
(Linux related) matches for 'syscall_nr_utimensat':
1) in file <installdir>/fpcsrc/rtl/linux/sysnr-gen.inc => const
syscall_nr_utimensat = 88;
2) in file <installdir>/fpcsrc/rtl/linux/bunxsysc.inc =>
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i {$if
defined(generic_linux_syscalls)}Function
fpUtime(path:pchar;times:putimbuf):cint;var  tsa: Array[0..1] of
timespec;begin  tsa[0].tv_sec := times^.actime;  tsa[0].tv_nsec := 0; 
tsa[1].tv_sec := times^.modtime;  tsa[1].tv_nsec := 0; 
fputime:=do_syscall(syscall_nr_utimensat,AT_FDCWD,TSysParam(path),   
                 TSysParam(@tsa),0);end;I had the idea to try to get
this running, but type 'timespec' uses only 32-bit variables for the
timestamps in a 32-bit program, so I think it can't work.

Please, what did you mean, what should I try exactly? Sorry, I'm a
beginner to this stuff. Thanks for your help.

----------------------------------------------------------------------
PascalDragon · 2021-01-27 12:43

Quote from: Hartmut on January 27, 2021, 12:22:24 pmQuote from:
PascalDragon on January 27, 2021, 09:45:43 amNo, from a 32-bit process
you only have access to the 32-bit compatible interface of the kernel.
Does this mean, that a 64-bit Linux provides 2 completely separate
"interfaces": one complete set with all system calls only for 64-bit
programs and a second complete set with all system calls only for
32-bit programs?
Quote from: Hartmut on January 27, 2021, 12:22:24 pmQuoteWhat you need
to do instead is to use the correct system call that provides 64-bit
timestamps (fpStat should already do that, if it doesn't please report
a bug).
Hmm: when I call fpStat() in a 32-bit program (FPC 3.2.0) it returns a
record, where the 3 timestamps are in 32-bit variables, as also shown
in https://www.freepascal.org/docs-html/current/rtl/baseunix/stat.html
(type Cardinal = LongWord = 32-bit). Is that the bug which I should
report?
Ah, that was only for the block counts, not the timestamps... So yes,
please report.

Quote from: Hartmut on January 27, 2021, 12:22:24 pmQuoteIn case of
utime or utimes you should look at the system calls utimensat and
utimensat_time64 (and maybe also report a bug so that the y2038
problem is handled correctly).
I did not find utimensat and utimensat_time64 in unit "baseunix". Then
I googled for both and found hard stuff but did not find something
what helped me to proceed: it seems that 'utimensat' uses only 32-bit
variables (in a 32-bit program) and that 'utimensat_time64' exists
only in kernel >= 5.1, but I have 4.15 (Ubuntu 18.04).
Then that essentially means that a 32-bit program on a kernel older
than 5.1 can not get 64-bit timestamps this way. At least if I haven't
overlooked anything, but for that I'd have to look deeper for which
right now I don't have the time. So best create a bugreport for
fpUtime as well so that it won't be forgotten.

Quote from: Hartmut on January 27, 2021, 12:22:24 pmPlease, what did
you mean, what should I try exactly? Sorry, I'm a beginner to this
stuff. Thanks for your help.

Best leave it then. Report the bugs and wait for a fix, cause that
stuff definitely isn't easy.

----------------------------------------------------------------------
Kays · 2021-01-27 15:22

Quote from: PascalDragon on January 27, 2021, 01:43:48 pm[…] Then that
essentially means that a 32-bit program on a kernel older than 5.1 can
not get 64-bit timestamps this way. […]I agree. Linux has a die-hard
system call policy: They maintain backward compatibility indefinitely.

Currently, AFAIK all …64 system calls are for file size reasons only.
As of yet, there are no legit reasons to provide 32-bit system calls
accepting 64-bit time stamps. Your software should deal with that
issue nevertheless, but you just do not have a legit reason to, for
example, change a file’s mtime to a point beyond 2021 (or even 2037).

I guarantee you we will adopt any necessary changes in our baseUnix in
due course. However, I can’t guarantee you 32-bit computers will still
be deemed “support-worthy” in 2038.

----------------------------------------------------------------------
Lutz Mändle · 2021-01-27 15:43

Hi,
the timestamps in the stat record are unsigned values, therefore they
go until 2106-02-07 06:28:15 UTC for a 32-bit executable on a 64-bit
system.

Here comes a small test program, which can compiled to 32-bit and
64-bit resp. and until the above mentioned date they show no
difference.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
testfpstat; {$mode objfpc}{$H+} uses  BaseUnix, DateUtils,
SysUtils; var  fname,s:string;  statbuf:stat;  dt:TDateTime; 
p:Pointer; begin  WriteLn('Bitness of executable: ', SizeOf(p) * 8);  
fname:=ParamStr(1);  if not FileExists(fname) then  begin   
WriteLn('no filename given');    Exit;  end;  
FpStat(PChar(fname),statbuf);   WriteLn('Filename:   ', fname); 
WriteLn('mtime__sec: ', statbuf.st_mtime);  WriteLn('mtime_nsec: ',
statbuf.st_mtime_nsec);  dt:=UnixToDateTime(statbuf.st_mtime)+
statbuf.st_mtime_nsec/86400/1000000000;  s:=FormatDateTime('yyyy-mm-dd
hh:nn:ss.zzz',dt);  WriteLn('Filedate:   ', s);end. 
HTH

----------------------------------------------------------------------
MarkMLl · 2021-01-27 15:59

Quote from: Lutz Mändle on January 27, 2021, 04:43:40 pmthe timestamps
in the stat record are unsigned values, therefore they go until
2106-02-07 06:28:15 UTC for a 32-bit executable on a 64-bit system.

I'd note at this point that that is highly platform-specific, since
that data structure is one that Linux does not attempt to keep
consistent across different architectures. The rational here is that
when Linux was ported to a new platform it tried to mimic a number of
things which were considered "graven in stone" by that platform's
dominant unix variant, and that data structure was one of them.

Apart from that I echo PascalDragon's comment about time handling
being complex: I've tried to work through the chain to find out what
could be done about accuracy being lost as TDateTime moves further
from the epoch and not enjoyed it.

MarkMLl

----------------------------------------------------------------------
PascalDragon · 2021-01-28 08:18

Quote from: Lutz Mändle on January 27, 2021, 04:43:40 pmthe timestamps
in the stat record are unsigned values, therefore they go until
2106-02-07 06:28:15 UTC for a 32-bit executable on a 64-bit system.
The question is not only whether the stat record can handle that, but
whether the system call provides dates beyond 2038.

----------------------------------------------------------------------
Hartmut · 2021-01-28 10:08

@PascalDragon:
Ok, I will create the 2 bug reports and mention them here.

Quote from: PascalDragon on January 27, 2021, 01:43:48 pmThen that
essentially means that a 32-bit program on a kernel older than 5.1 can
not get 64-bit timestamps this way.
...
Report the bugs and wait for a fix
If I understand you correctly, such a fix will only work on a kernel
>= 5.1 - Then it would make now no sense for me to wait... (then I
must decide either to be restricted to 64-bit programs or to give up
my idea).

@Kays:
Thanks for that background infos.

@Lutz Mändle:
Thanks a lot for your easy to read demo. Yor are right, that unsigned
values allow years up to 2106 (I've made a lot of tests with various
functions to get the timestamp of a file on different filesystems and
most of them returned signed 32-bit values, so I forgot to mention
this detail when creating this topic).

But this does not solve my problem (it would be a longer story)
entirely: if I treat timestamps as unsigned, I loose years before
1970. And it would not help me much, if I only can read timestamps >
2038, if I cannot set them (e.g. to preserve the timestamp after I
copy a file) and fpUtime() can only set timestamps up to 2038 in a
32-bit program, what I tried.

I had hoped that it would be possible to access 64-bit timestamps in a
32-bit Linux program (in Windows this is easy e.g. via
windows.GetFileTime() and windows.SetFileTime() since many years), but
I learned, that in Linux this is only possible with a kernel >= 5.1.
So now I know my options and can decide what to do.

@MarkMLl:
Thanks for that warning.

@all:
Thanks to all for your help.

----------------------------------------------------------------------
MarkMLl · 2021-01-28 11:46

Quote from: Hartmut on January 28, 2021, 11:08:19 amI had hoped that
it would be possible to access 64-bit timestamps in a 32-bit Linux
program (in Windows this is easy e.g. via windows.GetFileTime() and
windows.SetFileTime() since many years), but I learned, that in Linux
this is only possible with a kernel >= 5.1.
So now I know my options and can decide what to do.

Have you checked clock_gettime() with the CLOCK_REALTIME parameter? I
think that's good for a 32-bit kernel and will give you 64-bit POSIX
seconds, /but/ reconciling that with the result of the Now() function
will be /decidedly/ non-trivial.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i (* This is
an alternative to the RTL's Now() based on the POSIX clock if 
available otherwise falling back to Now(). It is intended to
compensate for  the fact that as timestamps move further from the
epoch they are able to store  fewer fractional bits, it is anticipated
that it will be used primarily for  interval comparisons and that the
operating environment will not change during  program execution in a
way that would cause clock_gettime() to stop working.   Note that this
relies on the extended (10-byte real) type for precision, and  as such
will have portability issues.*)function PosixSecs(): extended;
platform; const  useGettime= true; var  ts: timespec; begin  if
useGettime and (clock_gettime(CLOCK_REALTIME, @ts) = 0) then begin   
result := ts.tv_sec;    result += (ts.tv_nsec / 1E9) (* Since even the
extended type doesn't really have enough digits to express *)(* times
relative to the unix epoch in nanoseconds, assume we have to be    
*)(* very careful with the evaluation order.                         
        *)   end else begin    result := Sysutils.Now();    result -=
UnixEpoch;    result *= SecsPerDay  endend { PosixSecs } ; 
MarkMLl

----------------------------------------------------------------------
PascalDragon · 2021-01-28 12:29

Quote from: MarkMLl on January 28, 2021, 12:46:33 pmQuote from:
Hartmut on January 28, 2021, 11:08:19 amI had hoped that it would be
possible to access 64-bit timestamps in a 32-bit Linux program (in
Windows this is easy e.g. via windows.GetFileTime() and
windows.SetFileTime() since many years), but I learned, that in Linux
this is only possible with a kernel >= 5.1.
So now I know my options and can decide what to do.

Have you checked clock_gettime() with the CLOCK_REALTIME parameter? I
think that's good for a 32-bit kernel and will give you 64-bit POSIX
seconds, /but/ reconciling that with the result of the Now() function
will be /decidedly/ non-trivial.
Hartmut is concerned about the file times (created, modified, access)
not the current time.

----------------------------------------------------------------------
MarkMLl · 2021-01-28 13:30

Quote from: PascalDragon on January 28, 2021, 01:29:25 pmQuote from:
MarkMLl on January 28, 2021, 12:46:33 pmHave you checked
clock_gettime() with the CLOCK_REALTIME parameter? I think that's good
for a 32-bit kernel and will give you 64-bit POSIX seconds, /but/
reconciling that with the result of the Now() function will be
/decidedly/ non-trivial.Hartmut is concerned about the file times
(created, modified, access) not the current time.

Ah. My apologies, that's what comes from not re-reading the thread: I
was wondering how we'd got there via the stat call.

MarkMLl

----------------------------------------------------------------------
Hartmut · 2021-01-28 14:27

Hi MarkMLl, yes we misunderstood. But I noticed your suggestion for
possible future use, thanks.

----------------------------------------------------------------------
Hartmut · 2021-01-29 08:42

I created the requested bug reports:
 - for fpStat() = https://bugs.freepascal.org/view.php?id=38418
 - for fpUtime() = https://bugs.freepascal.org/view.php?id=38419

----------------------------------------------------------------------
marcov · 2021-01-29 09:24

Afaik there is no decision yet about 32-bit Ubuntu packages after the
current LTS. (see e.g. https://ubuntu.com/blog/statement-
on-32-bit-i386-packages-for-ubuntu-19-10-and-20-04-lts )

I sincerely doubt that practical, out of the box 32-bit binaries
support in mainstream distros reach 2025, let alone 2038

----------------------------------------------------------------------
MarkMLl · 2021-01-29 10:07

Quote from: Hartmut on January 28, 2021, 03:27:17 pmHi MarkMLl, yes we
misunderstood. But I noticed your suggestion for possible future use,
thanks.

See
https://github.com/MarkMLl/Mastech_ms2115b/blob/main/inifilesabout.pas

That unit started off reporting LCL version etc., gained the ability
to report the buildstamp and any embedded versioning, gained various
ways of getting the current time and date... it "just growed" :-)

MarkMLl

----------------------------------------------------------------------
Hartmut · 2021-01-29 11:28

Quote from: marcov on January 29, 2021, 10:24:31 amAfaik there is no
decision yet about 32-bit Ubuntu packages after the current LTS. (see
e.g. https://ubuntu.com/blog/statement-on-32-bit-i386-packages-for-
ubuntu-19-10-and-20-04-lts )
I sincerely doubt that practical, out of the box 32-bit binaries
support in mainstream distros reach 2025, let alone 2038
Thanks for that info. I'm not 100% sure that I understood correctly,
what you intended to tell me (my English is not the best). I don't
want to run my programs on a 32-bit Linux (and I have heard several
times that they think/plan about stopping 32-bit Linux). What I only
want is (for a couple of reasons) to run several 32-bit programs on a
64-bit Linux. I have not heard, that they think/plan to stop this
possibility and in your link I found "It has always been our intention
to maintain users’ ability to run 32-bit applications on 64-bit Ubuntu
– our kernels specifically support that".

Quote from: MarkMLl on January 29, 2021, 11:07:51 amSee
https://github.com/MarkMLl/Mastech_ms2115b/blob/main/inifilesabout.pas
That unit started off reporting LCL version etc., gained the ability
to report the buildstamp and any embedded versioning, gained various
ways of getting the current time and date... it "just growed" :-)
I had a look into this unit and found a couple of interesting things,
so I saved it for future activities. Thank you.

----------------------------------------------------------------------
PascalDragon · 2021-01-29 12:05

Quote from: Hartmut on January 29, 2021, 09:42:21 amI created the
requested bug reports:
 - for fpStat() = https://bugs.freepascal.org/view.php?id=38418
 - for fpUtime() = https://bugs.freepascal.org/view.php?id=38419

Thank you.

----------------------------------------------------------------------
Kays · 2021-01-29 16:17

Quote from: Hartmut on January 29, 2021, 12:28:05 pm[…] What I only
want is (for a couple of reasons) to run several 32-bit programs on a
64-bit Linux. […]You can’t do that “just magically out-of-the-box”.

Once you’ve compiled your program for a 32-bit platform, you can’t
just “run” it on a different platform. The loader will reject a non-
compatible ELF file. It’s unavoidable to use some sort of
intermediary, like Bytecode as a platform-independent format, or for
an executable file there are emulators and VMs, and there are probably
lots of other solutions, but you have to deliberately do something
about it.

For now, you can only hold your horses until proper platform-
independent (i.e. fixed record sizes) system calls become available,
and then use them. (Again, we’ll do that for you by adopting any
necessary changes in the RTL.)

----------------------------------------------------------------------
MarkMLl · 2021-01-29 17:49

Quote from: Kays on January 29, 2021, 05:17:12 pmQuote from: Hartmut
on January 29, 2021, 12:28:05 pm[…] What I only want is (for a couple
of reasons) to run several 32-bit programs on a 64-bit Linux. […]You
can’t do that “just magically out-of-the-box”.

Are you sure of your facts there? Because I believe that on Debian
(and derivatives) that's what the multi-arch family of libraries is
for...


$ vi test.pas

$ ppc386 test.pas
Free Pascal Compiler version 3.0.4 [2019/09/08] for i386
Copyright (c) 1993-2017 by Florian Klaempfl and others
Target OS: Linux for i386
Compiling test.pas
Linking test
/usr/bin/ld: warning: link.res contains output sections; did you
forget -T?
6 lines compiled, 0.5 sec

$ file test
test: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
statically linked, stripped

$ ./test
Hello, World!

$ uname -a
Linux xxxxxxxxxx 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28)
x86_64 GNU/Linux


MarkMLl

----------------------------------------------------------------------
Hartmut · 2021-01-29 18:27

Hello Kays and MarkMLl,
I definitely not want to run those 32-bit programs on a different
platform. But thanks for that infos - I did not know that and learned
something new.

----------------------------------------------------------------------
MarkMLl · 2021-01-29 18:50

Quote from: Hartmut on January 29, 2021, 07:27:54 pmHello Kays and
MarkMLl,
I definitely not want to run those 32-bit programs on a different
platform. But thanks for that infos - I did not know that and learned
something new.

Multi-arch isn't a different platform, it's simply making sure the
appropriate packages are installed after which the binaries run
transparently (i.e. the kernel knows how to handle them) and at full
speed.

MarkMLl

----------------------------------------------------------------------
PascalDragon · 2021-01-30 14:43

Quote from: Kays on January 29, 2021, 05:17:12 pmQuote from: Hartmut
on January 29, 2021, 12:28:05 pm[…] What I only want is (for a couple
of reasons) to run several 32-bit programs on a 64-bit Linux. […]You
can’t do that “just magically out-of-the-box”.
If the distro is prepared for it, then yes, you can. Especially Linux
on x86_64 in most distros is usually prepared to run i386 binaries
(both the kernel and the userland). The reason this works is because
the CPU itself supports this. Same is true for running ARM 32-bit code
on a AArch64 OS though I don't know right now whether Linux distros
support it, but the CPU definitely does.

----------------------------------------------------------------------
Hartmut · 2021-01-31 10:15

As mentioned above I created a bug report for improvement of fpStat()
= https://bugs.freepascal.org/view.php?id=38418
There Florian wrote:
QuoteDoesn't fpstatx (trunk only) provide what you are looking for?
That sounds interesting. I will install trunk version in next days and
test and report then.

Does someone know, if fpstatx() requires a Linux kernel >= 5.1 like
utimensat_time64() does?
Does someone know, if there exists something like fpstatx() in FPC
(trunc) but for fpUTime() to set 64-bit timestamps in a 32-bit
program?

----------------------------------------------------------------------
PascalDragon · 2021-01-31 10:38

Quote from: Hartmut on January 31, 2021, 11:15:14 amDoes someone know,
if fpstatx() requires a Linux kernel >= 5.1 like utimensat_time64()
does?
Does someone know, if there exists something like fpstatx() in FPC
(trunc) but for fpUTime() to set 64-bit timestamps in a 32-bit
program?

It's not about what's available in FPC, but what the kernel provides.
The statx system call was introduced with Linux 4.11 according to the
man page. And if the Linux kernel simply doesn't provide system calls
to change the times of a file with a 64-bit width (namely < 5.1 for
32-bit kernels) then there is nothing that an application can do.

----------------------------------------------------------------------
Hartmut · 2021-01-31 11:31

Quote from: PascalDragon on January 31, 2021, 11:38:23 amQuote from:
Hartmut on January 31, 2021, 11:15:14 amDoes someone know, if
fpstatx() requires a Linux kernel >= 5.1 like utimensat_time64() does?
Does someone know, if there exists something like fpstatx() in FPC
(trunc) but for fpUTime() to set 64-bit timestamps in a 32-bit
program?
It's not about what's available in FPC, but what the kernel provides.
The statx system call was introduced with Linux 4.11 according to the
man page. And if the Linux kernel simply doesn't provide system calls
to change the times of a file with a 64-bit width (namely < 5.1 for
32-bit kernels) then there is nothing that an application can do.

Thanks for clarification and the link to the documentation for statx.
For me as a bloody beginner to that stuff this was very helpful.
Because my kernel is 4.15 I should be able to use statx. I will have a
look in trunk how it is realized and try to "copy" it that I can use
it now in programs with FPC 3.2.0.

Because nobody mentioned statx/fpstatx in this Topic before I had hope
that maybe something similar for setting 64-bit file timestamps might
exist in kernel < 5.1 and only no one was aware of. But your answer
lets me think that this is probably not the case.

But solely statx will help me, because I read file timestamps more
often then I set them. And it will solve another problem: if you could
read only 32-bit timestamps, but if the real timestamp is outside the
possible 32-bit range, then on Linux - unfortunately - you don't get
an error signalized (like on Windows), but instead you simply get a
32-bit modulo of the real timestamp, which of course is completely
wrong and there seems no way to be able to detect this case. With
statx this will be solved :-)

----------------------------------------------------------------------
MarkMLl · 2021-01-31 11:56

Quote from: Hartmut on January 31, 2021, 12:31:27 pmBecause nobody
mentioned statx/fpstatx in this Topic before I had hope that maybe
something similar for setting 64-bit file timestamps might exist in
kernel < 5.1 and only no one was aware of. But your answer lets me
think that this is probably not the case.

Please excuse me here if I am "teaching grandmother to suck eggs", but
depending on the extent to which you really are a beginner I think
there's a detail that you're at risk of overlooking.

People have repeatedly said that the syscall for interacting with
(setting?) 64-bit timestamps has been added to the Linux kernel at
version 5.1.

There is a risk that that will only have a useful effect with
filesystems which know about 64-bit timestamps, and in those cases
there might be a minimum version of the filesystem driver (kernel
module) which is distinct from what's in the kernel (I'm thinking of
things like ZFS here, which for political reasons are outside the
definitive kernel tree as managed by Torvalds).

There's also a risk that even if both the kernel and the filesystem
driver/module support 64-bit timestamps, a particular filesystem on a
permanent storage medium won't unless it's been prepared with a
specific version of the maintenance program or marked as having that
specific feature later on (I'm thinking of things like Btrfs here,
where it certainly used to be possible to get the kernel and support
programs out of step).

I'm not saying that any of this is definite, but I think they're
things to be aware of.

MarkMLl

----------------------------------------------------------------------
Hartmut · 2021-01-31 12:43

Quote from: MarkMLl on January 31, 2021, 12:56:30 pmThere is a risk
that that will only have a useful effect with filesystems which know
about 64-bit timestamps, and in those cases there might be a minimum
version of the filesystem driver (kernel module) which is distinct
from what's in the kernel (I'm thinking of things like ZFS here, which
for political reasons are outside the definitive kernel tree as
managed by Torvalds).

Thanks for those infos and warnings. In my case only filesystems NTFS
and EXT4 are affected. I made enough tests, that both can handle
64-bit timestamps, if I e.g. set/get them with "touch" or "ls -l" or a
64-bit FPC program using fpUtime() or fpStat(). What I until now not
have tested is, if it works too on an external USB harddisk with NTFS,
but inspired by what you wrote I will do it next.

----------------------------------------------------------------------
MarkMLl · 2021-01-31 13:02

Quote from: Hartmut on January 31, 2021, 01:43:19 pmThanks for those
infos and warnings. In my case only filesystems NTFS and EXT4 are
affected. I made enough tests, that both can handle 64-bit timestamps,
if I e.g. set/get them with "touch" or "ls -l" or a 64-bit FPC program
using fpUtime() or fpStat(). What I until now not have tested is, if
it works too on an external USB harddisk with NTFS, but inspired by
what you wrote I will do it next.

The more interesting thing to test will be NTFS prepared by any old
Windows version you can lay your hands on.

MarkMLl

----------------------------------------------------------------------
Hartmut · 2021-01-31 15:06

Quote from: MarkMLl on January 31, 2021, 02:02:27 pmThe more
interesting thing to test will be NTFS prepared by any old Windows
version you can lay your hands on.
Yes. In my case this only concerns to Windows 7 and I had already
tested that it works perfectly.

----------------------------------------------------------------------
lucamar · 2021-01-31 19:32

Quote from: MarkMLl on January 31, 2021, 02:02:27 pmThe more
interesting thing to test will be NTFS prepared by any old Windows
version you can lay your hands on.
It shouldn't make any difference. I'm a little rusty on Windows-
related things but IIRC NTFS has had 64 bits (signed) timestamps from
the beggining (or at least since as far back as NT4 which used, I
think, NTFS 1.2), representing a time since ... 1601-1-1 0:0? (UTC) in
steps of 100 ns. So it can go up to $7FFFFFFFFFFFFFFF = 30828-09-14
02:48 or then about

Of course, whether any given system's user tools will be able to
manage the whole date/time range is quite another thing

----------------------------------------------------------------------
Hartmut · 2021-02-01 11:58

Quote from: lucamar on January 31, 2021, 08:32:00 pmQuote from:
MarkMLl on January 31, 2021, 02:02:27 pmThe more interesting thing to
test will be NTFS prepared by any old Windows version you can lay your
hands on.
It shouldn't make any difference. I'm a little rusty on Windows-
related things but IIRC NTFS has had 64 bits (signed) timestamps from
the beggining (or at least since as far back as NT4 which used, I
think, NTFS 1.2), representing a time since ... 1601-1-1 0:0? (UTC) in
steps of 100 ns. So it can go up to $7FFFFFFFFFFFFFFF = 30828-09-14
02:48 or then about
Thanks for that info. Year 30828 should be enough for a while ;-)

--
As mentioned above I created a bug report for improvement of fpUtime()
= https://bugs.freepascal.org/view.php?id=38419
Florian has already fixed it by adding function fputimensat() to unit
'linux' in trunk. Thanks a lot.

----------------------------------------------------------------------
PascalDragon · 2021-02-01 12:01

Quote from: Hartmut on February 01, 2021, 12:58:01 pmAs mentioned
above I created a bug report for improvement of fpUtime() =
https://bugs.freepascal.org/view.php?id=38419
Florian has already fixed it by adding function fputimensat() to unit
'linux' in trunk. Thanks a lot.

FPK has also worked a bit on the SysUtils unit using these functions
as you can see in the revision log (for example FileGetDate was
adjusted already).

----------------------------------------------------------------------
Hartmut · 2021-02-02 08:25

Quote from: PascalDragon on February 01, 2021, 01:01:45 pmFPK has also
worked a bit on the SysUtils unit using these functions as you can see
in the revision log (for example FileGetDate was adjusted already).
Great. Thanks for this info and the revision log link. Did not know it
yet and found it verry interesting.

----------------------------------------------------------------------
Hartmut · 2021-02-05 14:15

Like I said, I installed FPC 3.3.1 trunk and tested new function
linux.statx(), which enables you to read 64-bit file timestamps in
32-bit FPC programs. It works great!
I created a small unit, which copies all needed from FPC 3.3.1 trunk
to enable it's use also in FPC 3.2.0. Maybe it can help someone so I
place it here:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
{Replacement for FPC function linux.statx(), which was added in FPC
3.3.1 trunk, to enable it's use also in earlier FPC versions. Needs a
Linux kernel >= 4.11. With statx() you can e.g. read 64-bit file
timestamps in 32-bit FPC programs. Tested with FPC 3.2.0 and 3.0.4 on
Ubuntu 18.04 64-bit both in 64-bit programs and in cross-compiled
32-bit programs} unit ustatx; interface uses baseunix; const  //
copied from unit 'baseunix' of FPC 3.3.1-trunk rev 48497:   AT_FDCWD =
-100;   AT_SYMLINK_NOFOLLOW = $100;   AT_REMOVEDIR = $200; 
 AT_SYMLINK_FOLLOW = $400;   AT_NO_AUTOMOUNT = $800;   AT_EMPTY_PATH =
$1000;   AT_STATX_SYNC_TYPE = $6000;   AT_STATX_SYNC_AS_STAT = $0000; 
 AT_STATX_FORCE_SYNC = $2000;   AT_STATX_DONT_SYNC = $4000; 
 AT_RECURSIVE = $8000; const  // copied from unit 'linux' of FPC
3.3.1-trunk rev 48497:   STATX_TYPE = $00000001;   STATX_MODE =
$00000002;   STATX_NLINK = $00000004;   STATX_UID = $00000008; 
 STATX_GID = $00000010;   STATX_ATIME = $00000020;   STATX_MTIME =
$00000040;   STATX_CTIME = $00000080;   STATX_INO = $00000100; 
 STATX_SIZE = $00000200;   STATX_BLOCKS = $00000400; 
 STATX_BASIC_STATS = $000007ff;   STATX_BTIME = $00000800;   STATX_ALL
= $00000fff;   STATX__RESERVED = $80000000;    STATX_ATTR_COMPRESSED =
$00000004;   STATX_ATTR_IMMUTABLE = $00000010;   STATX_ATTR_APPEND =
$00000020;   STATX_ATTR_NODUMP = $00000040;   STATX_ATTR_ENCRYPTED =
$00000800;   STATX_ATTR_AUTOMOUNT = $00001000; type  __u16 = Word; 
__u32 = DWord;  __s32 = Longint;  __u64 = QWord;  __s64 = Int64;  
statx_timestamp = record    tv_sec : __s64;    tv_nsec : __u32;   
__reserved : __s32;  end;   tstatx = record    stx_mask : __u32;     
{Mask of bits indicating filled fields}    stx_blksize : __u32;  
 {Block size for filesystem I/O}    stx_attributes : __u64; {Extra
file attribute indicators}    stx_nlink : __u32;      {Number of hard
links}    stx_uid : __u32;        {User ID of owner}    stx_gid :
__u32;        {Group ID of owner}    stx_mode : __u16;       {File
type and mode. See inode for details}    __spare0 : array[0..0] of
__u16;    stx_ino : __u64;        {Inode number}    stx_size : __u64;
     {Total size in bytes}    stx_blocks : __u64;     {Number of 512B
blocks allocated}    stx_attributes_mask : __u64; {Mask to show
supported bits in stx_attributes}                                 {The
4 following fields are file timestamps: }    stx_atime :
statx_timestamp; {Last access}    stx_btime : statx_timestamp;
{Creation (B=Birth)}    stx_ctime : statx_timestamp; {Last Status
change}    stx_mtime : statx_timestamp; {Last modification}       
 {If this file represents a DEVICE, then follows the ID of the device:
}    stx_rdev_major : __u32; {Major ID}    stx_rdev_minor : __u32;
{Minor ID}       {The ID of the device containing the filesystem where
the file resides: }    stx_dev_major : __u32;  {Major ID}   
stx_dev_minor : __u32;  {Minor ID}    __spare2 : array[0..13] of
__u64;  end; function statx(dfd: cint; filename: pchar; flags,mask:
cuint; var buf: tstatx): cint; procedure statx_demo(filename:
string); implementation uses syscall; {$IFDEF CPU32}  // copied from
unit 'syscall' of FPC 3.3.1-trunk rev 48497:const syscall_nr_statx =
383; {only for 32-bit}{$ELSE}const syscall_nr_statx = 332; {only for
64-bit}{$ENDIF} function statx(dfd: cint; filename: pchar; flags,mask:
cuint; var buf: tstatx): cint;   {Needs a Linux kernel >= 4.11.
Result: 0 = OK / -1 = Error.    Good Documentation =
https://man7.org/linux/man-pages/man2/statx.2.html}   begin 
 exit(do_syscall(syscall_nr_statx,TSysParam(dfd),TSysParam(filename), 
                 TSysParam(flags),TSysParam(mask),TSysParam(@buf))); 
 end; procedure statx_demo(filename: string);   {a simple demo for
usage of statx()}   var STX: tstatx;       res: cint;   begin 
 res:=statx(AT_FDCWD,pchar(filename),0,STATX_ALL,STX);   writeln('file
', filename, ' => ', res);   if res <> 0 then exit; {if Error}  
 writeln('stx_mtime: ', STX.stx_mtime.tv_sec);   writeln('stx_atime:
', STX.stx_atime.tv_sec);   writeln('stx_ctime: ',
STX.stx_ctime.tv_sec);   end; end.

----------------------------------------------------------------------
Hartmut · 2021-02-07 16:07

Because this Topic got a little longer here is a short summary:

On Linux it is not possible, that a 32-bit program makes 64-bit system
calls, because a 64-bit Linux provides 2 completely separate
"interfaces": one complete set with all system calls only for 64-bit
programs and a 2nd complete set with all system calls only for 32-bit
programs and a program cannot call something from the "other" set.

To get 64-bit file timestamps in a 32-bit program you can use new
function linux.statx() in FPC 3.3.1 trunk. It needs a Linux kernel >=
4.11. A good documentation is in https://man7.org/linux/man-
pages/man2/statx.2.html. If you want to use this function also in
earlier FPC versions, you can do this with the help of a little unit
which you find in reply #34 of this Topic.

To set 64-bit file timestamps in a 32-bit program you can use improved
functions linux.utimensat() and linux.futimens() also in FPC 3.3.1
trunk. They require a Linux kernel >= 5.1.

Thanks again to all who helped me.