read() wont allow me to read files larger than 2 gig (on a 64bit)
glide creme
glidecreme at gmail.com
Fri Dec 25 12:35:50 UTC 2009
Yes I tried that (as I also wrote in my original message).
But adding D_FILE_OFFSET shouldnt be necessary on 64bit system right?
thanks
On Fri, Dec 25, 2009 at 5:29 AM, Stefan Bader
<stefan.bader at canonical.com> wrote:
> glide creme wrote:
>> Thanks for your reply
>>
>> Yes stat, calculates the filesize correctly,
>>
>> This is the output of the program
>>
>> $ls -l bigfile.dat
>> -rw-r--r-- 1 tmp tmp 2163946253 2009-12-23 07:57 bigfile.dat
>> $ ./a.out
>> LONG_MAX:9223372036854775807
>> SSIZE_MAX:9223372036854775807
>> a.out: Read only 2147479552 of 2163946253 bytes: Succes
>>
>> This was I find strange,
>> because the LONG_MAX is 64bit, and SSIZE_MAX should be defined to be LONG_MAX,
>> but it looks like read is only limited to 32bit.
>> -------------------------------------------------------------
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <err.h>
>> #include <fcntl.h>
>> #include <sysexits.h>
>> #include <unistd.h>
>> #include <sys/stat.h>
>> #include <limits.h>
>> #include <bits/posix1_lim.h>
>> #include <sys/types.h>
>> // get bytesize of file
>> size_t fsize(const char* fname){
>> struct stat st ;
>> stat(fname,&st);
>> return st.st_size;
>> }
>>
>>
>>
>>
>> int main() {
>> const char *infile = "bigfile.dat";
>> int fd;
>> size_t bytes_read, bytes_expected = fsize(infile);
>> char *data;
>> printf("LONG_MAX:%lu\n",LONG_MAX);
>> printf("SSIZE_MAX:%lu\n",SSIZE_MAX);
>>
>> if ((fd = open(infile,O_RDONLY)) < 0)
>> err(EX_NOINPUT, "%s", infile);
>>
>> if ((data =(char *) malloc(bytes_expected)) == NULL)
>> err(EX_OSERR, "data malloc");
>>
>> bytes_read = read(fd, data, bytes_expected);
>>
>> if (bytes_read != bytes_expected)
>> err(EX_DATAERR, "Read only %lu of %lu bytes",bytes_read, bytes_expected);
>>
>> /* ... operate on data ... */
>>
>> free(data);
>>
>> exit(EX_OK);
>> }
>> -------------------------------------------------------------
>>
>> On Fri, Dec 25, 2009 at 10:04 AM, AceLan Kao <acelan.kao at canonical.com> wrote:
>>> Dear Glide,
>>>
>>> Does the stat() return the correct file size?
>>> I remember that there is a 64bits version of stat(), stat64(), maybe
>>> that's the problem?
>>>
>>> Best regards,
>>> AceLan Kao.
>>>
>>> 2009/12/25 glide creme <glidecreme at gmail.com>:
>>>> Hi
>>>> Sorry if I'm asking on the wrong mailinglist,
>>>> but I've asked on various forums and I havn't found a solution.
>>>>
>>>> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
>>>> the problem is not related to the allocation itself, which works.
>>>> Attached is a sample program that illustrates this.
>>>>
>>>> I've tried with various, compiler flags like
>>>> -D_FILE_OFFSET_BITS=64
>>>>
>>>> According to man 2 read, the maximum is limited by SSIZE_MAX
>>>> which is defined in
>>>>
>>>> /usr/include/bits/posix1_lim.h
>>>> # define SSIZE_MAX LONG_MAX
>>>>
>>>> And LONG_MAX is defined in /usr/include/limits.h as
>>>> # if __WORDSIZE == 64
>>>> # define LONG_MAX 9223372036854775807L
>>>> # else
>>>> # define LONG_MAX 2147483647L
>>>> # endif
>>>> # define LONG_MIN (-LONG_MAX - 1L)
>>>>
>>>> Is this the expected behaviour for the read function on a 64bit platform,
>>>> or is something broken
>>>>
>>>> Thanks in advance
>>>>
>>>> edit:
>>>>
>>>> by the way
>>>>
>>>> readelf -h ./a.out
>>>> ELF Header:
>>>> Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>>>> Class: ELF64
>>>> Data: 2's complement, little endian
>>>> Version: 1 (current)
>>>> OS/ABI: UNIX - System V
>>>> ABI Version: 0
>>>> Type: EXEC (Executable file)
>>>> Machine: Advanced Micro Devices X86-64
>>>> Version: 0x1
>>>> Entry point address: 0x400750
>>>> Start of program headers: 64 (bytes into file)
>>>> Start of section headers: 5312 (bytes into file)
>>>> Flags: 0x0
>>>> Size of this header: 64 (bytes)
>>>> Size of program headers: 56 (bytes)
>>>> Number of program headers: 9
>>>> Size of section headers: 64 (bytes)
>>>> Number of section headers: 37
>>>> Section header string table index: 34
>>>>
>>>> ldd ./a.out
>>>> linux-vdso.so.1 => (0x00007fff689ff000)
>>>> libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
>>>> libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
>>>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
>>>> libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
>>>> /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)
>>>>
>>>> uname -a
>>>> Linux csi 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
>>>> 2009 x86_64 GNU/Linux
>>>>
>>>> --
>>>> kernel-team mailing list
>>>> kernel-team at lists.ubuntu.com
>>>> https://lists.ubuntu.com/mailman/listinfo/kernel-team
>>>>
>>>>
>>>
>>>
>>> --
>>> Chia-Lin Kao(AceLan)
>>> http://blog.acelan.idv.tw/
>>> E-Mail: acelan.kaoATcanonical.com (s/AT/@/)
>>>
>>
>
> Have you tried the following?
>
> -Stefan
>
> FEATURE_TEST_MACROS(7) Linux Programmer's Manual FEATURE_TEST_MACROS(7)
>
> NAME
> feature_test_macros - feature test macros
>
> SYNOPSIS
> #include <features.h>
>
> _FILE_OFFSET_BITS
> Defining this macro with the value 64 automatically converts
> references to 32-bit functions and data types related to file
> I/O and file system operations into references to their 64-bit
> counterparts. This is useful for performing I/O on large files
> (> 2 Gigabytes) on 32-bit systems. (Defining this macro permits
> correctly written programs to use large files with only a recom‐
> pilation being required.) 64-bit systems naturally permit file
> sizes greater than 2 Gigabytes, and on those systems this macro
> has no effect.
>
>
More information about the kernel-team
mailing list