[olug] large files

[Vincent]

I regularly deal with files larger than 2GB.  Most are mysql databases/ dumps and tarballs.  The only problem I've encountered is in
data corruption when transferring them with scp.  That dosen't happen often, but enough to be wary of.
I recall reading about some apps (specifically older versions of vim) which set "safe" environment limitations (ulimit -f/m) for
themselves to avoid encountering OS limitations.  That could have made sense at thr time since most distro's don't set memory or
file size limitations by default and no common filesystems or kernels could handle large files..

[--- at --- tmp]$ uname -r
2.4.18-5smp
[--- at --- tmp]$ mount | grep /pub
/dev/sda9 on /pub type ext3 (rw,noatime,data=journal)
[--- at --- tmp]$ ls -lhF news_archive_db.sql
-rw-r--r--    - ---      ---         3.1G Mar 21 01:21 news_archive_db.sql


----- Original Message -----
From: "Matthew G. Marsh" <mgm at midwestlinux.com>
To: "Omaha Linux User Group" <olug at olug.org>
Sent: Saturday, March 22, 2003 11:06 AM
Subject: Re: [olug] large files


> On Sat, 22 Mar 2003, Vincent wrote:
>
> >                              Ext3           ReiserFS    XFS                 JFS
> > File system max     16384 Gb  17592 Gb   18,000 Pb+     32 Pb
> > File size max          2048 Gb    1 Eb*          9,000 Pb         4 Pb
>
> Yes note your first numbers - reality is that on a 32 bit OS (aka Linux on
> x86) the _file_ size limit is still 2G (2048M). Try it some time -
> seriously I still run into this especially with Squid and SNORT logs...
>
> Once you have a _file_ over 2G the application usually just stops working.
>
> > The 2.4 kernel has a limit of 2048 Gb for a single block device, so no file system larger than that can be created at this time
> > (without patching the standard kernel). This restriction could be removed in the 2.5.x development kernel, and there are patches
> > available to remove this limit, but as of 2.5.29, the patches haven't been officially included yet.
> >
> > The largest block size supported on ia32 is 4Kb.
> >
> > http://www.linux-mag.com/2002-10/jfs_05.html
> >
> > ----- Original Message -----
> > From: "Phil Brutsche" <phil at brutsche.us>
> > To: "Omaha Linux User Group" <olug at olug.org>
> > Sent: Saturday, March 22, 2003 8:18 AM
> > Subject: Re: [olug] large files
> >
> >
> > > A long time ago, in a galaxy far, far way, someone said...
> > >
> > > > I was wondering, does anybody know the maximum filesize limitation for
> > > > ext3 filesystems on a 32-bit machine?
> > >
> > > Very big, provided your glibc and applications are compiled for 64-bit
> > > file support.
> > >
> > > Oh, and if it's not RedHat, chances are you're OK in terms of large file
> > > support.
> > >
> > > --
> > >
> > > Phil Brutsche
> > > phil at brutsche.us
> > > _______________________________________________
> > > OLUG mailing list
> > > OLUG at olug.org
> > > http://lists.olug.org/mailman/listinfo/olug
> >
> > _______________________________________________
> > OLUG mailing list
> > OLUG at olug.org
> > http://lists.olug.org/mailman/listinfo/olug
> >
>
> --------------------------------------------------
> Matthew G. Marsh,  President
> Paktronix Systems LLC
> 1506 North 59th Street
> Omaha  NE  68104
> Phone: (402) 553-2288
> Email: mgm at midwestlinux.com
> WWW:  http://www.midwestlinux.com
> --------------------------------------------------
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> http://lists.olug.org/mailman/listinfo/olug