CVE-2010-0727 kernel: gfs/gfs2 locking code DoS flaw
static int
gfs_lock(struct file *file, int cmd, struct file_lock *fl)
{
..
if ((ip->i_di.di_mode & (S_ISGID | S_IXGRP)) == S_ISGID)
return -ENOLCK;
..
}
This is a check for mandatory locking where the GFS/GFS2 locking code will skip the lock in case sgid bits are set for the file. This can be triggered to cause a crash on a system mounting a GFS/GFS2 filesystem.
I believe only GFS2 is part of the upstream kernel, and GFS only affects Red Hat Enterprise Linux.