Patching VMware Tools in Fedora 18

Patching VMware Tools so HGFS (host-guest file system) will run in Fedora 18

There are several bugs in the current release of VMware Tools (VMwareTools-9.2.2-893683). Below are some patches and scripts to fix those. You can download these files from my GitHub repo:

https://github.com/ebdevrepo/bin

People have also had good results with this patch: http://communities.vmware.com/message/2130250

(1) Kernel include file bug

There is currently a bug in the vmware-install.pl script. It will fail to find header files. To fix that, run this script [download] and then re-run vmware-install.pl:

#!/bin/bash
KVER=`uname -r`
cp /usr/src/kernels/$KVER/include/generated/uapi/linux/version.h /lib/modules/$KVER/build/include/linux/

(2) VMCI driver bug

There is a second bug in the vmware-install.pl script. It will fail to compile the vmci driver. See these posts:

https://patchwork.kernel.org/patch/1963371/
http://communities.vmware.com/message/2182440#2182440

You know you are encountering this bug if you see these compile errors when running vmware-install.pl:


make[1]: Entering directory `/usr/src/kernels/3.8.1-201.fc18.x86_64'
  CC [M]  /tmp/modconfig-grfWex/vmci-only/linux/driver.o
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:127:4: error: implicit declaration of function ‘__devexit_p’ [-Werror=implicit-function-declaration]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:127:4: error: initializer element is not constant
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:127:4: error: (near initialization for ‘vmci_driver.remove’)
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:1754:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘vmci_probe_device’
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:1982:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘vmci_remove_device’
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:119:12: warning: ‘vmci_probe_device’ used but never defined [enabled by default]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:121:13: warning: ‘vmci_remove_device’ used but never defined [enabled by default]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:2063:1: warning: ‘vmci_interrupt’ defined but not used [-Wunused-function]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:2137:1: warning: ‘vmci_interrupt_bm’ defined but not used [-Wunused-function]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:1717:1: warning: ‘vmci_enable_msix’ defined but not used [-Wunused-function]
cc1: some warnings being treated as errors
make[2]: *** [/tmp/modconfig-grfWex/vmci-only/linux/driver.o] Error 1
make[1]: *** [_module_/tmp/modconfig-grfWex/vmci-only] Error 2
make[1]: Leaving directory `/usr/src/kernels/3.8.1-201.fc18.x86_64'
make: *** [vmci.ko] Error 2
make: Leaving directory `/tmp/modconfig-grfWex/vmci-only'

The communication service is used in addition to the standard communication 
between the guest and the host.  The rest of the software provided by VMware 
Tools is designed to work independently of this feature.
If you wish to have the VMCI feature, you can install the driver by running 
vmware-config-tools.pl again after making sure that gcc, binutils, make and the
kernel sources for your running kernel are installed on your machine. These 
packages are available on your distribution's installation CD.


To work around this, first install the patch utility:

  sudo yum install patch

Create this vmware9.k3.8rc4.patch file [download] in your vmware-tools-distrib directory:

--- compat_mm.h.orig 2013-03-04 01:36:39.184925478 -0800
+++ compat_mm.h 2013-03-04 01:40:37.793728289 -0800
@@ -91,8 +91,9 @@
 /*
  * In 2.4.10, vmtruncate was changed from returning void to returning int.
+ * In 3.8.0,  vmtruncate was removed
  */
-#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 4, 10)
+#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 8, 0)
 #define compat_vmtruncate(inode, size)                                        \
 ({                                                                            \
    int result = 0;                                                            \
@@ -100,7 +101,16 @@
    result;                                                                    \
 })
 #else
-#define compat_vmtruncate(inode, size) vmtruncate(inode, size)
+#define compat_vmtruncate(inode, size)                                        \
+({                                                                            \
+   result = inode_newsize_ok(inode, size);                                    \
+   if (!result)                                                               \
+   {                                                                          \
+      truncate_setsize(inode, size);                                          \
+   }                                                                          \
+   result;                                                                    \
+})
+
 #endif

Then run this script from your vmware-tools-distrib directory [download]. If it asks you if this is a reverse patch, select no (type the letter 'n').

#!/bin/bash
pushd lib/modules/source
if [[ ! -f vmci.tar.orig ]]
then
  cp vmci.tar vmci.tar.orig
fi
rm -rf vmci-only
tar xf vmci.tar
pushd vmci-only
patch -p1 < ../../../../vmware9.k3.8rc4.patch
popd
tar cf vmci.tar vmci-only
rm -rf vmci-only
popd

(3) VMHGFS bug

You know you are encountering this bug if you see these compile errors when running vmware-install.pl:


  CC [M]  /tmp/modconfig-kxIA61/vmhgfs-only/inode.o
/tmp/modconfig-kxIA61/vmhgfs-only/inode.c: In function ‘HgfsTruncatePages’:
/tmp/modconfig-kxIA61/vmhgfs-only/inode.c:888:4: error: implicit declaration of function ‘vmtruncate’ [-Werror=implicit-function-declaration]
cc1: some warnings being treated as errors
make[2]: *** [/tmp/modconfig-kxIA61/vmhgfs-only/inode.o] Error 1
make[1]: *** [_module_/tmp/modconfig-kxIA61/vmhgfs-only] Error 2
make[1]: Leaving directory `/usr/src/kernels/3.8.1-201.fc18.x86_64'
make: *** [vmhgfs.ko] Error 2
make: Leaving directory `/tmp/modconfig-kxIA61/vmhgfs-only'

The filesystem driver (vmhgfs module) is used only for the shared folder 
feature. The rest of the software provided by VMware Tools is designed to work 
independently of this feature.

If you wish to have the shared folders feature, you can install the driver by 
running vmware-config-tools.pl again after making sure that gcc, binutils, make
and the kernel sources for your running kernel are installed on your machine. 
These packages are available on your distribution's installation CD.

Create this vmware9.compat_mm.patch file [download] in your vmware-tools-distrib directory:

--- ./compat_mm.h 2013-03-04 01:36:39.184925478 -0800

+++ ./compat_mm.h 2013-03-04 01:40:37.793728289 -0800

@@ -91,8 +91,9 @@

 /*

  * In 2.4.10, vmtruncate was changed from returning void to returning int.

+ * In 3.8.0,  vmtruncate was removed

  */

-#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 4, 10)

+#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 8, 0)

 #define compat_vmtruncate(inode, size)                                        \

 ({                                                                            \

    int result = 0;                                                            \

@@ -100,7 +101,16 @@

    result;                                                                    \

 })

 #else

-#define compat_vmtruncate(inode, size) vmtruncate(inode, size)

+#define compat_vmtruncate(inode, size)                                        \

+({                                                                            \

+   result = inode_newsize_ok(inode, size);                                    \

+   if (!result)                                                               \

+   {                                                                          \

+      truncate_setsize(inode, size);                                          \

+   }                                                                          \

+   result;                                                                    \

+})

+

 #endif

Then run this script [download] from your vmware-tools-distrib directory:

#!/bin/bash

pushd lib/modules/source

if [[ ! -f vmhgfs.tar.orig ]]

then

  cp vmhgfs.tar vmhgfs.tar.orig

fi

rm -rf vmhgfs-only

tar xf vmhgfs.tar

pushd vmhgfs-only/shared

patch -p1 < ../../../../../vmware9.compat_mm.patch

popd

tar cf vmhgfs.tar vmhgfs-only

rm -rf vmhgfs-only

popd

** NOTE ** Any time there is a kernel update you will need to re-install VMware Tools to get hgfs working again (the ability for the VM to access files from the host OS).