Patching VMware Tools in Fedora 18

Patching VMware Tools so HGFS (host-guest file system) will run in Fedora 18

There are several bugs in the current release of VMware Tools (VMwareTools-9.2.2-893683). Below are some patches and scripts to fix those. You can download these files from my GitHub repo:

https://github.com/ebdevrepo/bin

People have also had good results with this patch: http://communities.vmware.com/message/2130250

(1) Kernel include file bug

There is currently a bug in the vmware-install.pl script. It will fail to find header files. To fix that, run this script [download] and then re-run vmware-install.pl:

#!/bin/bash
KVER=`uname -r`
cp /usr/src/kernels/$KVER/include/generated/uapi/linux/version.h /lib/modules/$KVER/build/include/linux/

(2) VMCI driver bug

There is a second bug in the vmware-install.pl script. It will fail to compile the vmci driver. See these posts:

https://patchwork.kernel.org/patch/1963371/
http://communities.vmware.com/message/2182440#2182440

You know you are encountering this bug if you see these compile errors when running vmware-install.pl:


make[1]: Entering directory `/usr/src/kernels/3.8.1-201.fc18.x86_64'
  CC [M]  /tmp/modconfig-grfWex/vmci-only/linux/driver.o
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:127:4: error: implicit declaration of function ‘__devexit_p’ [-Werror=implicit-function-declaration]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:127:4: error: initializer element is not constant
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:127:4: error: (near initialization for ‘vmci_driver.remove’)
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:1754:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘vmci_probe_device’
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:1982:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘vmci_remove_device’
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:119:12: warning: ‘vmci_probe_device’ used but never defined [enabled by default]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:121:13: warning: ‘vmci_remove_device’ used but never defined [enabled by default]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:2063:1: warning: ‘vmci_interrupt’ defined but not used [-Wunused-function]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:2137:1: warning: ‘vmci_interrupt_bm’ defined but not used [-Wunused-function]
/tmp/modconfig-grfWex/vmci-only/linux/driver.c:1717:1: warning: ‘vmci_enable_msix’ defined but not used [-Wunused-function]
cc1: some warnings being treated as errors
make[2]: *** [/tmp/modconfig-grfWex/vmci-only/linux/driver.o] Error 1
make[1]: *** [_module_/tmp/modconfig-grfWex/vmci-only] Error 2
make[1]: Leaving directory `/usr/src/kernels/3.8.1-201.fc18.x86_64'
make: *** [vmci.ko] Error 2
make: Leaving directory `/tmp/modconfig-grfWex/vmci-only'

The communication service is used in addition to the standard communication 
between the guest and the host.  The rest of the software provided by VMware 
Tools is designed to work independently of this feature.
If you wish to have the VMCI feature, you can install the driver by running 
vmware-config-tools.pl again after making sure that gcc, binutils, make and the
kernel sources for your running kernel are installed on your machine. These 
packages are available on your distribution's installation CD.


To work around this, first install the patch utility:

  sudo yum install patch

Create this vmware9.k3.8rc4.patch file [download] in your vmware-tools-distrib directory:

--- compat_mm.h.orig 2013-03-04 01:36:39.184925478 -0800
+++ compat_mm.h 2013-03-04 01:40:37.793728289 -0800
@@ -91,8 +91,9 @@
 /*
  * In 2.4.10, vmtruncate was changed from returning void to returning int.
+ * In 3.8.0,  vmtruncate was removed
  */
-#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 4, 10)
+#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 8, 0)
 #define compat_vmtruncate(inode, size)                                        \
 ({                                                                            \
    int result = 0;                                                            \
@@ -100,7 +101,16 @@
    result;                                                                    \
 })
 #else
-#define compat_vmtruncate(inode, size) vmtruncate(inode, size)
+#define compat_vmtruncate(inode, size)                                        \
+({                                                                            \
+   result = inode_newsize_ok(inode, size);                                    \
+   if (!result)                                                               \
+   {                                                                          \
+      truncate_setsize(inode, size);                                          \
+   }                                                                          \
+   result;                                                                    \
+})
+
 #endif

Then run this script from your vmware-tools-distrib directory [download]. If it asks you if this is a reverse patch, select no (type the letter 'n').

#!/bin/bash
pushd lib/modules/source
if [[ ! -f vmci.tar.orig ]]
then
  cp vmci.tar vmci.tar.orig
fi
rm -rf vmci-only
tar xf vmci.tar
pushd vmci-only
patch -p1 < ../../../../vmware9.k3.8rc4.patch
popd
tar cf vmci.tar vmci-only
rm -rf vmci-only
popd


(3) VMHGFS bug


You know you are encountering this bug if you see these compile errors when running vmware-install.pl:


  CC [M]  /tmp/modconfig-kxIA61/vmhgfs-only/inode.o
/tmp/modconfig-kxIA61/vmhgfs-only/inode.c: In function ‘HgfsTruncatePages’:
/tmp/modconfig-kxIA61/vmhgfs-only/inode.c:888:4: error: implicit declaration of function ‘vmtruncate’ [-Werror=implicit-function-declaration]
cc1: some warnings being treated as errors
make[2]: *** [/tmp/modconfig-kxIA61/vmhgfs-only/inode.o] Error 1
make[1]: *** [_module_/tmp/modconfig-kxIA61/vmhgfs-only] Error 2
make[1]: Leaving directory `/usr/src/kernels/3.8.1-201.fc18.x86_64'
make: *** [vmhgfs.ko] Error 2
make: Leaving directory `/tmp/modconfig-kxIA61/vmhgfs-only'

The filesystem driver (vmhgfs module) is used only for the shared folder 
feature. The rest of the software provided by VMware Tools is designed to work 
independently of this feature.

If you wish to have the shared folders feature, you can install the driver by 
running vmware-config-tools.pl again after making sure that gcc, binutils, make
and the kernel sources for your running kernel are installed on your machine. 
These packages are available on your distribution's installation CD.


Create this vmware9.compat_mm.patch file [download] in your vmware-tools-distrib directory:

--- ./compat_mm.h 2013-03-04 01:36:39.184925478 -0800
+++ ./compat_mm.h 2013-03-04 01:40:37.793728289 -0800

@@ -91,8 +91,9 @@
 /*
  * In 2.4.10, vmtruncate was changed from returning void to returning int.
+ * In 3.8.0,  vmtruncate was removed
  */
-#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 4, 10)
+#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 8, 0)
 #define compat_vmtruncate(inode, size)                                        \
 ({                                                                            \
    int result = 0;                                                            \
@@ -100,7 +101,16 @@
    result;                                                                    \
 })
 #else
-#define compat_vmtruncate(inode, size) vmtruncate(inode, size)
+#define compat_vmtruncate(inode, size)                                        \
+({                                                                            \
+   result = inode_newsize_ok(inode, size);                                    \
+   if (!result)                                                               \
+   {                                                                          \
+      truncate_setsize(inode, size);                                          \
+   }                                                                          \
+   result;                                                                    \
+})
+
 #endif

Then run this script [download] from your vmware-tools-distrib directory:

#!/bin/bash


pushd lib/modules/source

if [[ ! -f vmhgfs.tar.orig ]]
then
  cp vmhgfs.tar vmhgfs.tar.orig
fi
rm -rf vmhgfs-only
tar xf vmhgfs.tar
pushd vmhgfs-only/shared
patch -p1 < ../../../../../vmware9.compat_mm.patch
popd
tar cf vmhgfs.tar vmhgfs-only
rm -rf vmhgfs-only
popd

** NOTE ** Any time there is a kernel update you will need to re-install VMware Tools to get hgfs working again (the ability for the VM to access files from the host OS).

Comments

  1. I encountered bug 3 while installing Fedora 18 in my VM. Your steps helped me fix it. Thank you!

    ReplyDelete
  2. HELO!
    For me it raises :



    patching file compat_mm.h
    patch: **** malformed patch at line 14: @@ -100,7 +101,16 @@


    Kernel 3.8.3.201 FC 18

    ReplyDelete
  3. Marfaru,

    It is likely just a cut-and-paste problem. Try downloading the patch directly from my GitHub repo:

    https://raw.github.com/ebdevrepo/bin/master/vmware9.compat_mm.patch

    ReplyDelete
  4. Thanks! from me as well :)

    ReplyDelete
  5. I still get the hgfs compile error after patching...

    # uname -a
    Linux localhost.localdomain 3.8.5-201.fc18.x86_64 #1 SMP Thu Mar 28 21:01:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

    Any help appreciated...

    ReplyDelete
  6. Donnie - I just tried a fresh install of VMWare Tools on 3.8.5 and the bug fixes each worked fine. I did find a bug in the shell script that applies the patch, though. It breaks if you invoke it with a relative path. Maybe the patch didn't get applied. Try it again, watching to make sure that the patch doesn't fail.

    ReplyDelete
  7. I managed to encounter all of these but got here after using a similar solution to the first bug.

    I am wondering why is it when I run the scripts from vmware-tools-distrib folder which i have in my home directory, I always get the error file not found. So instead of using the $PATH_DIR on both scripts, I used the absolute path ~/vmware-tools-distrib/. With this I managed to go through the second bug, but the vmhgfs still is giving me the same error which followed by the vmxnet throwing the similar error.

    Any thoughts on this? Thanks!

    ReplyDelete
    Replies
    1. I've fixed the path bugs, thanks for pointing those out.

      Did you get any further with the patches, or is it still broken?

      Delete
    2. Yes I did, but I did it with another patch as well. For me it seems to work with both solutions.


      Here was the other patch I used.
      http://communities.vmware.com/message/2130250

      Thanks you!

      Delete
  8. Brilliant, thanks!

    ReplyDelete
  9. Thanks a lot!

    for those that are experiencing the VMHGFS bug in Ubuntu 13.04 (bug number 3), the solution in this page works fine.
    you just have to copy both files to the vmware tools distrib directory (/usr/lib/vmware-tools), and tweak the script by replacing the line
    pushd lib/modules/source

    with
    pushd modules/source

    After applying the script, vmware-config-tools.pl will compile fine.

    ReplyDelete
  10. I encountered these issues with vmci and vmhgfs while installing vmware-tools from VMWare Fusion 5.0.3 on Mac host and Ubuntu 13.0.4 (kernel 3.8.0.19-generic). I followed the instructions above and it solved my issues.

    Thanks a lot.

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Ok, I have realized that the lib is located in the dystro directory. But, instead of ‘vmtruncate’ I have got

    /tmp/modconfig-FiaFGk/vmhgfs-only/inode.c:1821:64: error: macro "hlist_for_each_entry" passed 4 arguments, but takes just 3
    /tmp/modconfig-FiaFGk/vmhgfs-only/inode.c:1821:7: error: ‘hlist_for_each_entry’ undeclared (first use in this function)

    In function ‘HgfsPermission’: error.

    ReplyDelete
    Replies
    1. It is removed by one more patch, https://github.com/Sabayon/for-gentoo/blob/master/app-emulation/open-vm-tools-kmod/files/open-vm-tools-kmod-linux-3.9.patch

      All errors are gone, hgfs module is loaded but mount is not operational. I cannot access the host.

      Delete
  13. Ok, I had to re-enable the share in the VMWare. Thanks, now I have hgfs working.

    ReplyDelete
  14. Great work Eric! Thank you for sharing this with the community!

    -Trevor

    ReplyDelete
  15. Excellent fix! Thank you so much for your help.

    ReplyDelete
  16. HGFS-patch works for Ubuntu 12.04 under VMware Workstation 9 on Windows 7 Pro.

    Thank you so much.

    ReplyDelete

Post a Comment

Popular posts from this blog

Setting up gdb in Fedora

Installing Fedora 18 in a VMware Fusion VM