Depends on what one wants to pass through the FFI.  On x86-64/EMT64 structure passing conventions are efficient and complex, distributing structure fields across integer and floating-point registers.  It is possible to interpret such calls given a type signature for the structure to be passed (I did it for VisualWorks) but it is not easy and it is slow.  I would counsel writing a proper ABI compiler in Smalltalk that generates e.g. RTL that is either interpreted or compiled by the VM to make the actual call.

In theory the use of 64-bit libs, provided one has the relevant FFI support should work fine.  Smalltalk has the ability to deal with 64-bit and 128-bit values easily.  But one does have to write all the support.

To get acquainted with the de facto standard x86-64 ABI read the