When you have a data connection that pulls a large number of files or
data from somewhere, it will slow infopath down because it tries to
keep track of all the different files. Even if these are small files,
like the infopath xml documents, your form will suffer for them. Two
solutions I have found to this involve programming: the first is to
gather data from your data sources programmatically to limit the number
pulled; the second is to filter the data you get from your
dataconnection programmatically. The easiest of the two is the second
option. If possible, you can go through your data connection and filter
out the data that you know for certain will not be needed with this
specific form. So long as infopath doesn't have to keep track of more
than a couple hundred documents, it is somewhat manageable. Here is
some sample code:

'this gets the specific data that you want that you want
docHolder = myDoc.selectSingleNode("special xpath
query").cloneNode(true)

'this will clear the data connection and prevent your form from being
slow
myDoc.childNodes(0).removeChild(myDoc.childNodes(0).childNodes(0))

'add the files you selected back into the dataconnection
myDoc.childNodes(0).appendNode(docHolder)