You want to use an anonymous function:
%# define Matr here
Matr = rand(3); %# e.g. 3x3 double array with random numbers
%# define the anonymous function to pass to nlinfit
%# @ signals an anonymous function
%# (x,y) are the two inputs the function takes
%# f(x,y,Matr) is the function that is called with variable x,y
%# and constant Matr as defined in the workspace at the moment
%# modelfun gets defined.
modelfun = @(x,y)f(x,y,Matr);
%# call nlinfit
result = nlinfit(X,Y,modelfun,beta0);