You're just falling. You don't actually do anything, so while you may be distracted by the scrolling landscape, I see no reason for you to stand, arms crossed, being a sitting duck.
Liken it to the situation of someone falling with a parachute. Are they unable to speak and move their arms while falling?
By the way... The speed of the fall with a parachute is about 15 km/h. The speed of the fall with FF is about 11 km/h -- the fall is gentler, slower, than with the parachute.
So, yes, you can definitely cast while feather falling. If the character isn't used to it, I'd ask for a Concentration check, DC 5+spell level. If the character takes some training doing that (like, during down time, for a week or two, jump often from a high tower or a flying carpet with FF, and cast one spell or two during the fall); I would not even ask for the check. (It's not worth a feat.)