Answer
In the Name of Allah, Most Gracious, Most Merciful.
All praise and thanks are due to Allah, and peace and blessings be upon His Messenger.
In this fatwa:
The fine imposed on the borrower for late return of the books is not a riba-based penalty, because it is not in return for a loan; rather it is a fee for using the book after the end of the borrowing period.
Answering your question, the Fatwa Center at Islam Q and A, states:
There is nothing wrong with borrowing books, but they must be returned on time. The one who is late in returning them with no excuse is sinning and it is permissible to impose a fine on him for the delay; this fine is a fee for his making use of the book during the excess period.
If this fine is agreed upon from the outset, there is nothing wrong with that, such as if it is said that he is obliged to pay a certain amount of money for each day that he was late in returning the book.
If the fine was not stipulated from the outset, then he must pay the typical amount for each day.
It says in Kashshaf al-Qina‘ (4/68):
“It is permissible to borrow a mount to ride it to a specific place, and if he goes beyond that, then he has transgressed, because he has done that without the permission of the owner, and he must pay a typical fee for the excess distance.”
Sheikh Ibn Baz (may Allah have mercy on him) was asked:
“In some mosques they lend books, on condition that whoever is late in returning the books must pay a specific penalty or fine for each day, with the money to be spent on the mosque or on maintaining the mosque. Is this permissible?”
He answered:
“Yes, this comes under the heading of rental and hire. If he is late (in returning the books), then he has continued the rental or hire, using the books for longer than the specified period. I do not think there is anything wrong with this, because it encourages people to adhere to the terms and conditions, and not to be careless about the borrower keeping the books with him. If the borrowing period is defined as five days, or six days, and so on, and he keeps the books for longer, then he must pay such and such as a penalty for returning the books late. There is nothing wrong with that because it serves a great purpose.” (Fatawa Noor ‘ala ad-Darb (11/296)
Almighty Allah knows best.
Source: www.islamqa.info