I’d make some sort of adventure out of it. In order to research a new spell they’ll have to invest time, money and resources. Let’s say you need a study/library in your stronghold. At least. You might also need a laboratory. And If trying to raise new monsters, a breeding pen. As for time I’d say something like two months per spell level. Each month requires money (make it expensive. There’s a reason not every mage is doing this...) and ingredients. Ingredients require an adventure. Go raid a library of the Rust Brothers to get a tome of forbidden knowledge, raid a harpy nest and bring back (insert gross detail) fight this demon that terrorise the area to collect mog. Stuff like that. Then have them succeed with a Lore roll in the end to see if the experiments was a success or if they failed and need to start over. Push that roll, they want? Ok. So, a pushed roll is only allowed if there’s a stake, something at risk. One or two points of damage to an attribute while in their stronghold and not adventuring is no threat. I’d say, for each bane on a pushed roll something goes terribly wrong. The laboratory is destroyed by an explosion, the hireling working the breeding pen gets eaten and has to be replaced. Something like that. naturally the roll gets a minus modifier for each spell level above the first. There is also the possibility to get a bonus die if they spend more time on this. Let’s say doubling the time spent gives a plus one modifier. This is why it usually takes years to come up with new, powerful spells. Remember, each new month requires money and resources.
For researched animals or monsters I’d basically use the same rules but then add time for the creatures to grow. Which can be years for large beasts. And the cost of feeding a pack of hydras....
Now, if you run a campaign and don’t want them to take all this time because you have a campaign clock that’s ticking away but still want them to be able to do this you could, of course, alter the time needed. Or you could bring them on an adventure to find another dimension where time moves faster or let them travel there using the portal spell.
If my players want to do things like this, for which there are no rules, I usually come up with things like this and make adventure hooks for them. They also frequently suggest things like this to me. You want to research a potion of strength? Well, what could possibly be needed as a resource, I wonder...? Oh, you planned on travel to the mountains to find a minotaur lair because you think its horns would be needed? Actually, that’s just what you need!
Any ways, this is how I would run it. Hope it gives you some useful ideas. Best of luck. And please, feel free to tell us how it plays out.