You can enter the gardens at night. It's not necessarily peaceful with the wolves trying to get in at all times... but it's mostly safe.

[Mostly.]

I would just suggest you bring a weapon or take an escort just in case one of them makes it into the castle walls.