If you don't have a sword, then there will be an invisible wall behind the trees preventing you from accessing the Forest of Fairies early. However, that invisible wall doesn't extend onto the ledge on the left side. With enough speed, it is possible to land onto that ledge and avoid the invisible wall alltogether. To reach the speed necessary, you can either use superswimming speed, or use item sliding. The video below shows a setup to enter the Forest of Fairies with item sliding that doesn't involve having perfect speed:
If you picked up the gossip stone in FF1, then the bridge leading to FoF will be broken. It is however possible to item slide accross the broken bridge as seen in the video above.
Entering FoF early can be very important depending on the route. If you don't save Tetra in the forest normally, by performing Outset Escape to skip it, entering the forest later in the game can have unpleasant results. If you haven't saved Tetra, and if you enter FoF either during night, or after FF1, then the game will simply crash no matter what, which can prevent you from ever obtaining the last wallet upgrade. If you wish on not saving Tetra in a 100% speedrun, then you must obtain that wallet upgrade before beating FF1 and during the day.
By getting damaged by the fire and then quickly using the Hookshot on the torch, you can enter the hole before the temporary damage invincibility wears off. This trick can be performed on floors 21-29 as well as any other floors with torches. The Iron Boots can also be used in order to minimize the distance of the knockback from the fire so that it is easier to enter the hole with the Hookshot.
You can actually skip every single floor in savage labyrinth with a well positioned and precise Item Slide through the flames that block each grotto.