The Sensex was trading lower by 47 points, while the Nifty dropped 12 points as Asian markets were lower after US markets dropped after Yellen's comments.
Among the early losers in trade were banking stocks including the likes of ICICI Bank, HDFC Bank and IndusInd Bank.
Spice Jet shares fell after the DGCA issued the company a show cause notice as the why their license should not be cancelled after the employees were found dancing in the aisle.
Technology stocks were having a mixed day with HCL Tech and Tata Consultancy Services trading lower, while Infosys was trading marginally higher.
Mid cap stocks also fell in trade with notable losers being Union Bank and Adani Power.
Among the stocks that were trading higher from the Nifty set of stocks were the metal names including the likes of Tata Steel, Sesa Sterlite and Hindalco.
Shares in ONGC continued to be under pressure and was down more than 1 per cent in trade. Larsen and Toubro which had fallen on Wednesday continued to drift lower and was down 0.50 per cent.
Meanwhile, markets across Asia were trading with losses after the US Federal Reserve Chair hinted at an earlier than expected interest rate hike. The Hong Kong's Hang Sang and the Jakarta Composite were the major losers in trade.