In recent years, online learning has gained widespread attention in clinical trials and treatment recommendations due to its suitability for handling the bandit feedback nature of treatment outcomes. However, healthcare decision-making problems are more complex than the traditional applications of online learning, such as advertisement recommendations, dialogue response selection, and influence maximization. Two key features set healthcare problems apart. Firstly, doctors cannot engage in random exploration and change prescriptions frequently as this may cause excessive anxiety for patients. Secondly, treatment outcomes may be observed with delayed feedback, as some pharmaceutical ingredients take time to show their effects. In this talk, we will demonstrate how existing online learning algorithms can be modified to account for these unique features. We will also discuss the theoretical results and numerical performance of the proposed algorithms.